xx CarpetIOHDF5

CarpetIOHDF5

Erik Schnetter <schnetter@uni-tuebingen.de> Christian D. Ott <cott@aei.mpg.de> Thomas Radke <tradke@aei.mpg.de>

1 December 2004

Abstract

Thorn CarpetIOHDF5xprovides HDF5-based output to the Carpet mesh refinement driver in Cactus. This document explains CarpetIOHDF5’s usage and contains a specification of the HDF5 file format that was adapted from John Shalf’s FlexIO library.

Contents

x1 Introduction

x2 CarpetIOHDF5 Parameters x3 Serial versus Parallel Output x4 Using the flesh I/O API to produce HDF5 output x5 Checkpointing & Recovery and Importing Data x6 CarpetIOHDF5 Utility Programs

xx6.1 hdf5toasciixslicer
xx6.2 hdf5xextract

x7 Example Parameter File Excerpts xx7.1 Serial (unchunked) Output of Grid Variables xx7.2 Parallel (chunked) Output of Grid Variables xx7.3 Checkpointing & Recovery xx7.4 Importing Grid Variables via Filereader x8 Parameters x9 Interfaces x10 Schedule

1 Introduction Having encountered various problems with the Carpet I/O thorn CarpetIOFlexIO and the underlying FlexIO library, Erik Schnetter decided to write this thorn CarpetIOHDF5xwhich bypasses any intermediate binary I/O layer and outputs

in HDF51 ^{1Hierarchical Data Format version 5, see http://hdf.ncsa.uiuc.edu/whatishdf5.html for details} file format directly. CarpetIOHDF5xprovides output for the Carpet Mesh Refinement driver within the Cactus Code. Christian D. Ott added a file reader (analogous to Erik Schnetter’s implementation present in CarpetIOFlexIO) as well as checkpoint/recovery functionality to CarpetIOHDF5. Thomas Radke has taken over maintainence of this I/O thorn and is continuously working on fixing known bugs and improving the code functionality and efficiency.

The CarpetIOHDF5xI/O method can output any type of CCTK grid variables (grid scalars, grid functions, and grid arrays of arbitrary dimension); data is written into separate files named "<varname>.h5". It implements both serial and full parallel I/O – data files can be written/read either by processor 0 only or by all processors. Such datafiles can be used for further postprocessing (eg. visualization with OpenDX or

DataVault2 ^{2See our visualization page at http://www.cactuscode.org/Visualization/ for details.} ) or fed back into Cactus via the filereader capabilities of thorn IOUtil.

This document aims at giving the user a first handle on how to use CarpetIOHDF5. It also documents the HDF5 file layout used.

2 CarpetIOHDF5xParameters

Parameters to control the CarpetIOHDF5xI/O method are:

IOHDF5::outxevery (steerable)

How often to do periodic CarpetIOHDF5xoutput. If this parameter is set in the parameter file, it will override the setting of the shared IO::outxevery parameter. The output frequency can also be set for individual variables using the outxevery option in an option string appended to the

IOHDF5::outxvars parameter.

IOHDF5::outxdt (steerable)

output in intervals of that much coordinate time (overwrites IO::outxdt)

IOHDF5::outxcriterion (steerable)

criterion to select output intervals (overwrites IO::outxcriterion)

IOHDF5::outxvars (steerable)

The list of variables to output using the CarpetIOHDF5xI/O method. The variables must be given by their fully qualified variable or group name. The special keyword all requests CarpetIOHDF5xoutput for all variables. Multiple names must be separated by whitespaces.

Each group/variable name can have an option string attached in which you can specify a different output frequency for that individual variable, a set of individual refinement levels to be output, the compression level, or an individual output mode, e.g.

xxIOHDF5::out_varsx=x"wavetoy::phi{xout_everyx=x4xrefinement_levelsx=x{x1x2x}x}"

Option strings currently supported by CarpetIOHDF5xare: outxevery, outxunchunked,

refinementxlevels, and compressionxlevel.

IOHDF5::outxdir

The directory in which to place the CarpetIOHDF5xoutput files. If the directory doesn’t exist at startup it will be created. If parallel output is enabled and the directory name contains the substring "%u" it will be substituted by the processor ID. By this means each processor can have its own output directory. If this parameter is set to an empty string CarpetIOHDF5xoutput will go to the standard output directory as specified in IO::outxdir.

IOHDF5::compressionxlevel (steerable)

Compression level to use for writing HDF5 datasets. Automatic gzip dataset compression can be enabled by setting this integer parameter to values between 1 and 9 (inclusive), with increasing values requesting higher compression rates (at the cost of additional runtime for outputting HDF5 data); a value of zero (which is the default setting for this parameter) disables compression. The output compression level can also be set for individual variables using the compressionxlevel option in an option string appended to

the IOHDF5::outxvars parameter.

IO::outxsinglexprecision (steerable)

whether to output double-precision data in single precision

3 Serial versus Parallel Output

According to the ouptput mode parameter settings of (IO::outxmode, IO::outxunchunked, IO::outxprocxevery) of thorn IOUtil, thorn CarpetIOHDF5xwill output distributed grid variables either in serial from processor 0 into a single unchunked file

xxIO::out_modexxxxxx=x"onefile"
xxIO::out_unchunkedx=x"yes"

in serial from processor 0 into a single chunked file

xxIO::out_modexxxxxx=x"onefile"
xxIO::out_unchunkedx=x"no"

in parallel, that is, into separate chunked files (one per processor) containing the individual processors’ patches of the distributed grid variable

     xxIO::out_modexxxxxx=x"proc"

Unchunked means that an entire Cactus grid array (gathered across all processors) is stored in a single HDF5 dataset whereas chunked means that all the processor-local patches of this array are stored as separate HDF5 datasets (called chunks). Consequently, for unchunked data all interprocessor ghostzones are excluded from the output. In contrast, for chunked data the interprocessor ghostzones are included in the output.

When visualising chunked datasets, they probably need to be recombined for a global view on the data. This needs to be done within the visualisation tool (see also below), Cactus itself does not provide its own recombiner utility program for CarpetIOHDF5’s output files.

The default is to output distributed grid variables in parallel, each processor writing a file <varname>.filex<processor ID>.h5. The chunked/unchunked mode can also be set individually in a key/value option string (with the key outxunchunked and possible string values "true|false|yes|no") appended to a group/variable name in the outxvars parameter, eg.

xxIOHDF5::out_varsx=x"wavetoy::phi{out_unchunkedx=x'true'}xxgrid::coordinates"

will cause the variable phi to be output into a single unchunked file whereas other variables will still be output into separate chunked files (assuming the output mode is left to its default). Grid scalars and DISTRIB = CONST grid arrays are always output as unchunked data on processor 0 only.

Parallel output in a parallel simulation will ensure maximum I/O performance. Note that changing the output mode to serial I/O might only be necessary if the data analysis and visualisation tools cannot deal with chunked output files. Cactus itself, as well as many of the tools to visualise Carpet HDF5 data (see http://www.cactuscode.org/Visualization), can process both chunked and unchunked data. For instance, to visualise parallel output datafiles with DataVault, you would just send all the individual files to the DV server: hdf5todv phi.filex*.h5. In OpenDX the ImportCarpetIOHDF5 module can be given any filename from the set of parallel chunked files; the module will determine the total number of files in the set automatically and read them all.

4 Using the flesh I/O API to produce HDF5 output

Periodic output of grid variables is usually specified via I/O parameters in the parameter file and then automatically triggered by the flesh scheduler at each iteration step after analysis. If output should also be triggered at a different time, one can do that from within an application thorn by invoking one of the CCTKxOutputVar*() I/O routines provided by the flesh I/O API (see chapter B8.2 “IO” in the Cactus Users Guide). In this case, the application thorn routine which calls CCTKxOutputVar*() must be scheduled in level mode.

It should be noted here that – due to a restriction in the naming scheme of objects in an HDF5 data file – CarpetIOHDF5xcan output a given grid variable with given refinement level only once per timestep. Attempts of application thorns to trigger the output of the same variable multiple times during an iteration will result in a runtime warning and have no further effect. If output for a variable is required also for intermediate timesteps this can be achieved by calling CCTKxOutputVarAs*() with a different alias name; output for the same variable is then written into different HDF5 files based on the alias argument.

5 Checkpointing & Recovery and Importing Data

Thorn CarpetIOHDF5xcan also be used to create HDF5 checkpoint files and to recover from such files later on. In addition it can read HDF5 datafiles back in using the generic filereader interface described in the thorn documentation of IOUtil.

Checkpoint routines are scheduled at several timebins so that you can save the current state of your simulation after the initial data phase, during evolution, or at termination. Checkpointing for thorn CarpetIOHDF5xis enabled by setting the parameter IOHDF5::checkpoint = "yes".

A recovery routine is registered with thorn IOUtil in order to restart a new simulation from a given HDF5 checkpoint. The very same recovery mechanism is used to implement a filereader functionality to feed back data into Cactus.

Checkpointing and recovery are controlled by corresponding checkpoint/recovery parameters of thorn IOUtil (for a description of these parameters please refer to this thorn’s documentation).

6 CarpetIOHDF5 Utility Programs

6.1 hdf5toasciixslicer

This utility program extracts 1D lines and 2D slices from 3D HDF5 datasets produced by CarpetIOHDF5xand outputs them in CarpetIOASCII format (suitable to be further processed by gnuplot).

The hdf5toasciixslicer program is contained in the src/utils/ subdirectory of thorn CarpetIOHDF5. It is built with

xxmakex<configuration>-utils

where the executable ends up in the subdirectory exe/<configuration>/. For details on how to use the hdf5toasciixslicer program, run it with no command-line options (or with the --help option). hdf5toasciixslicer can be used on either chunked or unchunked data:

If the HDF5 data is unchunked, then hdf5toasciixslicer will output unchunked ASCII data.

If the HDF5 data is chunked, then hdf5toasciixslicer will output chunked ASCII data reflecting whatever subset of the HDF5 files are provided. That is, for example, the command

hdf5toascii_slicerxmy_variable.file_2.h5xmy_variable.file_4.h5

will output ASCII data only for that part of the Carpet grid which lived on processorsx2 andx4. It’s probably more useful to use the command

     hdf5toascii_slicerxmy_variable.file_*.h5

which will output ASCII data for the entire Carpet grid.

6.2 hdf5xextract

This utility program extracts selected datasets from any given HDF5 output file which may be useful when only certain parts (eg. a specific timestep) of large files are required (eg. for copying to some other location for further processing).

The hdf5xextract program is contained in the src/utils/ subdirectory of thorn CactusPUGHIO/IOHDF5. It is built with

xxmakex<configuration>-utils

where the executable ends up in the subdirectory exe/<configuration>/.

7 Example Parameter File Excerpts

7.1 Serial (unchunked) Output of Grid Variables

xx#xhowxoftenxtoxoutputxandxwherexoutputxfilesxshouldxgo
xxIO::out_everyx=x2
xxIO::out_dirxxx=x"wavetoy-data"
xx#xrequestxoutputxforxwavetoy::psixatxeveryxotherxiterationxforxtimelevelx0,
xx#xxxxxxxxxxxxxxxxforxwavetoy::phixeveryx4thxiterationxwithxtimelevelsx1xandx2
xxIOHDF5::out_varsx=x"wavetoy::phi{xout_everyx=x4xrefinement_levelsx=x{x1x2x}x}
xxxxxxxxxxxxxxxxxxxxxxwavetoy::psi"
xx#xwexwantxunchunkedxoutput
xx#x(becausexthexvisualisationxtoolxcannotxdealxwithxchunkedxdataxfiles)
xxIO::out_modexxxxxx=x"onefile"
xxIO::out_unchunkedx=x1

7.2 Parallel (chunked) Output of Grid Variables

xx#xhowxoftenxtoxoutput
xxIO::out_everyx=x2
xx#xeachxprocessorxwritesxtoxitsxownxoutputxdirectory
xxIOHDF5::out_dirx=x"wavetoy-data-proc%u"
xx#xrequestxoutputxforxwavetoy::psixatxeveryxotherxiterationxforxtimelevelx0,
xx#xxxxxxxxxxxxxxxxforxwavetoy::phixeveryx4thxiterationxwithxtimelevelsx1xandx2
xxIOHDF5::out_varsx=x"wavetoy::phi{xout_everyx=x4xrefinement_levelsx=x{x1x2x}x}
xxxxxxxxxxxxxxxxxxxxxxwavetoy::psi"
xx#xwexwantxparallelxchunkedxoutputx(notexthatxthisxalreadyxisxthexdefault)
xxIO::out_modex=x"proc"

7.3 Checkpointing & Recovery

xx#xsayxhowxoftenxwexwantxtoxcheckpoint,xhowxmanyxcheckpointsxshouldxbexkept,
xx#xhowxthexcheckpointsxshouldxbexnamed,xandxtheyxshouldxbexwrittenxto
xxIO::checkpoint_everyx=x100
xxIO::checkpoint_keepxx=x2
xxIO::checkpoint_filexx=x"wavetoy"
xxIO::checkpoint_dirxxx=x"wavetoy-checkpoints"
xx#xenablexcheckpointingxforxCarpetIOHDF5
xxIOHDF5::checkpointx=x"yes"
xx#######################################################
xx#xrecoverxfromxthexlatestxcheckpointxfound
xxIO::recover_filex=x"wavetoy"
xxIO::recover_dirxx=x"wavetoy-checkpoints"
xxIO::recoverxxxxxx=x"auto"

7.4 Importing Grid Variables via Filereader

xx#xwhichxdataxfilesxtoximportxandxwherextoxfindxthem
xxIO::filereader_ID_filesx=x"phixpsi"
xxIO::filereader_ID_dirxxx=x"wavetoy-data"
xx#xwhatxvariablesxandxwhichxtimestepxtoxread
xx#x(ifxthisxparameterxisxleftxempty,xallxvariablesxandxtimestepsxfound
xx#xxinxthexdataxfilesxwillxbexread)
xxIO::filereader_ID_varsxx=x"WaveToyMoL::phi{xcctk_iterationx=x0x}
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxWaveToyMoL::psi"

8 Parameters

11checkpoint 10Scope: privatex01 BOOLEAN

11Description: Do checkpointing with CarpetIOHDF5 ?

11                           10  x01  Default: no

11

11checkpointxeveryxdivisor  10Scope: privatex01  INT

11Description: Checkpoint if (iteration % outxevery) == 0

11 Range 10 x01 Default: -1
111:*  10Every so many iterations
11-1:0  10Disable periodic checkpointing

11

11checkpointxnext  10Scope: privatex01  BOOLEAN

11Description: Checkpoint at next iteration ?

11                           10  x01  Default: no

11

11compressionxlevel  10Scope: privatex01  INT

11Description: Compression level to use for writing HDF5 data

11 Range 10 x01 Default: (none)
110:9  10Higher numbers compress better, a value of zero disables compression

11

11minimumxsizexforxcompression10Scope: privatex01  INT

11Description: Only compress datasets larger than this many bytes

11 Range 10 x01 Default: 32768
110:*  10This should to be large enough so that compression gains outweigh the overhead

11

11onexfilexperxgroup  10Scope: privatex01  BOOLEAN

11Description: Write one file per group instead of per variable

11                           10  x01  Default: no

11onexfilexperxproc  10Scope: privatex01  BOOLEAN

11Description: Write one file per process instead of per variable

11                           10  x01  Default: no

11

11openxonexinputxfilexatxaxtime10Scope: privatex01  BOOLEAN

11Description: Open only one HDF5 file at a time when reading data from multiple chunked checkpoint/data files

11  Range  10  x01  Default: no
11no  10Open all input files first, then import data (most efficient)
11yes  10Process input files one after another (reduces memory requirements)

11

11out0dxcriterion  10Scope: privatex01  KEYWORD

11Description: Criterion to select 0D HDF5 slice output intervals, overrides outxevery

11 Range 10 x01 Default: default
11default  10Use IO::outxcriterion
11never  10Never output
11iteration  10Output every so many iterations
11divisor  10Output if iteration mod divisor == 0.
11time  10Output every that much coordinate time

11

11out0dxdir  10Scope: privatex01  STRING

11Description: Name of 0D HDF5 slice output directory, overrides IO::outxdir

11 Range 10 x01 Default: (none)
11ˆ$  10Empty: use IO::outxdir
11.+  10Not empty: directory name

11

11out0dxdt  10Scope: privatex01  REAL

11Description: How often to do 0D HDF5 slice output, overrides IO::outxdt

11 Range 10 x01 Default: -2
11(0:*  10In intervals of that much coordinate time
11                           10As often as possible
11-1  10Disable output
11-2  10Default to IO::outxdt

11

11out0dxevery  10Scope: privatex01  INT

11Description: How often to do 0D HDF5 slice output, overrides outxevery

11 Range 10 x01 Default: -2
111:*  10Output every so many time steps
11-1:0  10No output
11-2  10Use IO::outxevery

11

11out0dxpointxx  10Scope: privatex01  REAL

11Description: x coordinate for 0D points

11  Range  10  x01  Default: (none)

11*:*  10

11

11out0dxpointxxi  10Scope: privatex01  INT

11Description: x-index (counting from 0) for 0D points

11  Range  10  x01  Default: (none)

110:*  10

11out0dxpointxy  10Scope: privatex01  REAL

11Description: y coordinate for 0D points

11  Range  10  x01  Default: (none)

11*:*  10

11

11out0dxpointxyi  10Scope: privatex01  INT

11Description: y-index (counting from 0) for 0D points

11  Range  10  x01  Default: (none)

110:*  10

11

11out0dxpointxz  10Scope: privatex01  REAL

11Description: z coordinate for 0D points

11  Range  10  x01  Default: (none)

11*:*  10

11

11out0dxpointxzi  10Scope: privatex01  INT

11Description: z-index (counting from 0) for 0D points

11  Range  10  x01  Default: (none)

110:*  10

11

11out0dxvars  10Scope: privatex01  STRING

11Description: Variables to output in 0D HDF5 file format

11 Range 10 x01 Default: (none)
11                           10List of group or variable names

11

11out1dxcriterion  10Scope: privatex01  KEYWORD

11Description: Criterion to select 1D HDF5 slice output intervals, overrides outxevery

11 Range 10 x01 Default: default
11default  10Use IO::outxcriterion
11never  10Never output
11iteration  10Output every so many iterations
11divisor  10Output if (iteration % outxevery) == 0.
11time  10Output every that much coordinate time

11

11out1dxd  10Scope: privatex01  BOOLEAN

11Description: Do output along the diagonal

11                           10  x01  Default: yes

11

11out1dxdir  10Scope: privatex01  STRING

11Description: Name of 1D HDF5 slice output directory, overrides IO::outxdir

11 Range 10 x01 Default: (none)
11ˆ$  10Empty: use IO::outxdir
11.+  10Not empty: directory name

11

11out1dxdt  10Scope: privatex01  REAL

11Description: How often to do 1D HDF5 slice output, overrides IO::outxdt

11 Range 10 x01 Default: -2
11(0:*  10In intervals of that much coordinate time
11                           10As often as possible
11-1  10Disable output
11-2  10Default to IO::outxdt

11out1dxevery  10Scope: privatex01  INT

11Description: How often to do 1D HDF5 slice output, overrides outxevery

11 Range 10 x01 Default: -2
111:*  10Output every so many time steps
11-1:0  10No output
11-2  10Use IO::outxevery

11

11out1dxvars  10Scope: privatex01  STRING

11Description: Variables to output in 1D HDF5 file format

11 Range 10 x01 Default: (none)
11                           10List of group or variable names

11

11out1dxx  10Scope: privatex01  BOOLEAN

11Description: Do 1D HDF5 slice output in the x-direction

11                           10  x01  Default: yes

11

11out1dxxlinexy  10Scope: privatex01  REAL

11Description: y coordinate for 1D lines in x-direction

11  Range  10  x01  Default: (none)

11*:*  10

11

11out1dxxlinexyi  10Scope: privatex01  INT

11Description: y-index (counting from 0) for 1D lines in x-direction

11  Range  10  x01  Default: (none)

110:*  10

11

11out1dxxlinexz  10Scope: privatex01  REAL

11Description: z coordinate for 1D lines in x-direction

11  Range  10  x01  Default: (none)

11*:*  10

11

11out1dxxlinexzi  10Scope: privatex01  INT

11Description: z-index (counting from 0) for 1D lines in x-direction

11  Range  10  x01  Default: (none)

110:*  10

11

11out1dxy  10Scope: privatex01  BOOLEAN

11Description: Do 1D HDF5 slice output in the y-direction

11                           10  x01  Default: yes

11

11out1dxylinexx  10Scope: privatex01  REAL

11Description: x coordinate for 1D lines in y-direction

11  Range  10  x01  Default: (none)

11*:*  10

11

11out1dxylinexxi  10Scope: privatex01  INT

11Description: x-index (counting from 0) for 1D lines in y-direction

11  Range  10  x01  Default: (none)

110:*  10

11

11out1dxylinexz  10Scope: privatex01  REAL

11Description: z coordinate for 1D lines in y-direction

11  Range  10  x01  Default: (none)

11*:*  10

11out1dxylinexzi  10Scope: privatex01  INT

11Description: z-index (counting from 0) for 1D lines in y-direction

11  Range  10  x01  Default: (none)

110:*  10

11

11out1dxz  10Scope: privatex01  BOOLEAN

11Description: Do 1D HDF5 slice output in the z-direction

11                           10  x01  Default: yes

11

11out1dxzlinexx  10Scope: privatex01  REAL

11Description: x coordinate for 1D lines in z-direction

11  Range  10  x01  Default: (none)

11*:*  10

11

11out1dxzlinexxi  10Scope: privatex01  INT

11Description: x-index (counting from 0) for 1D lines in z-direction

11  Range  10  x01  Default: (none)

110:*  10

11

11out1dxzlinexy  10Scope: privatex01  REAL

11Description: y coordinate for 1D lines in z-direction

11  Range  10  x01  Default: (none)

11*:*  10

11

11out1dxzlinexyi  10Scope: privatex01  INT

11Description: y-index (counting from 0) for 1D lines in z-direction

11  Range  10  x01  Default: (none)

110:*  10

11

11out2dxcriterion  10Scope: privatex01  KEYWORD

11Description: Criterion to select 2D HDF5 slice output intervals, overrides outxevery

11 Range 10 x01 Default: default
11default  10Use IO::outxcriterion
11never  10Never output
11iteration  10Output every so many iterations
11divisor  10Output if (iteration % outxevery) == 0.
11time  10Output every that much coordinate time

11

11out2dxdir  10Scope: privatex01  STRING

11Description: Name of 2D HDF5 slice output directory, overrides IO::outxdir

11 Range 10 x01 Default: (none)
11ˆ$  10Empty: use IO::outxdir
11.+  10Not empty: directory name

11

11out2dxdt  10Scope: privatex01  REAL

11Description: How often to do 2D HDF5 slice output, overrides IO::outxdt

11 Range 10 x01 Default: -2
11(0:*  10In intervals of that much coordinate time
11                           10As often as possible
11-1  10Disable output
11-2  10Default to IO::outxdt

11out2dxevery  10Scope: privatex01  INT

11Description: How often to do 2D HDF5 slice output, overrides outxevery

11 Range 10 x01 Default: -2
111:*  10Output every so many time steps
11-1:0  10No output
11-2  10Use IO::outxevery

11

11out2dxvars  10Scope: privatex01  STRING

11Description: Variables to output in 2D HDF5 file format

11 Range 10 x01 Default: (none)
11                           10List of group or variable names

11

11out2dxxy  10Scope: privatex01  BOOLEAN

11Description: Do 2D HDF5 slice output in the xy-direction

11                           10  x01  Default: yes

11

11out2dxxyplanexz  10Scope: privatex01  REAL

11Description: z coordinate for 2D planes in xy-direction

11  Range  10  x01  Default: (none)

11*:*  10

11

11out2dxxyplanexzi  10Scope: privatex01  INT

11Description: z-index (counting from 0) for 2D planes in xy-direction

11  Range  10  x01  Default: (none)

110:*  10

11

11out2dxxz  10Scope: privatex01  BOOLEAN

11Description: Do 2D HDF5 slice output in the xz-direction

11                           10  x01  Default: yes

11

11out2dxxzplanexy  10Scope: privatex01  REAL

11Description: y coordinate for 2D planes in xz-direction

11  Range  10  x01  Default: (none)

11*:*  10

11

11out2dxxzplanexyi  10Scope: privatex01  INT

11Description: y-index (counting from 0) for 2D planes in xz-direction

11  Range  10  x01  Default: (none)

110:*  10

11

11out2dxyz  10Scope: privatex01  BOOLEAN

11Description: Do 2D HDF5 slice output in the yz-direction

11                           10  x01  Default: yes

11

11out2dxyzplanexx  10Scope: privatex01  REAL

11Description: x coordinate for 2D planes in yz-direction

11  Range  10  x01  Default: (none)

11*:*  10

11

11out2dxyzplanexxi  10Scope: privatex01  INT

11Description: x-index (counting from 0) for 2D planes in yz-direction

11  Range  10  x01  Default: (none)

110:*  10

11out3dxcriterion  10Scope: privatex01  KEYWORD

11Description: Criterion to select 3D HDF5 slice output intervals, overrides outxevery

11 Range 10 x01 Default: default
11default  10Use IO::outxcriterion
11never  10Never output
11iteration  10Output every so many iterations
11divisor  10Output if (iteration % outxevery) == 0.
11time  10Output every that much coordinate time

11

11out3dxdir  10Scope: privatex01  STRING

11Description: Name of 3D HDF5 slice output directory, overrides IO::outxdir

11 Range 10 x01 Default: (none)
11ˆ$  10Empty: use IO::outxdir
11.+  10Not empty: directory name

11

11out3dxdt  10Scope: privatex01  REAL

11Description: How often to do 3D HDF5 slice output, overrides IO::outxdt

11 Range 10 x01 Default: -2
11(0:*  10In intervals of that much coordinate time
11                           10As often as possible
11-1  10Disable output
11-2  10Default to IO::outxdt

11

11out3dxevery  10Scope: privatex01  INT

11Description: How often to do 3D HDF5 slice output, overrides outxevery

11 Range 10 x01 Default: -2
111:*  10Output every so many time steps
11-1:0  10No output
11-2  10Use IO::outxevery

11

11out3dxghosts  10Scope: privatex01  BOOLEAN

11Description: Output ghost zones (DEPRECATED)

11                           10  x01  Default: yes

11

11out3dxouterxghosts  10Scope: privatex01  BOOLEAN

11Description: Output outer boundary zones (assuming that there are nghostzones boundary points) (DEPRECATED)

11                           10  x01  Default: yes

11

11out3dxvars  10Scope: privatex01  STRING

11Description: Variables to output in 3D HDF5 file format

11 Range 10 x01 Default: (none)
11                           10List of group or variable names

11

11outxcriterion  10Scope: privatex01  KEYWORD

11Description: Criterion to select CarpetIOHDF5 output intervals, overrides outxevery

11 Range 10 x01 Default: default
11default  10Use ’IO::outxcriterion’
11never  10Never output
11iteration  10Output every so many iterations
11divisor  10Output if (iteration % outxevery) == 0.
11time  10Output every that much coordinate time

11outxdir  10Scope: privatex01  STRING

11Description: Name of CarpetIOHDF5 output directory, overrides ’IO::outxdir’

11 Range 10 x01 Default: (none)
11ˆ$  10Empty: use IO::outxdir
11.+  10Not empty: directory name

11

11outxdt  10Scope: privatex01  REAL

11Description: How often to do CarpetIOHDF5 output, overrides ’IO::outxdt’

11 Range 10 x01 Default: -2
11(0:*  10In intervals of that much coordinate time
11                           10As often as possible
11-1  10Disable output
11-2  10Default to ’IO::outxdt’

11

11outxevery  10Scope: privatex01  INT

11Description: How often to do CarpetIOHDF5 output, overrides outxevery

11 Range 10 x01 Default: -2
111:*  10Output every so many time steps
11-1:0  10No output
11-2  10Use ’IO::outxevery’

11

11outxextension  10Scope: privatex01  STRING

11Description: File extension to use for CarpetIOHDF5 output

11 Range 10 x01 Default: .h5
11                           10File extension (including a leading dot, if desired)

11

11outxvars  10Scope: privatex01  STRING

11Description: Variables to output in CarpetIOHDF5 file format

11 Range 10 x01 Default: (none)
11                           10List of group or variable names

11

11outputxallxtimelevels  10Scope: privatex01  BOOLEAN

11Description: Output all timelevels instead of only the current

11                           10  x01  Default: no

11

11outputxboundaryxpoints  10Scope: privatex01  BOOLEAN

11Description: Output outer boundary points (assuming that there are nghostzones boundary points)

11                           10  x01  Default: yes

11

11outputxbufferxpoints  10Scope: privatex01  BOOLEAN

11Description: Output refinement buffer points

11                           10  x01  Default: yes

11

11outputxghostxpoints  10Scope: privatex01  BOOLEAN

11Description: Output ghost points

11                           10  x01  Default: yes

11

11outputxindex  10Scope: privatex01  BOOLEAN

11Description: Output an index file for each output file

11                           10  x01  Default: no

11

11outputxsymmetryxpoints  10Scope: privatex01  BOOLEAN

11Description: Output symmetry points (assuming that there are nghostzones symmetry points)

11                           10  x01  Default: yes

11skipxrecoverxvariables  10Scope: privatex01  STRING

11Description: Skip these variables while recovering

11  Range  10  x01  Default: (none)

11                           10

11

11usexchecksums  10Scope: privatex01  BOOLEAN

11Description: Use checksums for the HDF5 data

11                           10  x01  Default: no

11

11usexgridxstructurexfromxcheckpoint 10Scope: privatex01  BOOLEAN

11Description: Use the grid structure stored in the checkpoint file

11                           10  x01  Default: yes

11

11usexreflevelsxfromxcheckpoint 10Scope: privatex01  BOOLEAN

11Description: Use ’CarpetRegrid::refinementxlevels’ from the checkpoint file rather than from the parameter file ?

11                           10  x01  Default: no

11

11abortxonxioxerrors  10Scope: shared from IOx01BOOLEAN

11

11checkpointxdir  10Scope: shared from IOx01STRING

11

11checkpointxevery  10Scope: shared from IOx01INT

11

11checkpointxeveryxwalltimexhours 10Scope: shared from IOx01REAL

11

11checkpointxfile  10Scope: shared from IOx01STRING

11

11checkpointxid  10Scope: shared from IOx01BOOLEAN

11

11checkpointxidxfile  10Scope: shared from IOx01STRING

11

11checkpointxkeep  10Scope: shared from IOx01INT

11

11checkpointxonxterminate  10Scope: shared from IOx01BOOLEAN

11

11filereaderxidxdir  10Scope: shared from IOx01STRING

11

11ioxoutxcriterion  10Scope: shared from IOx01KEYWORD

11ioxoutxdir  10Scope: shared from IOx01STRING

11

11ioxoutxdt  10Scope: shared from IOx01REAL

11

11ioxoutxevery  10Scope: shared from IOx01INT

11

11ioxoutxunchunked  10Scope: shared from IOx01BOOLEAN

11

11outxgroupxseparator  10Scope: shared from IOx01STRING

11

11outxmode  10Scope: shared from IOx01KEYWORD

11

11outxsavexparameters  10Scope: shared from IOx01KEYWORD

11

11outxsinglexprecision  10Scope: shared from IOx01BOOLEAN

11

11outxtimestepsxperxfile  10Scope: shared from IOx01INT

11

11outxxlinexy  10Scope: shared from IOx01REAL

11

11outxxlinexyi  10Scope: shared from IOx01INT

11

11outxxlinexz  10Scope: shared from IOx01REAL

11

11outxxlinexzi  10Scope: shared from IOx01INT

11

11outxxyplanexz  10Scope: shared from IOx01REAL

11

11outxxyplanexzi  10Scope: shared from IOx01INT

11

11outxxzplanexy  10Scope: shared from IOx01REAL

11

11outxxzplanexyi  10Scope: shared from IOx01INT

11outxylinexx  10Scope: shared from IOx01REAL

11

11outxylinexxi  10Scope: shared from IOx01INT

11

11outxylinexz  10Scope: shared from IOx01REAL

11

11outxylinexzi  10Scope: shared from IOx01INT

11

11outxyzplanexx  10Scope: shared from IOx01REAL

11

11outxyzplanexxi  10Scope: shared from IOx01INT

11

11outxzlinexx  10Scope: shared from IOx01REAL

11

11outxzlinexxi  10Scope: shared from IOx01INT

11

11outxzlinexy  10Scope: shared from IOx01REAL

11

11outxzlinexyi  10Scope: shared from IOx01INT

11

11recover  10Scope: shared from IOx01KEYWORD

11

11recoverxandxremove  10Scope: shared from IOx01BOOLEAN

11

11recoverxdir  10Scope: shared from IOx01STRING

11

11recoverxfile  10Scope: shared from IOx01STRING

11

11strictxioxparameterxcheck  10Scope: shared from IOx01BOOLEAN

11

11verbose  10Scope: shared from IOx01KEYWORD

9 Interfaces

General Implements: iohdf5

Grid Variables

9.0.1 PRIVATE GROUPS

11 x Group Names x 10 x Variable Names x x01 Details x11x
11 nextxoutputxiteration 10 nextxoutputxiteration x01  compact110
11                      10  x01 dimensions110
11                      10  x01 distribution11CONSTANT
11                      10  x01 group type11SCALAR
11                      10  x01 timelevels111
11                      10  x01variable type11INT
11 nextxoutputxtime 10 nextxoutputxtime x01  compact110
11                      10  x01 dimensions110
11                      10  x01 distribution11CONSTANT
11                      10  x01 group type11SCALAR
11                      10  x01 timelevels111
11                      10  x01variable type11REAL
11 thisxiteration 10 thisxiteration x01 compact110
11                      10  x01 dimensions110
11                      10  x01 distribution11CONSTANT
11                      10  x01 group type11SCALAR
11                      10  x01 timelevels111
11                      10  x01variable type11INT
11lastxoutputxiterationxslice10lastxoutputxiterationxslicex01  compact110
11                      10  x01 dimensions110
11                      10  x01 distribution11CONSTANT
11                      10  x01 group type11SCALAR
11                      10  x01 timelevels111
11                      10  x01vararrayxsize114
11                      10  x01variable type11INT
11 lastxoutputxtimexslice 10 lastxoutputxtimexslice x01  compact110
11                      10  x01 dimensions110
11                      10  x01 distribution11CONSTANT
11                      10  x01 group type11SCALAR
11                      10  x01 timelevels111
11                      10  x01vararrayxsize114
11                      10  x01variable type11REAL
11 thisxiterationxslice 10 thisxiterationxslice x01  compact110
11                      10  x01 dimensions110
11                      10  x01 distribution11CONSTANT
11                      10  x01 group type11SCALAR
11                      10  x01 timelevels111
11                      10  x01vararrayxsize114
11                      10  x01variable type11INT

11

Adds header: CarpetIOHDF5.hh

Uses header: Timer.hh carpet.hh defs.hh bbox.hh vect.hh data.hh gdata.hh ggf.hh gh.hh typecase.hh typeprops.hh mpixstring.hh

Provides: IOxSetCheckpointGroups to 10 Schedule This section lists all the variables which are assigned storage by thorn Carpet/CarpetIOHDF5. Storage can either last for the duration of the run (Always means that if this thorn is activated storage will be assigned, Conditional means that if this thorn is activated storage will be assigned for the duration of the run if some condition is met), or can be turned on for the duration of a schedule function.

Storage x

11Always:  11x
11nextxoutputxiteration nextxoutputxtime thisxiteration  11x
11lastxoutputxiterationxslice lastxoutputxtimexslice thisxiterationxslice11x

11x  11x

11

Scheduled Functions

CCTKxSTARTUP x carpetiohdf5xstartup x startup routine x

11x11After: 11ioutilxstartup
11 x 11Language:11c
11 x 11Type: 11function

11

CCTKxINITIAL x carpetiohdf5xinit x initialisation routine x

11x11Language:11c
11 x 11Options: 11global
11 x 11Type: 11function
11 x 11Writes: 11carpetiohdf5::thisxiterationxslice
11 x 11x  11lastxoutputxiterationxslice
11 x 11x  11lastxoutputxtimexslice
11 x 11x  11carpetiohdf5::thisxiteration
11 x 11x  11nextxoutputxiteration
11 x 11x  11nextxoutputxtime

11

CCTKxPOSTxRECOVERxVARIABLES x carpetiohdf5xinitcheckpointingintervals x initialisation of checkpointing intervals after recovery

11x11Language:11c
11 x 11Options: 11global
11 x 11Type: 11function

11

CCTKxCPINITIAL x carpetiohdf5xinitialdatacheckpoint x initial data checkpoint routine x

11x11Language:11c
11 x 11Options: 11meta
11 x 11Type: 11function

11

CCTKxCHECKPOINT x carpetiohdf5xevolutioncheckpoint x evolution checkpoint routine x

11x11Language:11c
11 x 11Options: 11meta
11 x 11Type: 11function

11

CCTKxTERMINATE x carpetiohdf5xterminationcheckpoint x termination checkpoint routine x

11x11Language:11c
11 x 11Options: 11meta
11 x 11Type: 11function

11

CCTKxPOSTINITIAL x carpetiohdf5xclosefiles x close all filereader input files x

11x11Language:11c
11 x 11Options: 11global
11 x 11Type: 11function

11

CCTKxRECOVERxPARAMETERS (conditional) x carpetiohdf5xrecoverparameters x parameter recovery routine x

11x11Language:11c
11 x 11Options: 11meta
11 x 11Type: 11function

11

CCTKxSTARTUP (conditional) x carpetiohdf5xsetnumrefinementlevels x overwrite ’carpetregrid::refinementxlevels’ with the number of levels found in the checkpoint file x

11x11Before: 11carpetiohdf5xstartup
11 x 11Language:11c
11 x 11Options: 11meta
11 x 11Type: 11function

11

CCTKxPOSTxRECOVERxVARIABLES (conditional) x carpetiohdf5xclosefiles x close all initial data checkpoint files after recovery x

11x11Language:11c
11 x 11Options: 11meta
11 x 11Type: 11function