xx IOUtil

IOUtil

Thomas Radke, Gabrielle Allen

Date

1 Introduction

This document details the thorns provided in the standard Cactus distribution for the output of grid variables, and describes how to set parameters for I/O and checkpointing/recovery using these thorns.

Input and output of data (I/O) in Cactus is provided by infrastructure thorns, which interact with the flesh via a fixed interface, which is described in the Users’ Guide. The standard release of Cactus contains a number of thorns which provide so-called I/O methods implementing the actual I/O in a variety of data formats and styles. All these provided I/O methods use thorn IOUtil which provides general utilities for I/O (such as parsing parameter strings to decide which variables to output), and a general set of parameters which are inherited by the different I/O methods (such as the output directory). Thorn IOUtil by itself provides no I/O methods.

More information about I/O and visualisation of Cactus data can be found in the individual I/O thorns, and in the Visualization-HOWTO available on the Cactus web pages.

2 I/O Methods in Cactus

Cactus has several I/O methods for the output of grid variables in different data formats and styles. Each I/O method comes with its own parameters by which it can be customised, and all methods are registered with the flesh, satisfying the Cactus API, allowing them to be called directly from application thorns. An I/O method registers itself with the flesh along with it’s name, and these registered names are the labels we now use to describe the various methods.

11I/O method11Description  11Providing Thorn
11Scalar      11output of scalars or grid array reductions in xgraph or gnuplot format11CactusBase/IOBasic
11Info        11screen output of scalars or grid array reductions  11CactusBase/IOBasic

11            11                                                          11

11IOASCIIx1D  111D line output of grid arrays in xgraph or gnuplot format  11CactusBase/IOASCII
11IOASCIIx2D  112D slice output of grid arrays in gnuplot format  11CactusBase/IOASCII
11IOASCIIx3D  11full output of 3D grid arrays in gnuplot format  11CactusBase/IOASCII

11            11                                                          11

11IOJpeg      112D slice output of grid arrays in jpeg image format  11CactusIO/IOJpeg

11            11                                                          11

11IOHDF5      11full output of arbitrary grid variables in HDF5 format  11CactusPUGHIO/IOHDF5

11            11                                                          11

11IOFlexIOx2D 112D slice output of grid arrays in FlexIO format  11CactusPUGHIO/IOFlexIO
11IOFlexIO    11full output of arbitrary grid variables in FlexIO format  11CactusPUGHIO/IOFlexIO

11

Tablex1: Standard I/O methods provided with the Cactus distribution The standard provided Cactus I/O methods are shown in Tablex1. As described above, each of these I/O thorns inherit parameters from thorn IOUtil, which must be included in your ThornList and activated in your parameter files before any of these I/O methods can be used. IOUtil allows you to set the default behaviour for all the I/O methods described above, for example, setting the parameter IO::outxevery = 1 will result in any chosen I/O method providing output on each iteration. The default behaviour can be overridden by specific parameters for each method. For example, you may want scalar and 1D output at every iteration but computationally expensive 3D output only every 10th iteration, with these files going into another directory on a scratch partition.

3 Providing Your Own I/O Method

If you as a Cactus developer have your own input/output routines and want to share this functionality with other people you should do this by putting them into an new I/O thorn and register them as an I/O method. A description on how to register a new I/O method with the flesh’s I/O subsystem can be found in the Infrastructure Thorn Writer’s Guide (as part of the Cactus User’s Guide).

New I/O thorns should always inherit from thorn IOUtil in order to reuse as much of the existing I/O infrastructure as possible, and to maintain a uniform interface on how to use the I/O methods.

4 Standard Parameters

Here we describe a few of the standard parameters used by IOUtil to control output.

     IO::outxdir

The name of the directory to be used for output. All the I/O methods described here will write by default to this directory (which itself defaults to the current working directory). Individual methods have parameters which can direct their output to a different directory.

     IO::outxcriterion

The criterion that decides when to output. The default is to output every so many iterations (see

     IO::outxevery).

IO::outxevery

How often, in terms of iterations, each of the Cactus I/O methods will write output. Again, individual methods can set their own parameters to override this. The default is to never write output.

IO::outxdt

How often, in terms of simulation time, each of the Cactus I/O methods will write output. Again, individual methods can set their own parameters to override this. The default is to never write output.

5 Saving/Generating Parameter Files

Thorn IOUtil can save a copy of the parameter file of a run, or can also automatically generate a parameter file from all current parameter settings. This is controlled by the IO::parfilexwrite parameter:

IO::parfilexwrite="copy"

This is the default option, and makes an exact replica of the input parameter file in the standard output directory (this is particularly useful when the output directory is going to be archived).

IO::parfilexwrite="generate"

Generate a new parameter file from runtime information, containing the Cactus version, the name of the original parameter file, the run time/date, the host to run on, and the number of processors – all on comment lines. Following this the parameter file contains the ActiveThorns list plus a sorted list of all active thorns’ parameters which have been set in the original parameter file.

IO::parfilexwrite="no"

Switch off writing of a new parameter file.

The name of the new parameter file defaults to the original filename, unless the parameter IO::parfilexname is set. Note that an already existing file with the chosen name will be overwritten unless it is identical with the original parameter file, or if Cactus was recovered from a checkpoint (in which case you don’t want to overwrite an existing parameter file with your recovery parameter file).

6 I/O Modes

For a run on multiple processors, scalar, 1D, and 2D output will always be written from only processor zero (that is, required data from all other processors will be sent to processor zero, which then outputs all the gathered data). For full-dimensional output of grid arrays this may become a quite expensive operation since output by only a single processor will probably result in an I/O bottleneck and delay further computation. For this reason Cactus offers different I/O modes for such output which can be controlled by the IO::outxmode parameter, in combination with IO::outxunchunked and IO::outxprocxevery. These parameters allow I/O to be optimised for your particular machine architecture and needs:

IO::outxmode = "onefile"

As for the 1D and 2D I/O methods, writing to file is performed only by processor zero. This processor gathers all the output data from the other processors and then writes to a single file. The gathered grid array data from each processor can be either written in chunks (IO::outxunchunked = "no") with each chunk containing the data from a single processor, or collected into a single global array before writing (IO::outxunchunked = "yes"). The default is to write the data in chunks. This can be changed by adding an option string to the group/variable name(s) in the outxvars parameter with the key outxunchunked and an associated string value "yes|no|true|false".

IO::outxmode = "np"

Output is written in parallel for groups of processors. Each group consists of IO::outxprocxevery processors which have assigned one I/O processor which gathers data from the group and writes it to file. The chunked output will go into IO::outxprocxevery files. The default number of processors in a group is eight.

IO::outxmode = "proc"

This is the default output mode. Every processor writes its own chunk of data into a separate output file.

Probably the single-processor "proc" mode is the most efficient output mode on machines with a fast I/O subsystem and many I/O nodes (e.g. a Linux cluster with local disks attached to each node) because it provides the highest parallelity for outputting data. Note that on very large numbers of processors you may have to fall back to "np", doing output by every so many processors, mode if the system limit of maximum open file descriptors is exceeded (this is true for large jobs on a T3E).

While the "np" and "proc" I/O modes are fast for outputting large amounts of data from all or a group of processors in parallel, they have the disadvantage of writing chunked files. These files then have to be recombined during a postprocessing phase so that the final unchunked data can be visualized by standard tools. For that purpose a recombiner utility program is provided by the thorns offering parallel I/O methods.

7 Output of Hyperslab Data

While some I/O methods (IOHDF5, IOFlexIO) can dump the full contents of a multidimensional CCTK variable, others such as IOASCIIx1D and IOASCIIx2D will output only a subset of the data (e.g. 1D lines or 2D planes of 3D grid functions). Such a subset (called a hyperslab) is generally defined as an orthogonal region into the multidimensional dataset, with a start point and a length in any direction, and an optional downsampling factor.

Thorn IOUtil defines a set of hyperslab parameters for all I/O methods which determine the default positions of 1D line or 2D slice output along the axes. I/O thorns can also define their own hyperslab parameters which then will overwrite the defaults provided by IOUtil.

     IO::outx[xyz]linex[xyz]

specifies the slice center for 1D x,y,z-line output by coordinate values of the underlying physical grid

     IO::outx[xyz]linex[xyz]i

specifies the slice center of 1D x,y,z-line output by index points of the underlying computational grid

     IO::outx[{xy}{xz}{yz}]planex[xyz]

specifies the slice center of 2D xy,xz,yz-plane output by coordinate values of the underlying physical grid

IO::outx[{xy}{xz}{yz}]planex[xyz]i

specifies the slice center of 2D xy,xz,yz-plane output by index points of the underlying computational grid

IO::outxdownsamplex[xyz]

specifies the downsampling factor for output in every direction

Setting the index points for the slice centers in a parameter file has precedence over setting their location by coordinate values. If nothing was specified the default values of IO::outx[xyz]linex[xyz] and IO::outx[{xy}{xz}{yz}]planex[xyz] will be used. These are set to be all zeros which causes the output to go through the coordinate system’s origin or the closest grid point to this (see figure 1 for an example).

PIC PIC Figurex1: Default 1D x,y,z-line output for a 3D grid in box mode (left) and xy-bitant mode (right)

If the coordinate values specified in IO::outx[xyz]linex[xyz] or IO::outx[{xy}{xz}{yz}]planex[xyz] lie

outside the physical grid, the slice center simply reverts to the center of the box in that direction. This fallback method method can be changed to using 0-slices instead by setting the corresponding

IO::outx[xyz]linex[xyz]i and IO::outx[{xy}{xz}{yz}]planex[xyz]i parameter(s) to the value
-2.

Although they are not hyperslabs by the above definition, output of 1D diagonals for 3D grid arrays is also supported by I/O method IOASCIIx1D but has the restriction that the line will always start in the bottom-left corner of the computational grid and steadily rise by one grid point in every direction (see figure 2 for an example).

PIC PIC Figurex2: 1D diagonal output for a 3D cubical (left) and non-cubical (right) computational grid 8 Data Filenames

The standard I/O thorns in Cactus make use of a consistent set of filenames and extensions, which identify the variables and data format used in the file. The filenames are listed in the following table.

11I/O method11Filename for output of variable var
11Info        11only outputs to screen

11Scalar      11<var>.{asc|xg} for scalar variables

11            11<var>x<reduction>.{asc|xg} for reduction values from grid arrays

11IOASCIIx1D  11<var>x<slice>x[<centerxi>][centerxj>].{asc|xg}
11IOASCIIx2D  11<var>x<plane>x[<center>].asc
11IOASCIIx3D  11<var>x3D.asc
11IOJpeg      11<var>x<plane>x[<center>].jpeg
11IOHDF5      11<var>x3D.h5
11IOFlexIOx2D 11<var>x2D.ieee
11IOFlexIO    11<var>x3D.ieee

11

Tablex2: Filenames used by standard I/O thorns 9 Checkpointing and Recovery in Cactus

The I/O methods for arbitrary output of CCTK variables also provide functionality for checkpointing and recovery. A checkpoint is a snapshot of the current state of the simulation (i.e. the contents of all the grid variables and the parameter settings) at a chosen timestep. Each checkpoint is saved into a checkpoint file which can be used to restart a new simulation at a later time, recreating the exact state at which it was checkpointed.

Checkpointing is especially useful when running Cactus in batch queue systems where jobs get only limited CPU time. A more advanced use of checkpointing would be to restart your simulation after a crash or a problem had developed, using a different parameter set recovering from the latest stable timestep. Additionally, for performing parameter studies, compute-intensive initial data can be calculated just once and saved in a checkpoint file from which each job can be started.

Again, thorn IOUtil provides general checkpoint & recovery parameters. The most important ones are:

     IO::checkpointxevery (steerable)

specifies how often to write a evolution checkpoint in terms of iteration number.

     IO::checkpointxeveryxwalltimexhours (steerable)

specifies how often to write a evolution checkpoint in terms of wall time. Checkpointing will be triggered if either of these conditions is met.

     IO::checkpointxnext (steerable)

triggers a checkpoint at the end of the current iteration. This flag will be reset afterwards.

     IO::checkpointxID

triggers a checkpoint of initial data

     IO::checkpointxonxterminate (steerable)

triggers a checkpoint at the end of the last iteration of a simulation run

IO::checkpointxfile (steerable)

holds the basename for evolution checkpoint file(s) to create Iteration number and file extension are appended by the individual I/O method used to write the checkpoint.

IO::checkpointxIDxfile (steerable)

holds the basename for initial data checkpoint file(s) to create Iteration number and file extension are appended by the individual I/O method used to write the checkpoint.

IO::checkpointxdir

names the directory where checkpoint files are stored

IO::checkpointxkeep (steerable)

specifies how many evolution checkpoints should be kept The default value of means that only the latest evolution checkpoint is kept and older checkpoints are removed in order to save disk space. Setting IO::checkpointxkeep to a positive value will keep so many evolution checkpoints around. A value of will keep all (future) checkpoints.

IO::recoverxandxremove

determines whether the checkpoint file that the current simulation has been successfully recovered from, should also be subject of removal, according to the setting of IO::checkpointxkeep

IO::recover

keyword parameter telling if/how to recover.

Choices are "no", "manual", "auto", and "autoprobe".

IO::recoverxfile

basename of the recovery file Iteration number and file extension are appended by the individual I/O method used to recover from the recovery file.

IO::recoverxdir

directory where the recovery file is located

IO::truncatexfilesxafterxrecovering

whether or not to truncate already existing output files after recovering To checkpoint your simulation, you need to enable checkpointing by setting the boolean parameter checkpoint, for one of the appropriate I/O methods to yes. Checkpoint filenames consist of a basename (as specified in

IO::checkpointxfile) followed by ".chkpt.itxiterationxnumber" plus the file extension indicating the file

format ("*.ieee" for IEEEIO data from CactusPUGHIO/IOFlexIO, or "*.h5" for HDF5 data from

CactusPUGHIO/IOHDF5).

Use the "manual" mode to recover from a specific checkpoint file by adding the iteration number to the basename parameter.

The "auto" recovery mode will automatically recover from the latest checkpoint file found in the recovery directory. In this case IO::recoverxfile should contain the basename only (without any iteration number).

The "autoprobe" recovery mode is similar to the "auto" mode except that it would not stop the code if no checkpoint file was found but only print a warning message and then continue with the simulation. This mode allows you to enable checkpointing and recovery in the same parameter file and use that without any changes to restart your simulation. On the other hand, you are responsible now for making the checkpoint/recovery directory/file parameters match — a mismatch will not be detected by Cactus in order to terminate it. Instead the simulation would always start from initial data without any recovery.

Because the same I/O methods implement both output of arbitrary data and checkpoint files, the same I/O modes are used (see Sectionx6). Note that the recovery routines in Cactus can process both chunked and unchunked checkpoint files if you restart on the same number of processors — no recombination is needed here. That’s why you should always use one of the parallel I/O modes for checkpointing. If you want to restart on a different number of processors, you first need to recombine the data in the checkpoint file(s) to create a single file with unchunked data. Note that Cactus checkpoint files are platform independent so you can restart from your checkpoint file on a different machine/architecture.

By default, existing output files will be appended to rather than truncated after successful recovery. If you don’t want this, you can force I/O methods to always truncate existing output files. Thorn IOUtil provides an aliased function for other I/O thorns to call:

xxCCTK_INTxFUNCTIONxIO_TruncateOutputFilesx(CCTK_POINTER_TO_CONSTxINxcctkGH)

This function simply returns 1 or 0 if output files should or should not be truncated.

WARNING:

Checkpointing and recovery should always be tested for a new thorn set. This is because only Cactus grid variables and parameters are saved in a checkpoint file. If a thorn has made use of saved local variables, the state of a recovered simulation may differ from the original run. To test checkpointing and recovery, simply perform one run of say 10 timesteps, and compare output data with a checkpointed and recovered run at say the 5th timestep. The output data should match exactly if recovery was successful.

10 Reading Data from Files into Cactus

The very same routines which implement checkpointing/recovery functionality in the IOHDF5 and IOFlexIO thorns are also used to provide file reader capabilities within Cactus. They enable users to read variables, whose contents were written to files in HDF5 or IEEEIO data format, back into Cactus at a later time. This is especially useful if compute-intensive initial data is calculated only once and stored in a file. Such data can then be read back in at startup and immediately used by following evolution runs.

The following IOUtil parameters exist to specify what variables should be read from file(s) as initial data:

     IO::filereaderxIDxdir

root directory for files to be read

     IO::filereaderxIDxfiles

list of files to read in as initial data (multiple filenames must be separated by spaces)

The same file naming conventions (what I/O mode used, which iteration number) apply as for checkpoint files.

IO::filereaderxIDxvars

list of CCTK variables to read in from the given initial data files (variables are identified by their full name, multiple variable names must be separated by spaces) This is useful if a datafile contains multiple variables but only some of them should be read. Thus it is possible to recover distinguished variables from a full checkpoint file. Note that if the file contains several timesteps of the same variable only the last one is taken by default. This can be changed by adding an option string with the key cctkxiteration and an associated integer scalar value to the variable name denoting the iteration number to

choose, like in IO::filereaderxIDxvars = "wavetoy::phi{ cctkxiteration = 10 }". The file

reader supports a further option alias and an associated string scalar value which can be used to set the dataset in the initial data files that will be read into this variable. For example

IO::filereaderxIDxvars = "wavetoy::phi{ alias='wavetoy::psi' }" will read the data of

variable wavetoy::psi from file into wavetoy::phi. This option requires that explicit variable names are used, group names are not supported for the alias option.

Thorn IOUtil also provides a filereader API which can be called by any application thorn at any time. It gets passed the equivalent information to the filereader parameters, plus a pointer to the underlying CCTK grid hierarchy. The return code denotes the total number of variables recovered by the filereader.

C API:

xx#includex"CactusBase/IOUtil/src/ioutil_CheckpointRecovery.h"

xxintxIOUtil_RecoverVarsFromDatafilesx(cGHx*GH,
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxconstxcharx*in_files,
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxconstxcharx*in_vars);

Fortran API:

xxcallxIOUtil_RecoverVarsFromDatafilesx(result,xGH,xin_files,xin_vars)
xxxxintegerxxxxxxxresult
xxxxCCTK_POINTERxxGH
xxxxcharacter*(*)xin_files
xxxxcharacter*(*)xin_vars

If data is to be imported from files which were not created by IOHDF5 or IOFlexIO it needs to be converted first into the appropriate HDF5 or IEEEIO file format and the file layout which either one of these thorns uses. This is described in detail in the thorns’ documentation, along with a simple C source file which can be used as a template to build your own data converter program.

11 Example Parameter Files

Here we give examples of the parameters for the different I/O methods.

Output information to screen using IOBasic's "Info" I/O method

xxActiveThornsx=x"IOBasicxIOUtilxPUGHReducex..."
xx#xOutputxusingxallxmethodsxonxiterationx0,x10,x20,x...
xxIO::out_everyx=x10
xx#xGroupxofxvariablesxtoxoutputxtoxscreen
xxIOBasic::outInfo_varsx=x"evolve::vars"

Scalar Output from IOBasic's "Scalar" I/O method

xxActiveThornsx=x"IOBasicxIOUtilxPUGHReducex..."
xx#xOutputxvarsxusingxscalarxmethodxonxiterationx0,x10,x20,x...
xxIOBasic::outScalar_everyx=x10
xx#xGroupxofxvariablesxtoxoutputxtoxfile
xxIOBasic::outScalar_varsx=x"evolve::vars"

ASCII 1D and 2D Output with IOASCII's "IOASCIIx1D" and "IOASCIIx2D" I/O

methods

xxActiveThornsx=x"IOASCIIxIOUtilxPUGHSlabx..."
xx#xOutputxvarsxinx1Dxonxiterationx0,x10,x20,x...
xxIOASCII::out1D_everyx=x10
xx#xOutputxvarsxinx2Dxonxiterationx0,x50,x100,x...
xxIOASCII::out2D_everyx=x50

HDF5 Output with IOHDF5's "IOHDF5" I/O method

xxActiveThornsx=x"IOHDF5xIOUtilxPUGHSlabx..."
xx#xOutputxvarsxinxHDF5xformatxonxiterationx0,x5,x10,x...
xxIOHDF5::out_everyx=x5
xx#xGroupxofxvariablesxtoxoutput
xxIOHDF5::out_varsx=x"evolve::vars"
xx#xSpecialxI/OxdirectoryxforxHDF5xoutput
xxIOHDF5::out_dirx=x"/scratch/tmp"
xx#xFullxoutputxunchunkedxtoxonexfile
xx#x(Onlyxusingxaxsmallxnumberxofxprocessors)
xxIO::out_modexxxxxx=x"onefile"
xxIO::out_unchunkedx=x"yes"
xx#xDownsamplexfullxdataxbyxaxfactorxofx3xinxeachxdirection
xxIO::out_downsample_xx=x3
xxIO::out_downsample_yx=x3
xxIO::out_downsample_zx=x3

IEEEIO 2D hyperslab and full output using IOFlexIO's "IOFlexIOx2D" and "IOFlexIO" I/O methods

xxActiveThornsx=x"IOFlexIOxFlexIOxIOUtilxPUGHSlabx..."
xx#xOutputxvarsxinx2DxIEEEIOxformatxonxiterationx0,x100,x200,x...
xxIOFlexIO::out2D_everyx=x100
xx#xOutputxvarsxinxIEEEIOxformatxonxiterationx0,x5,x10,x...
xxIOFlexIO::out_everyx=x5
xx#xGroupxofxvariablesxtoxoutputxtoxfilexforxeachxmethod
xxIOFlexIO::out2D_varsx=x"evolve::vars"
xxIOFlexIO::out_varsxxx=x"evolve::vars"
xx#x2DxoutputxgoesxintoxstandardxI/Oxdirectory
xxIO::out_dirx=x"test"
xx#xSpecialxI/Oxdirectoryxforxfullxoutput
xxIOFlexIO::out_dirx=x"/scratch/tmp"
xx#xFullxoutputxchunkedxtoxonexfilexforxeveryxeightxprocessors
xx#x(Runxonxlargexnumberxofxprocessors)
xxIO::out_modexxxxxxx=x"np"
xxIO::out_proc_everyx=x8
xxIO::out_unchunkedxx=x"no"
xx#xDownsamplexfullxdataxbyxaxfactorxofx3xinxeachxdirection
xxIO::out_downsample_xx=x3
xxIO::out_downsample_yx=x3
xxIO::out_downsample_zx=x3

Checkpointing using thorn IOFlexIO

xx#xUsexIEEEIOxdataxformatxforxcheckpointxfiles
xxIOFlexIO::checkpointx=x"yes"
xx#xMakexaxnewxcheckpointxfilexeveryx300xiterations
xxIO::checkpoint_everyx=x300
xx#xNamexandxdirectoryxofxcheckpointxfile
xxIO::checkpoint_filex=x"run5"
xxIO::checkpoint_dirxx=x"/scratch/tmp"

Recovering from a checkpoint file

xx#xautomaticallyxchoosexthexlatestxcheckpointxfile
xxIO::recoverx=x"auto"
xx#xNamexandxdirectoryxofxcheckpointxfilextoxrecoverxfrom
xxIO::recover_filex=x"run5"
xxIO::recover_dirxx=x"/scratch/tmp"

12 Providing Your own Checkpointing/Recovery Method

This section is for Cactus developers to describe how they can add a new method for checkpointing/recovery using the existing I/O parameters and the function API of thorn IOUtil. Inheriting this functionality from thorn IOUtil helps you to reuse existing code and maintain a uniform user interface on how to invoke different checkpointing/recovery methods.

12.1 Adding a Checkpointing Method

Checkpointing is similar to performing full output of an individual grid variable, except that

the output is done for all grid variables existing in a grid hierarchy in addition to the contents of all variables, also the current setting of all parameters is saved as well as some other information necessary for recovering from the checkpoint at a later time

A thorn routine providing this checkpointing capability should register itself with the flesh’s scheduler at the CPINITIAL (for initial data checkpoints), CHECKPOINT (for periodic checkpoints of evolution data), and TERMINATE time bins (for checkpointing the last timestep of a simulation).

It should also decide whether checkpointing is needed by evaluating the corresponding checkpoint parameters of IOUtil (see section 9).

Before dumping the contents of a distributed grid array into a checkpoint file the variable should be synchronized in case synchronization was not done before implicitly by the scheduler.

To gather the current parameter values you can use the C routine

xxcharx*IOUtil_GetAllParametersx(constxcGHx*GH,xintxall);

from thorn IOUtil. This routine returns the parameter settings in an allocated single large string. Its second argument all flags whether all parameter settings should be gathered () or just the ones which have been set before (). Note that you should always save all parameters in a checkpoint in order to reproduce the same state after recovery.

As additional data necessary for proper recovery, the following information must be saved in a checkpoint file:

the current main loop index (used by the driver as the main evolution loop index)

the current CCTK iteration number (GH->cctkxiteration)

the physical simulation time (GH->cctkxtime)

Moreover, information about the I/O mode used to create the checkpoint (chunked/unchunked, serial versus parallel I/O), the active thorns list, or this run’s Cactus version ID (for compatibility checking at recovery time) could be relevant to save in a checkpoint.

12.2 Adding a Recovery Method

Recovering from a checkpoint is a two step operation:

Right after reading the parameter file and finding out if recovery was requested, all the parameters are restored from the checkpoint file (overwriting any previous settings for non-steerable parameters).

After the flesh has created the grid hierarchy with all containing grid variables and the driver has set up storage for these, their contents is restored from the checkpoint (overwriting any previously initialized contents).

The flesh provides the special time bins RECOVERxPARAMETERS and RECOVERxVARIABLES for these two steps (see also the chapter on Adding a Checkpointing/Recovery Method in the Infrastructure Thorn Writer’s Guide as part of the Cactus User’s Guide).

Thorn IOUtil evaluates the recovery parameters (determines the recovery mode to use, construct the name(s) of the recovery file(s) etc.). It also controls the recovery process by invoking the recovery methods of other I/O thorns, one after another until one method succeeded.

A recovery method must provide a routine with the following prototype:

xxintxRecoverx(cGHx*GH,xconstxcharx*basefilename,xintxcalled_from);

This routine will be invoked by IOUtil with the following arguments:

a GH pointer refer to the grid hierarchy and its grid variables Note that this can also be a NULL pointer if the routine was called at CPxRECOVERxPARAMETERS when no grid hierarchy exists yet.

the basename of the checkpoint file to recover from This name is constructed by IOUtil from the settings of IO::recoveryxdir and IO::recoverxfile. It will also include an iteration number if one of the auto recovery modes is used. The filename extension by which checkpoint files of different recovery methods are distinguished must be appended explicitly.

a flag identifying when the routine was called This

can be one of the keywords CPxINITIALxDATA, CPxEVOLUTIONxDATA, CPxRECOVERxPARAMETERS,

CPxRECOVERxDATA, or FILEREADERxDATA). This flag tells the routine what it should do when it is being called by IOUtil. Note that IOUtil assumes that the recovery method can also be used as a filereader routine here which is essentially the same as recovering (individual) grid variables from (data) files.

To perform the first step of recovery process, a recovery method must register a routine with the flesh’s scheduler at the RECOVERxPARAMETERS time bin. This routine itself should call the IOUtil API

xxintxIOUtil_RecoverParametersx(intx(*recover_fn)x(cGHx*GH,
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxconstxcharx*basefilename,
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxintxcalled_from),
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxconstxcharx*file_extension,
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxconstxcharx*file_type)

which will determine the recovery filename(s) and in turn invoke the actual recovery method’s routine as a callback function. The arguments to pass to this routine are

a function pointer recoverxfn as a reference to the recovery method’s actual recovery routine (as described above)

filexextension – the filename extension for recovery files which are accepted by this recovery method When IOUtil constructs the recovery filename and searches for potential recovery files (in the auto recovery modes) it will only match filenames with the basename as given in the IO::recoveryxfile parameter, appended by filexextension.

filextype – the type of checkpoint files which are accepted by this recovery method This is just a descriptive string to print some info output during recovery.

The routine registered at RECOVERxPARAMETERS should return to the scheduler a negative value if parameter recovery failed for this recovery method for some reason (e.g. if no appropriate recovery file was found). The scheduler will then continue with the next recovery method until one finally succeeds (a positive value is returned). If none of the available recovery methods were successful the flesh would stop the code.

A value of zero should be returned to the scheduler to indicate that no recovery was requested.

The second step during recovery — restoring the contents of grid variables from the recovery file — is invoked by thorn IOUtil which registers a routine at RECOVERxVARIABLES. This routine calls all recovery methods which were registered before with IOUtil via the API

xxintxIOUtil_RegisterRecoverx(constxcharx*name,
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxintx(*recover_fn)x(cGHx*GH,
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxconstxcharx*basefilename,
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxintxcalled_from));

With this registration, all recovery method’s actual recovery routines are made known to IOUtil, along with a descriptive name under which they are registered.

At RECOVERxVARIABLES thorn IOUtil will loop over all available recovery routines (passing the same arguments as for parameter recovery) until one succeeds (returns a positive value).

13 Parameters

11truncatexfiles 10Scope: privatex01 BOOLEAN

11Description: Truncate existing output files from previous runs (except when recovering) ?

11                         10  x01  Default: yes

11

11truncatexfilesxafterxrecovering10Scope: privatex01  BOOLEAN

11Description: Truncate existing output files after recovering ?

11                         10  x01  Default: no

11

11abortxonxioxerrors  10Scope: restrictedx01  BOOLEAN

11Description: Abort on I/O errors (rather than just print a warning) ?

11                         10  x01  Default: no

11

11checkpointxdir  10Scope: restrictedx01  STRING

11Description: Output directory for checkpoint files

11 Range 10 x01 Default: (none)
11.+  10A valid directory name

11

11checkpointxevery  10Scope: restrictedx01  INT

11Description: How often to checkpoint

11 Range 10 x01 Default: -1
111:*  10Every so many iterations
11-1:0  10Disable periodic checkpointing

11checkpointxeveryxwalltimexhours 10Scope: restrictedx01  REAL

11Description: How often to checkpoint

11 Range 10 x01 Default: -1
11(0:*  10After so much walltime has passed
11-1  10Disable periodic walltime checkpointing

11

11checkpointxfile  10Scope: restrictedx01  STRING

11Description: File name for regular checkpoint

11 Range 10 x01 Default: checkpoint.chkpt
11.+  10A valid filename

11

11checkpointxid  10Scope: restrictedx01  BOOLEAN

11Description: Checkpoint initial data ?

11                         10  x01  Default: no

11

11checkpointxidxfile  10Scope: restrictedx01  STRING

11Description: File name for initial data checkpoint

11 Range 10 x01 Default: checkpoint.chkpt
11.+  10A valid filename

11

11checkpointxkeep  10Scope: restrictedx01  INT

11Description: How many checkpoint files to keep

11 Range 10 x01 Default: 1
111:*  101 overwrites the latest checkpoint file
11-1:  10Keep all checkpoint files

11

11checkpointxonxterminate  10Scope: restrictedx01  BOOLEAN

11Description: Checkpoint after last iteration

11                         10  x01  Default: no

11

11filereaderxidxdir  10Scope: restrictedx01  STRING

11Description: Directory to look for input files

11 Range 10 x01 Default: (none)
11.+  10A valid directory name

11

11filereaderxidxfiles  10Scope: restrictedx01  STRING

11Description: List of basenames of files to read in as initial data (e.g. omit the filename extention here)

11  Range  10  x01  Default: (none)
11.+  10Space-separated list of initial data filenames (basenames, e.g. excluding the file name extention)
11ˆ$  10An empty string for not recovering initial data

11

11filereaderxidxvars  10Scope: restrictedx01  STRING

11Description: List of variables to read in from the given initial data files

11 Range 10 x01 Default: all
11all  10Read all variables contained in the initial data files
11.+  10Space-separated list of fully qualified variable/group names
11ˆ$  10An empty string for not recovering initial data

11

11maxxentriesxperxsubdir  10Scope: restrictedx01  INT

11Description: Number of processes that can access the same directory

11 Range 10 x01 Default: (none)
11                         10unlimited
112:*  10at most that many processes

11newxfilenamexscheme  10Scope: restrictedx01  BOOLEAN

11Description: Use the new filename scheme for output files ?

11                         10  x01  Default: yes

11

11out3dxseptimefiles  10Scope: restrictedx01  BOOLEAN

11Description: Write one file per time slice, as opposed to all data in one file

11                         10  x01  Default: no

11

11outxcriterion  10Scope: restrictedx01  KEYWORD

11Description: Criterion to select output intervals

11 Range 10 x01 Default: iteration
11never  10Never output
11iteration  10Output every so many iterations
11time  10Output every that much coordinate time

11

11outxdir  10Scope: restrictedx01  STRING

11Description: Default output directory

11 Range 10 x01 Default: (none)
11.+  10A valid directory name

11

11outxdownsamplexx  10Scope: restrictedx01  INT

11Description: Factor by which to downsample output in x direction. Point (0,0,0) is always included.

11  Range  10  x01  Default: 1
111:*  10A positive integer

11

11outxdownsamplexy  10Scope: restrictedx01  INT

11Description: Factor by which to downsample output in y direction. Point (0,0,0) is always included.

11  Range  10  x01  Default: 1
111:*  10A positive integer

11

11outxdownsamplexz  10Scope: restrictedx01  INT

11Description: Factor by which to downsample output in z direction. Point (0,0,0) is always included.

11  Range  10  x01  Default: 1
111:*  10A positive integer

11

11outxdt  10Scope: restrictedx01  REAL

11Description: How often to do output by default

11 Range 10 x01 Default: -2
11(0:*  10In intervals of that much coordinate time
11                         10As often as possible
11-1  10Disable output
11-2  10Disable output

11

11outxevery  10Scope: restrictedx01  INT

11Description: How often to do output by default

11 Range 10 x01 Default: -1
111:*  10Every so many iterations
11-1:0  10Disable output

11outxfileinfo  10Scope: restrictedx01  KEYWORD

11Description: Add some useful file information to output files ?

11 Range 10 x01 Default: all
11none  10no file information
11creation date  10add creation date
11parameter filename  10add parameter filename
11axis labels  10add axis labels information to output files
11all  10add all available file information

11

11outxgroupxseparator  10Scope: restrictedx01  STRING

11Description: String to separate group name from variable name in file name

11 Range 10 x01 Default: (none)
11                         10Note: The old default was’::’

11

11outxmode  10Scope: restrictedx01  KEYWORD

11Description: Which mode to use for output

11 Range 10 x01 Default: proc
11proc  10Every processor writes its share of data into a separate file
11np  10Data is collected and written by every N’th processor into a separate file, where N is specified by the parameter IO::outxprocxevery
11onefile  10All output is written into a single file by processor 0

11

11outxprocxevery  10Scope: restrictedx01  INT

11Description: Do output on every N processors

11 Range 10 x01 Default: 8
111:*  10A number between [1, nprocs)

11

11outxsavexparameters  10Scope: restrictedx01  KEYWORD

11Description: Save current parameter settings in output files ?

11 Range 10 x01 Default: only set
11all  10Save all parameter settings
11only set  10Only save parameters which have been set before
11no  10Don’t save parameter settings

11

11outxsinglexprecision  10Scope: restrictedx01  BOOLEAN

11Description: Output data in single precision ?

11                         10  x01  Default: no

11

11outxtimestepsxperxfile  10Scope: restrictedx01  INT

11Description: How many timesteps to write to a single file

11 Range 10 x01 Default: -1
111:1  10Number of timesteps per file (can only be 1 so far)
11-1:  10All timesteps in a single file

11

11outxunchunked  10Scope: restrictedx01  BOOLEAN

11Description: Don’t write data in chunks. This parameter is ignored for single-processor runs where output is always done in unchunked mode.

11                         10  x01  Default: no

11

11outxxlinexy  10Scope: restrictedx01  REAL

11Description: y-coord for 1D lines in x-direction

11 Range 10 x01 Default: 0.0
11*:*  10A value between [ymin, ymax]

11outxxlinexyi  10Scope: restrictedx01  INT

11Description: y-index (from 0) for 1D lines in x-direction, overrides IO::outxxlinexy

11 Range 10 x01 Default: -2
110:*  10An index between [0, ny)
11-1:  10Default to physical coordinate IO::outxxlinexy if it is within grid bounds, otherwise revert to using 0
11-2:  10Default to physical coordinate IO::outxxlinexy if it is within grid bounds, otherwise revert to using the y-center of the box

11

11outxxlinexz  10Scope: restrictedx01  REAL

11Description: z-coord for 1D lines in x-direction

11 Range 10 x01 Default: 0.0
11*:*  10A value between [zmin, zmax]

11

11outxxlinexzi  10Scope: restrictedx01  INT

11Description: z-index (from 0) for 1D lines in x-direction, overrides IO::outxxlinexz

11 Range 10 x01 Default: -2
110:*  10An index between [0, nz)
11-1:  10Default to physical coordinate IO::outxxlinexz if it is within grid bounds, otherwise revert to using 0
11-2:  10Default to physical coordinate IO::outxxlinexz if it is within grid bounds, otherwise revert to using the z-center of the box

11

11outxxyplanexz  10Scope: restrictedx01  REAL

11Description: z-coord for 2D planes in xy

11 Range 10 x01 Default: 0.0
11*:*  10A value between [zmin, zmax]

11

11outxxyplanexzi  10Scope: restrictedx01  INT

11Description: z-index (from 0) for 2D planes in xy, overrrides IO::outxxyplanexz

11 Range 10 x01 Default: -2
110:*  10An index between [0, nz)
11-1:  10Default to physical coordinate IO::outxxyplanexz if it is within grid bounds, otherwise revert to using 0
11-2:  10Default to physical coordinate IO::outxxyplanexz if it is within grid bounds, otherwise revert to using the z-center of the box

11

11outxxzplanexy  10Scope: restrictedx01  REAL

11Description: y-coord for 2D planes in xz

11 Range 10 x01 Default: 0.0
11*:*  10A value between [ymin, ymax]

11

11outxxzplanexyi  10Scope: restrictedx01  INT

11Description: y-index (from 0) for 2D planes in xz, overrrides IO::outxxzplanexy

11 Range 10 x01 Default: -2
110:*  10An index between [0, ny)
11-1:  10Default to physical coordinate IO::outxxzplanexy if it is within grid bounds, otherwise revert to using 0
11-2:  10Default to physical coordinate IO::outxxzplanexy if it is within grid bounds, otherwise revert to using the y-center of the box

11

11outxylinexx  10Scope: restrictedx01  REAL

11Description: x-coord for 1D lines in y-direction

11 Range 10 x01 Default: 0.0
11*:*  10A value between [xmin, xmax]

11outxylinexxi  10Scope: restrictedx01  INT

11Description: x-index (from 0) for 1D lines in y-direction, overrides IO::outxylinexx

11 Range 10 x01 Default: -2
110:*  10An index between [0, nx)
11-1:  10Default to physical coordinate IO::outxylinexx if it is within grid bounds, otherwise revert to using 0
11-2:  10Default to physical coordinate IO::outxylinexx if it is within grid bounds, otherwise revert to using the x-center of the box

11

11outxylinexz  10Scope: restrictedx01  REAL

11Description: z-coord for 1D lines in y-direction

11 Range 10 x01 Default: 0.0
11*:*  10A value between [zmin, zmax]

11

11outxylinexzi  10Scope: restrictedx01  INT

11Description: z-index (from 0) for 1D lines in y-direction, overrides IO::outxylinexz

11 Range 10 x01 Default: -2
110:*  10An index between [0, nz)
11-1:  10Default to physical coordinate IO::outxylinexz if it is within grid bounds, otherwise revert to using 0
11-2:  10Default to physical coordinate IO::outxylinexz if it is within grid bounds, otherwise revert to using the z-center of the box

11

11outxyzplanexx  10Scope: restrictedx01  REAL

11Description: x-coord for 2D planes in yz

11 Range 10 x01 Default: 0.0
11*:*  10A value between [xmin, xmax]

11

11outxyzplanexxi  10Scope: restrictedx01  INT

11Description: x-index (from 0) for 2D planes in yz, overrrides IO::outxyzplanexx

11 Range 10 x01 Default: -2
110:*  10An index between [0, nx)
11-1:  10Default to physical coordinate IO::outxyzplanexx if it is within grid bounds, otherwise revert to using 0
11-2:  10Default to physical coordinate IO::outxyzplanexx if it is within grid bounds, otherwise revert to using the x-center of the box

11

11outxzlinexx  10Scope: restrictedx01  REAL

11Description: x-coord for 1D lines in z-direction

11 Range 10 x01 Default: 0.0
11*:*  10A value between [xmin, xmax]

11

11outxzlinexxi  10Scope: restrictedx01  INT

11Description: x-index (from 0) for 1D lines in z-direction, overrides IO::outxzlinexx

11 Range 10 x01 Default: -2
110:*  10An index between [0, nx)
11-1:  10Default to physical coordinate IO::outxzlinexx if it is within grid bounds, otherwise revert to using 0
11-2:  10Default to physical coordinate IO::outxzlinexx if it is within grid bounds, otherwise revert to using the x-center of the box

11outxzlinexy  10Scope: restrictedx01  REAL

11Description: y-coord for 1D lines in z-direction

11 Range 10 x01 Default: 0.0
11*:*  10A value between [ymin, ymax]

11

11outxzlinexyi  10Scope: restrictedx01  INT

11Description: y-index (from 0) for 1D lines in z-direction, overrides IO::outxzlinexy

11 Range 10 x01 Default: -2
110:*  10An index between [0, ny)
11-1:  10Default to physical coordinate IO::outxzlinexy if it is within grid bounds, otherwise revert to using 0
11-2:  10Default to physical coordinate IO::outxzlinexy if it is within grid bounds, otherwise revert to using the y-center of the box

11

11parfilexname  10Scope: restrictedx01  STRING

11Description: Filename for the parameter file to be written

11 Range 10 x01 Default: (none)
11.+  10A valid filename
11ˆ$  10An empty string to choose the original parameter filename

11

11parfilexupdatexevery  10Scope: restrictedx01  INT

11Description: How often to update the parameter file for steered parameters

11 Range 10 x01 Default: (none)
111:*  10Every so many iterations
110:  10Disable updating

11

11parfilexwrite  10Scope: restrictedx01  KEYWORD

11Description: Write a parameter file to ’IO::outxdir’

11 Range 10 x01 Default: copy
11no  10Do not write a parameter file
11copy  10Copy the original parameter file
11generate  10Generate a parameter file from the current settings

11

11printxtimingxinfo  10Scope: restrictedx01  BOOLEAN

11Description: Print timing information on I/O operations.

11                         10  x01  Default: no

11

11recover  10Scope: restrictedx01  KEYWORD

11Description: Recover from a checkpoint file ?

11 Range 10 x01 Default: no
11no  10Don’t recover
11manual  10Recover from the checkpoint file as given in IO::recoverxdir and IO::recoverxfile
11auto  10Automatically recover from the latest checkpoint file found in <recoverxdir>
11autoprobe  10Probe for checkpoint files and automatically recover, continue as usual if nothing was found

11

11recoverxandxremove  10Scope: restrictedx01  BOOLEAN

11Description: Remove checkpoint file after successful recovery ?

11                         10  x01  Default: no

11

11recoverxdir  10Scope: restrictedx01  STRING

11Description: Directory to look for recovery files

11 Range 10 x01 Default: (none)
11.+  10A valid directory name

11recoverxfile  10Scope: restrictedx01  STRING

11Description: Basename of recovery file

11 Range 10 x01 Default: checkpoint.chkpt
11.+  10A valid filename

11

11requirexemptyxoutputxdirectory 10Scope: restrictedx01  BOOLEAN

11Description: Require that IO::outxdir is empty at startup ?

11                         10  x01  Default: no

11

11strictxioxparameterxcheck 10Scope: restrictedx01  BOOLEAN

11Description: Stop on errors while parsing I/O parameters from parameter file ?

11                         10  x01  Default: yes

11

11verbose  10Scope: restrictedx01  KEYWORD

11Description: Level of screen output for I/O

11 Range 10 x01 Default: standard
11none  10No output
11standard  10Initial description for each I/O method
11full  10Maximal output

11

14 Interfaces

General Implements: io

Provides: IOxTruncateOutputFiles to

15 Schedule This section lists all the variables which are assigned storage by thorn CactusBase/IOUtil. Storage can either last for the duration of the run (Always means that if this thorn is activated storage will be assigned, Conditional means that if this thorn is activated storage will be assigned for the duration of the run if some condition is met), or can be turned on for the duration of a schedule function.

Storage NONE

Scheduled Functions

CCTKxSTARTUP x ioutilxstartup x startup routine x

11x11After: 11driverxstartup
11 x 11Language:11c
11 x 11Type: 11function

11

CCTKxRECOVERxVARIABLES (conditional) x ioutilxrecovergh x checkpoint recovery routine x

11x11Language:11c
11 x 11Options: 11level
11 x 11Type: 11function

11

CCTKxINITIAL (conditional) x ioutilxrecoveridfromdatafiles x initial data recovery routine x

11x11Language:11c
11 x 11Options: 11level
11 x 11Type: 11function

11

CCTKxPOSTSTEP (conditional) x ioutilxupdateparfile x append steered parameters to parameter file x

11x11Language:11c
11 x 11Options: 11meta
11 x 11Type: 11function