xx IOHDF5

IOHDF5

Thomas Radke

Date

Abstract

Thorn IOHDF5xprovides an I/O method to output variables in HDF5 file format. It also implements checkpointing/recovery functionality using HDF5.

1 Purpose Thorn IOHDF5xuses the standard I/O library

HDF51 ^{1Hierarchical Data Format version 5, see http://hdf.ncsa.uiuc.edu/whatishdf5.html for details} to output any type of CCTK grid variables (grid scalars, grid functions, and grid arrays of arbitrary dimension) in the HDF5 file format.

Output is done by invoking the IOHDF5xI/O method which thorn IOHDF5registers with the flesh’s I/O interface at startup.

Data is written into files named "<varname>.h5". Such datafiles can be used for further postprocessing (eg. visualization) or fed back into Cactus via the filereader capabilities of thorn IOUtil.

2 IOHDF5xParameters

Parameters to control the IOHDF5xI/O method are:

IOHDF5::outxevery (steerable)

How often to do periodic IOHDF5xoutput. If this parameter is set in the parameter file, it will override the setting of the shared IO::outxevery parameter. The output frequency can also be set for individual variables using the outxevery option in an option string appended to the

IOHDF5::outxvars parameter.

IOHDF5::outxvars (steerable)

The list of variables to output using the IOHDF5xI/O method. The variables must be given by their fully qualified variable or group name. The special keyword all requests IOHDF5xoutput for all variables. Multiple names must be separated by whitespaces. An option string can be appended in curly braces to a group/variable name. Supported options are outxevery (to set the output frequency for individual variables) and hyperslab options (see section 4 for details).

IOHDF5::outxdir

The directory in which to place the IOHDF5xoutput files. If the directory doesn’t exist at startup it will be created. If this parameter is set to an empty string IOHDF5xoutput will go to the standard output directory as specified in IO::outxdir.

3 Serial versus Parallel Output According to the ouptput mode parameter settings (IO::outxmode, IO::outxunchunked, IO::outxprocxevery) of thorn IOUtil, thorn IOHDF5xwill output distributed data either

in serial into a single unchunked file

xxIO::out_modexxxxxx=x"onefile"
xxIO::out_unchunkedx=x"yes"

in parallel, that is, into separate files containing chunks of the individual processors’ patches of the distributed array

     xxIO::out_modexxxxxx=x"proc|np"

The default is to output data in parallel, in order to get maximum I/O performance. If needed, you can recombine the resulting chunked datafiles into a single unchunked file using the recombiner utility program. See section 9 for information how to build the recombiner program.

4 Output of Hyperslab Data

By default, thorn IOHDF5xoutputs multidimensional Cactus variables with their full contents resulting in maximum data output. This can be changed for individual variables by specifying a hyperslab as a subset of the data within the N-dimensional volume. Such a subset (called a hyperslab) is generally defined as an orthogonal region into the multidimensional dataset, with an origin (lower left corner of the hyperslab), direction vectors (defining the number of hyperslab dimensions and spanning the hyperslab within the N-dimensional grid), an extent (the length of the hyperslab in each of its dimensions), and an optional downsampling factor.

Hyperslab parameters can be set for individual variables using an option string appended to the variables’ full names in the IOHDF5::outxvars parameter.

Here is an example which outputs two 3D grid functions Grid::r and Wavetoy::phi. While the first is output with their full contents at every 5th iteration (overriding the IOHDF5::outxevery parameter for this variable), a two-dimensional hyperslab is defined for the second grid function. This hyperslab defines a subvolume to output, starting with a 5 grid points offset into the grid, spanning in the yz-plane, with an extent of 10 and 20 grid points in y- and z-direction respectively. For this hyperslab, only every other grid point will be output.

xxIOHDF5::out_everyx=x1
xxIOHDF5::out_varsxx=x"Grid::x{xout_everyx=x5x}
xxxxxxxxxxxxxxxxxxxxxxxWavetoy::phi{xoriginxxxxx=x{4x4x4}
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxdirectionxx=x{0x1x0
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0x0x1}
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxextentxxxxx=x{10x20}
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxdownsamplex=x{2x2}xxx}"

The hyperslab parameters which can be set in an option string are:

origin[N]

This specifies the origin of the hyperslab. It must be given as an array of integer values with elements. Each value specifies the offset in grid points in this dimension into the N-dimensional volume of the grid variable. If the origin for a hyperslab is not given, if will default to 0.

direction[N][M]

The direction vectors specify both the directions in which the hyperslab should be spanned (each vector defines one direction of the hyperslab) and its dimensionality ( the total number of dimension vectors). The direction vectors must be given as a concatenated array of integer values. The direction vectors must not be a linear combination of each other or null vectors. If the direction vectors for a hyperslab are not given, the hyperslab dimensions will default to , and its directions are parallel to the underlying grid.

extent[M]

This specifies the extent of the hyperslab in each of its dimensions as a number of grid points. It must be given as an array of integer values with elements ( being the number of hyperslab dimensions). If the extent for a hyperslab is not given, it will default to the grid variable’s extent. Note that if the origin is set to a non-zero value, you should also set the hyperslab extent otherwise the default extent would possibly exceed the variable’s grid extent.

downsample[M]

To select only every so many grid points from the hyperslab you can set the downsample option.

It must be given as an array of integer values with elements ( being the number of hyperslab dimensions). If the downsample option is not given, it will default to the settings of the general downsampling parameters IO::outxdownsamplex[xyz] as defined by thorn IOUtil.

5 IOHDF5xOutput Restrictions

Due to the naming scheme used to build unique names for HDF5 datasets (see 7, the IOHDF5xI/O method currently has the restriction that it can output a given variable (with a specific timelevel) – or a hyperslab of it – only once per iteration.

As a workaround, you should request output in such a case by using the flesh’s I/O API CCTKxOutputVarAsByMethod() routine with a different alias name for each output. Note that this will create multiple output files for the same variable then.

6 Checkpointing & Recovery

Thorn IOHDF5xcan also be used for creating HDF5 checkpoint files and recovering from such files later on.

Checkpoint routines are scheduled at several timebins so that you can save the current state of your simulation after the initial data phase, during evolution, or at termination. Checkpointing for thorn IOHDF5xis enabled by

setting the parameter IOHDF5::checkpoint = "yes".

A recovery routine is registered with thorn IOUtil in order to restart a new simulation from a given HDF5 checkpoint. The very same recovery mechanism is used to implement a filereader functionality to feed back data into Cactus.

Checkpointing and recovery are controlled by corresponding checkpoint/recovery parameters of thorn IOUtil (for a description of these parameters please refer to this thorn’s documentation). The parameter

IO::checkpointxeveryxwalltimexhours is not (yet) supported.

7 Importing External Data Into Cactus With IOHDF5

In order to import external data into Cactus (eg. to initialize some variable) you first need to convert this data into an HDF5 datafile which then can be processed by the registered recovery routine of thorn IOHDF5.

The following description explains the HDF5 file layout of an unchunked datafile which thorn IOHDF5xexpects in order to restore Cactus variables from it properly. There is also a well-documented example C program provided (IOHDF5/doc/CreateIOHDF5datafile.c) which illustrates how to create a datafile with IOHDF5xfile layout. This working example can be used as a template for building your own data converter program.

Actual data is stored as multidimensional datasets in an HDF5 file. There is no nested grouping structure, every dataset is located in the root group. A dataset’s name must match the following naming pattern which guarantees to generate unique names:

xx"<fullxvariablexname>xtimelevelx<timelevel>xatxiterationx<iteration>"

IOHDF5’s recovery routine parses a dataset’s name according to this pattern to determine the Cactus variable to restore, along with its timelevel. The iteration number is just informative and not needed here.

The type of your data as well as its dimensions are already inherited by a dataset itself as metainformation. But this is not enough for IOHDF5xto safely match it against a specific Cactus variable. For that reason, the variable’s groupname, its grouptype, and the total number of timelevels must be attached to every dataset as attribute information.

Finally, the recovery routine needs to know how the datafile to recover from was created:

Does the file contain chunked or unchunked data ?

How many processors were used to produce the data ?

How many I/O processors were used to write the data ?

What Cactus version is this datafile compatible with ?

Such information is put into as attributes into a group named "Global Attributes". Since we assume unchunked data here the processor information isn’t relevant — unchunked data can be fed back into a Cactus simulation running on an arbitrary number of processors. The Cactus version ID must be present to indicate that grid variables with multiple timelevels should be recovered following the new timelevel scheme (as introduced in Cactus beta 10).

The example C program goes through all of these steps and creates a datafile x.h5 in IOHDF5xfile layout which contains a single dataset named "grid::x timelevel 0 at iteration 0", with groupname "grid::coordinates", grouptype CCTKxGF (thus identifying the variable as a grid function), and the total number of timelevels set to 1.

The global attributes are set to "unchunked"  "yes", nprocs  1, and ioprocxevery  1.

Once you’ve built and ran the program you can easily verify if it worked properly with

xxh5dumpxx.h5

which lists all objects in the datafile along with their values. It will also dump the contents of the 3D dataset. Since it only contains zeros it would probably not make much sense to feed this datafile into Cactus for initializing your x coordinate grid function :-)

8 Building A Cactus Configuration with IOHDF5

The Cactus distribution does not contain the HDF5 header files and library which is used by thorn IOHDF5. You have to include the thorn HDF5 (located in the Cactus ExternalLibraries arrangement). This thorn will either build its own HDF5 library, or use an installed version in some cases.

Thorn IOHDF5xinherits from IOUtil and IOHDF5Util so you need to include these thorns in your thorn list to build a configuration with IOHDF5.

9 Utility Programs provided by IOHDF5

Thorn IOHDF5xprovides the following utility programs:

     hdf5xrecombiner

Recombines chunked HDF5 datafile(s) into a single unchunked HDF5 datafile. By applying the -singlexprecision command line option, double precision floating-point datasets can be converted into single precision during the recombination.

     hdf5xconvertxfromxieeeio

Converts a datafile created by thorn IOFlexIO into an HDF5 datafile. Your thornlist must include this thorn in its thornlist in order to build the FlexIO-to-HDF5 utility program.

     hdf5xconvertxfromxsdf

Converts a datafile created by thorn CactusIO/IOSDF or other Cactus-external programs into an HDF5 datafile. Your thornlist must include this thorn in its thornlist in order to build the SDF-to-HDF5 utility program.

All utility programs are located in the src/util/ subdirectory of thorn IOHDF5. To build the utilities just do a

xxmakex<configuration>-utils

in the Cactus toplevel directory. The executables will then be placed in the exe/<configuration>/ subdirectory.

All utility programs are self-explaining – just call them without arguments to get a short usage info. If any of these utility programs is called without arguments it will print a usage message.

10 Parameters

11checkpoint 10Scope: privatex01 BOOLEAN

11Description: Do checkpointing with HDF5

11                   10  x01  Default: no

11

11checkpointxnext  10Scope: privatex01  BOOLEAN

11Description: Checkpoint at next iteration

11                   10  x01  Default: no

11

11outxdir  10Scope: privatex01  STRING

11Description: Output directory for IOHDF5 files, overrides IO::outxdir

11 Range 10 x01 Default: (none)
11.+  10A valid directory name
11ˆ$  10An empty string to choose the default from IO::outxdir

11

11outxevery  10Scope: privatex01  INT

11Description: How often to do IOHDF5 output, overrides IO::outxevery

11 Range 10 x01 Default: -1
111:*  10Every so many iterations
110:  10Disable IOHDF5 output
11-1:  10Default to IO::outxevery

11

11outxvars  10Scope: privatex01  STRING

11Description: Variables to output in HDF5 file format

11 Range 10 x01 Default: (none)
11.+  10Space-separated list of fully qualified variable/group names
11ˆ$  10An empty string to output nothing

11

11checkpointxevery  10Scope: shared from IOx01INT

11checkpointxid  10Scope: shared from IOx01BOOLEAN

11

11checkpointxkeep  10Scope: shared from IOx01INT

11

11checkpointxonxterminate 10Scope: shared from IOx01BOOLEAN

11

11ioxoutxdir  10Scope: shared from IOx01STRING

11

11ioxoutxevery  10Scope: shared from IOx01INT

11

11outxmode  10Scope: shared from IOx01KEYWORD

11

11outxsavexparameters 10Scope: shared from IOx01KEYWORD

11

11outxtimestepsxperxfile10Scope: shared from IOx01INT

11

11printxtimingxinfo  10Scope: shared from IOx01BOOLEAN

11

11recover  10Scope: shared from IOx01KEYWORD

11

11recoverxandxremove 10Scope: shared from IOx01BOOLEAN

11

11recoverxfile  10Scope: shared from IOx01STRING

11

11strictxioxparameterxcheck 10Scope: shared from IOx01BOOLEAN

11

11verbose  10Scope: shared from IOx01KEYWORD

11 Interfaces

General Implements: iohdf5

12 Schedule This section lists all the variables which are assigned storage by thorn CactusPUGHIO/IOHDF5. Storage can either last for the duration of the run (Always means that if this thorn is activated storage will be assigned, Conditional means that if this thorn is activated storage will be assigned for the duration of the run if some condition is met), or can be turned on for the duration of a schedule function.

Storage NONE

Scheduled Functions

CCTKxSTARTUP x iohdf5xstartup x iohdf5 startup routine x

11x11After: 11iohdf5utilxstartup
11 x 11Language:11c
11 x 11Type: 11function

11

CCTKxCPINITIAL x iohdf5xinitialdatacheckpoint x initial data checkpoint routine x

11x11Language:11c
11 x 11Type: 11function

CCTKxCHECKPOINT x iohdf5xevolutioncheckpoint x evolution data checkpoint routine x

11x11Language:11c
11 x 11Type: 11function

11

CCTKxTERMINATE x iohdf5xterminationcheckpoint x termination checkpoint routine x

11x11Before: 11iohdf5utilxterminate
11 x 11Language:11c
11 x 11Type: 11function

11

CCTKxRECOVERxPARAMETERS (conditional) x iohdf5xrecoverparameters x parameter recovery routine x

11x11Language:11c
11 x 11Type: 11function