xx NaNChecker

NaNChecker

Thomas Radke

Date

Abstract

Thorn NaNChecker reports NaN values found in CCTK grid variables.

1 Purpose

The NaNChecker thorn can be used to analyze Cactus grid variables (that is grid functions, arrays or scalars) of real or complex data type for NaN (Not-a-Number) and (on availability of finite(3)) infinite values. Grid variables can be periodically checked, or a call can be inserted into a thorn to check at a specific point.

This thorn is a utility thorn, designed to be used for debugging and testing code for uninitialised variables, or for variables which become corrupted during a simulation, for example following a division by zero or illegal memory usage.

On many architectures, uninitialised variables will be given the value zero, and simulations using such variables will seemingly run perfectly well. However, not only is it dubious programming practise to assume such behaviour, but also moving to a new machine may well cause pathalogical problems (for example, with Alpha processors used in Compaq or Cray machines). It is thus recommended to test codes periodically with the NaNChecker, and to fix any problems as soon as they are seen.

2 Periodic Testing

Periodic testing of variables can easily be achieved by adding NaNChecker to the ActiveThorns parameter, and setting the parameters

NaNChecker::checkxevery, NaNChecker::checkxafter, and NaNChecker::checkxvars

to the required values. (For most testing purposes these can be set to 1, 0, and ”all” respectively).

The NaNChecker then registers a routine at CCTKxANALYSIS which checks at every

NanChecker::checkxevery iteration – starting at iteration number NaNChecker::checkxafter

– all the variables listed in NaNChecker::checkxvars for NaN or infinite values (depending on NaNChecker::checkxfor) and — if such a value is found — performs an action as specified in

NaNChecker::actionxifxfound.

Currently these actions can be to

justxwarn (the default)

just print a level 1 warning message telling you where NaNs/Infs were found and how many (for grid array variables). If the keyword parameter verbose is set to "all" then for each grid array it will also print the grid indices (in Fortran notation) and the physical coordinates for all NaN/Inf elements found. You can limit the number of such warnings by setting the NanChecker::reportxmax parameter.

     terminate

also set the CCTK termination flag so that Cactus will stop the evolution loop and gracefully terminate at the next time possible (giving you the choice of outputting the data from the last evolution timestep),

     abort

print the warning message(s) and immediately terminate Cactus after checking all variables from

     NaNChecker::checkxvars by a call to CCTKxAbort()

By default, the current timelevel of the variables given in NaNChecker::checkxvars will be checked. This can be overwritten by an optional string [timelevel=<timelevel>] appended to the variable/group name. For example, to apply the NaNChecker to timelevel 0 of the variable grid::x, timelevel 1 of grid::y and timelevel 2 of grid::z you would use the parameter

NaNChecker::checkxvars = "grid::x grid::y[timelevel=1] grid::z[timelevel=2]"

3 Tracking and Visualizing NaNs Positions

The NaNChecker thorn can also mark the positions (in grid index points) of all the NaNs found for a given list of CCTK grid functions in a mask array and save this array to an HDF5 file.

The mask array is declared as a grid function NaNChecker::NaNmask with data type INTEGER. Each bit in an integer element is used to flag a NaN value found in grid function at the corresponding grid position (the counting for starts at 0 and is incremented for each grid function as it appears in NaNChecker::checkxvars). Thus the NaN locations of up to 32 individual grid functions can be coded in the NaNmask array.

In order to activate the NaNmask output you need to set the parameter NaNChecker::outxNaNmask to "yes" (which is already the default) and have the IOHDF5 thorn activated.

The NaN locations can be visualized with OpenDX. An example DX network VisualizeNaNs.net and a sample NaNmask HDF5 output file NaNmask.h5 are available via anonymous CVS from the NumRel CVS server:

xx#xthisxisxforx(t)csh;xusexexportxCVSROOTxforxbash
xxsetenvxCVSROOTx:pserver:cvs_anon@cvs.aei.mpg.de:/numrelcvs
xx#xCVSxpserverxpasswordxisx'anon'
xxcvsxlogin
xxcvsxcheckoutxAEIPhysics/Visualization/OpenDX/Networks/Miscellaneous

4 NaNChecker API

Thorn NaNChecker also provides a function API which can be used by other code to invoke the NaNChecker routines to test for NaN/Inf values or to set NaN values for a list of variables:

C API

xxintxNaNChecker_CheckVarsForNaNx(constxcGHx*cctkGH,
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxintxreport_max,
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxconstxcharx*vars,
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxconstxcharx*check_for,
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxconstxcharx*action_if_found);
xxintxNaNChecker_SetVarsToNaNx(constxcGHx*cctkGH,
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxconstxcharx*vars);

Fortran API

xxcallxNaNChecker_CheckVarsForNaNx(ierror,xcctkGH,xreport_max,
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxvars,xcheck_for,xaction_if_found)
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxintegerxierror
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxCCTK_POINTERxcctkGH
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxintegerxreport_max
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxcharacter*(*)xvars
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxcharacter*(*)xcheck_for
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxcharacter*(*)xaction_if_found
xxcallxNaNChecker_SetVarsToNaNx(ierror,xcctkGH,xvars)
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxintegerxierror
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxCCTK_POINTERxcctkGH
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxcharacter*(*)xvars

The reportxmax, checkxvars, checkxfor and actionxifxfound arguments have the same semantics as their

parameter counterparts. If actionxifxfound is given as a NULL pointer (C API) or as an empty string (Fortran API) the routine will be quiet and just return the number of NaN values found.

The C function NaNCheckerxCheckVarsForNaN() returns the total number of NaN/Inf values found, NaNCheckerxSetToNaN() returns the total number of variables set to NaN; this return value is stored in the ierror argument for the corresponding fortran wrapper routines.

5 Parameters

11 actionxifxfound 10 Scope: private01 KEYWORD

11 Description: What to do if a NaN was found

11 Range 10 01 Default: just warn
11 just warn  10 Just print a level 1 warning
11 terminate  10 Warn and terminate Cactus gracefully as soon as possible
11 abort  10 Warn and abort Cactus immediately

11

11 checkxafter  10 Scope: private01  INT

11 Description: Start checking for NaNs after so many iterations

11 Range 10 01 Default: (none)
11 0:*  10 Any valid iteration number

11 checkxevery  10 Scope: private01  INT

11 Description: How often to check for NaNs

11 Range 10 01 Default: (none)
11                     10 Never (default)
11 1:*  10 Every so many iterations

11

11 checkxfor  10 Scope: private01  KEYWORD

11 Description: Check for NaNs and/or infinite numbers (only evaluated if finite(3) is available)

11 Range 10 01 Default: both
11 NaN  10 Check only for NaNs
11 Inf  10 Check only for infinite numbers
11 both  10 Check for both NaNs and infinite numbers

11

11 checkxvars  10 Scope: private01  STRING

11 Description: Groups and/or variables to check for NaNs

11 Range 10 01 Default: (none)
11 .*  10 List of full group and/or variable names, or ’all’ for everything

11

11 ignorexrestrictedxpoints10 Scope: private01  BOOLEAN

11 Description: do not check grid points whose values will be restricted away

11                     10              01  Default: no

11

11 outxnanmask  10 Scope: private01  BOOLEAN

11 Description: Dump the NaN grid function mask into an HDF5 file

11                     10              01  Default: yes

11

11 reportxmax  10 Scope: private01  INT

11 Description: How many NaNs to report for a single variable

11 Range 10 01 Default: -1
11 -1  10 Report all (default)
11 0:*  10 Do not report more than reportxmax number of NaNs

11

11 restrictionxmask  10 Scope: private01  STRING

11 Description: grid function to use to decide which points are restricted away, points where the mask is zero are ignored

11 Range 10 01 Default: CarpetReduce::weight
11 see [1] below  10 Carpet’s reduction mask
11 see [1] below  10 takes prolongation stencil into account
11 .*[:][:].*  10 any grid function with points masked out set to zero

11

CarpetReduce[:][:]weight

[1]

CarpetEvolutionMask[:][:]evolution\_mask

11 setupxtest  10 Scope: private01  BOOLEAN

11 Description: set up grid function with NaNs

11                     10              01  Default: no

11

11 verbose  10 Scope: private01  KEYWORD

11 Description: How much information to give

11 Range 10 01 Default: standard
11 all  10 All information
11 standard  10 Standard information

11

6 Interfaces

General Implements: nanchecker

Inherits: reduce

Grid Variables

6.0.1 PRIVATE GROUPS

11 x Group Names x 10 x Variable Names x01 Details x 11 x
11 nanmask 10 NaNmask 01 compact 11 0
11                    10                    01 description 11 Grid function mask for NaN locations
11                    10                    01 dimensions 11 3
11                    10                    01 distribution 11 DEFAULT
11                    10                    01 group type 11 GF
11                    10                    01  tags 11 Prolongation=”None” checkpoint=”no”
11                    10                    01 timelevels 11 1
11                    10                    01 variable type 11 INT
11 nansfound 10 NaNsFound 01 compact 11 0
11                    10                    01 description 11 Scalar variable counting the number of NaNs found
11                    10                    01 dimensions 11 0
11                    10                    01 distribution 11 CONSTANT
11                    10                    01 group type 11 SCALAR
11                    10                    01  tags 11 checkpoint=”no”
11                    10                    01 timelevels 11 1
11                    10                    01 variable type 11 INT
11 testgf 10 TestGF 01 compact 11 0
11                    10                    01 description 11 Grid function to hold NaNs for testsuite
11                    10                    01 dimensions 11 3
11                    10                    01 distribution 11 DEFAULT
11                    10                    01 group type 11 GF
11                    10                    01  tags 11 Prolongation=”None” checkpoint=”no”
11                    10                    01 timelevels 11 1
11                    10                    01 variable type 11 REAL

11

Adds header: NaNCheck.h to NaNChecker.h

Provides: CheckVarsForNaN to SetVarsToNaN to

7 Schedule This section lists all the variables which are assigned storage by thorn CactusUtils/NaNChecker. Storage can either last for the duration of the run (Always means that if this thorn is activated storage will be assigned, Conditional means that if this thorn is activated storage will be assigned for the duration of the run if some condition is met), or can be turned on for the duration of a schedule function.

Storage x

11 Always:  11 Conditional:
11 NaNmask NaNsFound 11 TestGF

11 x  11 x

11

Scheduled Functions CCTKxBASEGRID x nancheckerxresetcounter x reset the nanchecker::nansfound counter x

11 x 11 Language: 11 c
11 x 11 Options: 11 global
11 x 11 Type:  11 function
11 x 11 Writes: 11 nanchecker::nansfound(everywhere)

11

CCTKxPRESTEP x nancheckerxresetcounter x reset the nanchecker::nansfound counter x

11 x 11 Language: 11 c
11 x 11 Options: 11 global
11 x 11 Type:  11 function
11 x 11 Writes: 11 nanchecker::nansfound(everywhere)

11

NaNCheckerxNaNCheck x nancheckerxnancheckxprepare x prepare data structures to check for nans x

11 x 11 Language: 11 c
11 x 11 Options: 11 level
11 x 11 Type:  11 function
11 x 11 Writes: 11 nanchecker::nanmask(everywhere)

11

CCTKxPOSTSTEP x nancheckerxnancheck x check for nans and count them in nanchecker::nansfound x

11 x 11 Type: 11 group

11

NaNCheckerxNaNCheck x nancheckerxnancheckxcheck x check for nans x

11 x 11 After:  11 nancheckerxnancheckxprepare
11 x 11 Language: 11 c
11 x 11 Options: 11 local
11 x 11 Type:  11 function

11

NaNCheckerxNaNCheck x nancheckerxnancheckxfinish x count nans in nanchecker::nansfound x

11 x 11 After:  11 nancheckerxnancheckxcheck
11 x 11 Language: 11 c
11 x 11 Options: 11 level
11 x 11 Type:  11 function
11 x 11 Writes: 11 nanchecker::nansfound(everywhere)

11

CCTKxPOSTSTEP x nancheckerxtakeaction x output nanchecker::nanmask and take action according to nanchecker::actionxifxfound

11 x 11 After:  11 zzzxnancheckerxnancheck
11 x 11 Language: 11 c
11 x 11 Options: 11 global
11 x 11 x  11 loop-level
11 x 11 Reads:  11 nanchecker::nansfound(everywhere)
11 x 11 Type:  11 function

11

CCTKxPOSTxRECOVERxVARIABLES x nancheckerxnancheck x check for nans and count them in nanchecker::nansfound x

11 x 11 Type: 11 group

11

CCTKxPOSTxRECOVERxVARIABLES x nancheckerxtakeaction x output nanchecker::nanmask and take action according to nanchecker::actionxifxfound x

11 x 11 After:  11 zzzxnancheckerxnancheck
11 x 11 Language: 11 c
11 x 11 Options: 11 global
11 x 11 x  11 loop-level
11 x 11 Reads:  11 nanchecker::nansfound(everywhere)
11 x 11 Type:  11 function

11

CCTKxINITIAL (conditional) x nancheckerxsetuptest x set test grid function to nan x

11 x 11 Language: 11 c
11 x 11 Type:  11 function

11

Aliased Functions x

11 Alias Name: xxxxxxx 11 Function Name:
11 NaNCheckerxNaNCheck 11 zzzxNaNCheckerxNaNCheck