xx PUGH

PUGH

The Cactus Team

cactusmaint@cactuscode.org

Date

Abstract

The default unigrid driver for Cactus for both multiprocessor and single process runs, handling grid variables and communications.

1 Description PUGH can create, handle and communicate grid scalars, arrays and functions in 1, 2 or 3-dimensions.

2 Compilation PUGH can be compiled with or without MPI. Compiling without MPI results in an executable which can only be used on a single processor, compiling with MPI leads to an executable which can be used with either single or multiple processors. (Sectionx6 describes how you can tell if your executable has been compiled with or without MPI). For configuring with MPI, see the Cactus User’s Guide.

3 Grid Size The number of grid points used for a simulation can be set in PUGH either globally (that is, the total number of points across all processors), or locally (that is, the number of points on each processor). To set the global size of a N-D grid to be 40 grid points in each direction use

xxPUGH::global_nsizex=x40

To set the global size of a 2D grid to be use

xxPUGH::global_nxx=x40
xxPUGH::global_nyx=x20

To set the local size of a 2D grid to be on each processor, use

xxPUGH::local_nxx=x40
xxPUGH::local_nyx=x20

4 Periodic Boundary Conditions

PUGH can implement periodic boundary conditions during the synchronization of grid functions. Although this may at first seem a little confusing, and unlike the usual use of boundary conditions which are directly called from evolution routines, it is the most efficient and natural place for periodic boundary conditions.

PUGH applies periodic conditions by simply communicating the appropriate ghostzones between “end” processors. For example, for a 1D domain with two ghostzones, split across two processors, Figurex1 shows the implementation of periodic boundary conditions.

PIC

Figurex1: Implementation of periodic boundary conditions for a 1D domain, with two ghostzones, split across two processors. The lines labelled A show the standard communications during synchronisation, the lines labelled B show the additional communications for periodic boundary conditions.

Periodic boundary conditions are applied to all grid functions, by default they are applied in all directions, although this behaviour can be customised to switch them off in given directions.

By default, no periodic boundary conditions are applied. To apply periodic boundary conditions in all directions, set

xxPUGH::periodicx=x"yes"

To apply periodic boundary conditions in just the x- and y- directions in a 3 dimensional domain, use

xxPUGH::periodicx=x"yes"
xxPUGH::periodic_zx=x"no"

5 Processor Decomposition

By default PUGH will distribute the computational grid evenly across all processors (as in Figurex2a). This may not be efficient if there is a different computational load on different processors, or for example for a simulation distributed across processors with different per-processor performance.

PIC Figurex2: Partitioning of the computational grid across processors, Figurexa) is the default type of partition used by PUGH, Figurexb) can be set manually, and Figurexc) is not possible with PUGH The computational grid can be manually partitioned in each direction in a regularly way as in Figurex2b.

The computational grid can be manually distributed using PUGH’s string parameters

partition_[1d_x|2d_x|2d_y|3d_x|3d_y|3d_z]. To manually specify the load distribution, set

PUGH::partition = "manual" and then, depending on the grid dimension, set the remaining parameters to distribute the load in each direction. Note that for this you need to know apriori the processor decomposition.

The decomposition is easiest to explain with a simple example: to distribute a 30-cubed grid across 4 processors (decomposed as , with processors 0 and 2 performing twice as fast as processors 1 and 3) as:

11 proc 2: 11 proc 3:
11 proc 0: 11 proc 1:

11

you would use the following topology and partition parameter settings:

xx#xthexoverallxgridxsize
xxPUGH::global_nsizex=x30
xx#xprocessorxtopology
xxPUGH::processor_topologyxxxxxx=xmanual
xxPUGH::processor_topology_3d_xx=x2
xxPUGH::processor_topology_3d_yx=x1
xxPUGH::processor_topology_3d_zx=x2xxxxx#xredundant
xx#xgridxpartitioning
xxPUGH::partitionxxxxxx=x"manual"
xxPUGH::partition_3d_xx=x"20x10"

Each partition parameter lists the number of grid points for every processor in that direction, with the numbers delimited by any non-digit characters. Note that an empty string for a direction (which is the default value for the partition parameters) will apply the automatic distribution. That’s why it is not necessary

to set PUGH::partition_3d_yx=x"30" or PUGH::partition_3d_zx=x"15x15" in the parameter

file.

Because the previous automatic distribution gave problems in some cases (e.g.xvery long box in one, but short in other directions), there is now an improved algorithm that tries to do a better job in decomposing the grid evenly to the processors. However, it can fail in certain situations, in which it is gracefully falling back to the previous ("automatic_old") giving a warning. Note that, if one or more of the parameters

PUGH::processor_topology_3d_* or PUGH::partition_3d_* are set, this mode automatically falls back to
"automatic_old" without warning.

6 Understanding PUGH Output

PUGH reports information about the processor decomposition to standard output at the start of a job. This section describes how to interpret that output.

Single Processor (no MPI)

Type of evolution If an executable has been compiled for only single processor use (without MPI), the first thing which PUGH reports is this fact:

INFO (PUGH): Single processor evolution

Multiple Processor (with MPI)

Type of evolution If an executable has been compiled using MPI, the first thing which PUGH reports is this fact, together with the number of processors being used:

INFO (PUGH): MPI Evolution on 3 processors

Maximum load skew The maximum load skew describes the variance in the number of gridpoints on each processor, and is defined by

. Max Skew = 100 \frac{Max Points − Min Points}{Average Points}

For most purposes, the maximum skew should ideally be close to zero, however if your simulation has a different load at different grid points, or if you are running across processors with different properties, the optimal skew could be quite different. By default, PUGH tries to minize the skew in gridpoints, however this may be overriden by performing the load balancing manually.

7 Useful Parameters

There are several parameters in PUGH which are useful for debugging and optimisation:

PUGH::enablexallxstorage

Enables storage for all grid variables (that is, not only those set in a thorn’s schedule.ccl file). Try this parameter if you are getting segmentation faults. If enabling all storage removes the problem, it most likely means that you are accessing a grid variable (probably in a Fortran thorn) for which storage has not been set.

PUGH::initialisexmemory

By default, when PUGH allocates storage for a grid variable it does not initialise its elements. If you access an uninitialised variable on some platforms you will get a segmentation fault (and in general you will see erratic behaviour). This parameter can be used to initialise all elements to zero, if this removes your segmentation fault you can then track down the cause of the problem by using the same parameter to initialize all elements to NaNs and then track them with the

thorn CactusUtils/NaNChecker.

Note that it isn’t recommended to simply use this parameter to initialise all elements to zero, instead we recommend you to set all variables to their correct values before using them.

PUGH::storagexverbose

This parameter can be set to print out the number of grid variables which have storage allocated at each iteration, and the total size of the storage allocated by Cactus. Note that this total does not include storage allocated independently in thorns.

PUGH::timerxoutput

This parameter can be set to provide the time spent communicating variables between processors.

8 Parameters

11 cachelinexmult 10 Scope: private01 INT

11 Description: Multiplier for cacheline number

11 Range 10 01 Default: 4001
11 0:*  10 Any positive number

11

11 enablexallxstorage  10 Scope: private01  BOOLEAN

11 Description: Enable storage for all GFs?

11                          10              01  Default: no

11

11 ghostxsize  10 Scope: private01  INT

11 Description: The width of the ghost zone in each direction

11 Range 10 01 Default: -1
11 -1:*  10 Any positive number to override the ghostxsizex[xyz] parameters

11

11 ghostxsizexx  10 Scope: private01  INT

11 Description: The width of the ghost zone in the x direction

11 Range 10 01 Default: 1
11 0:*  10 Must be a positive integer

11

11 ghostxsizexy  10 Scope: private01  INT

11 Description: The width of the ghost zone in the y direction

11 Range 10 01 Default: 1
11 0:*  10 Must be a positive integer

11

11 ghostxsizexz  10 Scope: private01  INT

11 Description: The width of the ghost zone in the z direction

11 Range 10 01 Default: 1
11 0:*  10 Must be a positive integer

11

11 globalxnsize  10 Scope: private01  INT

11 Description: The size of the grid in each spatial direction

11 Range 10 01 Default: -1
11 -1:*  10 Grid of this size in each dir distributed across all processors

11

11 globalxnx  10 Scope: private01  INT

11 Description: The size of the grid in the x direction

11 Range 10 01 Default: 10
11 0:*  10 Grid of this size distributed across all processors

11 globalxny  10 Scope: private01  INT

11 Description: The size of the grid in the y direction

11 Range 10 01 Default: 10
11 0:*  10 Grid of this size distributed across all processors

11

11 globalxnz  10 Scope: private01  INT

11 Description: The size of the grid in the z direction

11 Range 10 01 Default: 10
11 0:*  10 Grid of this size distributed across all processors

11

11 info  10 Scope: private01  KEYWORD

11 Description: Provide additional information about what PUGH is doing

11 Range 10 01 Default: none
11 none  10 No extra information
11 load  10 Load on each processor

11

11 initializexmemory  10 Scope: private01  KEYWORD

11 Description: How to initialize memory for grid variables at allocation time

11 Range 10 01 Default: none
11 none  10 Do not initialize storage for allocated grid variables (default)
11 zero  10 Zero out all elements of all allocated grid variables
11 NaN  10 Set all elements of allocated floating point grid variables to Not-a-Number values

11

11 localxnsize  10 Scope: private01  INT

11 Description: The size of the grid in each spatial direction

11 Range 10 01 Default: -1
11 -1:*  10 Grid of this size in each dir on each processor

11

11 localxnx  10 Scope: private01  INT

11 Description: The size of the grid in the x direction

11 Range 10 01 Default: -1
11 -1:*  10 Grid of this size on each processor

11

11 localxny  10 Scope: private01  INT

11 Description: The size of the grid in the y direction

11 Range 10 01 Default: -1
11 -1:*  10 Grid of this size on each processor

11

11 localxnz  10 Scope: private01  INT

11 Description: The size of the grid in the z direction

11 Range 10 01 Default: -1
11 -1:*  10 Grid of this size on each processor

11 localxsizexincludesxghosts 10 Scope: private01  BOOLEAN

11 Description: Does the local grid size include the ghost zones?

11                          10              01  Default: yes

11

11 overloadabort  10 Scope: private01  BOOLEAN

11 Description: Overload Abort driver function

11                          10              01  Default: yes

11

11 overloadarraygroupsizeb  10 Scope: private01  BOOLEAN

11 Description: Overload ArrayGroupSizeB driver function

11                          10              01  Default: yes

11

11 overloadbarrier  10 Scope: private01  BOOLEAN

11 Description: Overload Barrier driver function

11                          10              01  Default: yes

11

11 overloaddisablegroupcomm 10 Scope: private01  BOOLEAN

11 Description: Overload DisableGroupComm driver function

11                          10              01  Default: yes

11

11 overloaddisablegroupstorage 10 Scope: private01  BOOLEAN

11 Description: Overload DisableGroupStorage driver function

11                          10              01  Default: yes

11

11 overloadenablegroupcomm 10 Scope: private01  BOOLEAN

11 Description: Overload EnableGroupComm driver function

11                          10              01  Default: yes

11

11 overloadenablegroupstorage 10 Scope: private01  BOOLEAN

11 Description: Overload EnableGroupStorage driver function

11                          10              01  Default: yes

11

11 overloadevolve  10 Scope: private01  BOOLEAN

11 Description: Overload Evolve driver function

11                          10              01  Default: yes

11

11 overloadexit  10 Scope: private01  BOOLEAN

11 Description: Overload Exit driver function

11                          10              01  Default: yes

11 overloadgroupdynamicdata 10 Scope: private01  BOOLEAN

11 Description: Overload GroupDynamicData driver function

11                          10              01  Default: yes

11

11 overloadmyproc  10 Scope: private01  BOOLEAN

11 Description: Overload MyProc driver function

11                          10              01  Default: yes

11

11 overloadnprocs  10 Scope: private01  BOOLEAN

11 Description: Overload nProcs driver function

11                          10              01  Default: yes

11

11 overloadparallelinit  10 Scope: private01  BOOLEAN

11 Description: Overload ParallelInit driver function

11                          10              01  Default: yes

11

11 overloadquerygroupstorageb 10 Scope: private01  BOOLEAN

11 Description: Overload QueryGroupStorageB driver function

11                          10              01  Default: yes

11

11 overloadsyncgroup  10 Scope: private01  BOOLEAN

11 Description: Overload SyncGroup driver function

11                          10              01  Default: no

11

11 overloadsyncgroupsbydiri 10 Scope: private01  BOOLEAN

11 Description: Overload SyncGroupsByDirI driver function

11                          10              01  Default: yes

11

11 partition  10 Scope: private01  KEYWORD

11 Description: Is the partition manual

11 Range 10 01 Default: automatic
11 automatic  10 even
11 manual  10 specified by partitionxXYZ ..

11

11 partitionx1dxx  10 Scope: private01  STRING

11 Description: Tells how to partition on direction X

11 Range 10 01 Default: (none)
11 .*  10 A regex which matches anything

11

11 partitionx2dxx  10 Scope: private01  STRING

11 Description: Tells how to partition on direction X

11 Range 10 01 Default: (none)
11 .*  10 A regex which matches anything

11 partitionx2dxy  10 Scope: private01  STRING

11 Description: Tells how to partition on direction y

11 Range 10 01 Default: (none)
11 .*  10 A regex which matches anything

11

11 partitionx3dxx  10 Scope: private01  STRING

11 Description: Tells how to partition on direction X

11 Range 10 01 Default: (none)
11 .*  10 A regex which matches anything

11

11 partitionx3dxy  10 Scope: private01  STRING

11 Description: Tells how to partition on direction y

11 Range 10 01 Default: (none)
11 .*  10 A regex which matches anything

11

11 partitionx3dxz  10 Scope: private01  STRING

11 Description: Tells how to partition on direction z

11 Range 10 01 Default: (none)
11 .*  10 A regex which matches anything

11

11 physical2logical  10 Scope: private01  KEYWORD

11 Description: Physical process to logical process mapping method to use

11 Range 10 01 Default: direct
11 direct  10 Maps MPI IDs directly to IJKs
11 example  10 Maps MPI IDs directly to IJKs using a lookup table

11

11 processorxtopology  10 Scope: private01  KEYWORD

11 Description: How to determine the processor topology

11 Range 10 01 Default: automatic
11 manual  10 Specified by procxtopxnx etc
11 automatic  10 Automatically generated
11 automaticxold  10 Automatically generated (old method)

11

11 processorxtopologyx1dxx 10 Scope: private01  INT

11 Description: Number of processors in X direction

11 Range 10 01 Default: (none)
11 0:*  10 See procxtopology

11

11 processorxtopologyx2dxx 10 Scope: private01  INT

11 Description: Number of processors in X direction

11 Range 10 01 Default: (none)
11 0:*  10 See procxtopology

11 processorxtopologyx2dxy 10 Scope: private01  INT

11 Description: Number of processors in Y direction

11 Range 10 01 Default: (none)
11 0:*  10 See procxtopology

11

11 processorxtopologyx3dxx 10 Scope: private01  INT

11 Description: Number of processors in X direction

11 Range 10 01 Default: (none)
11 0:*  10 See procxtopology

11

11 processorxtopologyx3dxy 10 Scope: private01  INT

11 Description: Number of processors in Y direction

11 Range 10 01 Default: (none)
11 0:*  10 See procxtopology

11

11 processorxtopologyx3dxz 10 Scope: private01  INT

11 Description: Number of processors in Z direction

11 Range 10 01 Default: (none)
11 0:*  10 See procxtopology

11

11 storagexreportxevery  10 Scope: private01  INT

11 Description: How often to provide a report on storage information

11 Range 10 01 Default: (none)
11 0:0  10 Never report
11 1:*  10 Report at intervals

11

11 storagexverbose  10 Scope: private01  KEYWORD

11 Description: Report on memory assignment

11 Range 10 01 Default: no
11 yes  10 Standard storage information
11 report  10 ”Provide a report of storage every storagexreportxevery iterations and at termination”
11 no  10 Provide no information

11

11 timerxoutput  10 Scope: private01  BOOLEAN

11 Description: Print time spent in communication

11                          10              01  Default: no

11

11 periodic  10 Scope: restricted01  BOOLEAN

11 Description: Periodic boundary conditions

11                          10                01  Default: no

11

11 periodicxx  10 Scope: restricted01  BOOLEAN

11 Description: Periodic boundary conditions in x-direction

11                          10                01  Default: yes

11 periodicxy  10 Scope: restricted01  BOOLEAN

11 Description: Periodic boundary conditions in y-direction

11                          10                01  Default: yes

11

11 periodicxz  10 Scope: restricted01  BOOLEAN

11 Description: Periodic boundary conditions in z-direction

11                          10                01  Default: yes

11

11 cctkxfinalxtime  10 Scope: shared from CACTUS01REAL

11

11 cctkxinitialxtime  10 Scope: shared from CACTUS01REAL

11

11 cctkxitlast  10 Scope: shared from CACTUS01INT

11

11 maxxruntime  10 Scope: shared from CACTUS01REAL

11

11 terminate  10 Scope: shared from CACTUS01KEYWORD

11

11 terminatexnext  10 Scope: shared from CACTUS01BOOLEAN

11

9 Interfaces General Implements: driver

Adds header: Register.h to pughxRegister.h

Uses header: pughxRegister.h vectors.h

10 Schedule This section lists all the variables which are assigned storage by thorn CactusPUGH/PUGH. Storage can either last for the duration of the run (Always means that if this thorn is activated storage will be assigned, Conditional means that if this thorn is activated storage will be assigned for the duration of the run if some condition is met), or can be turned on for the duration of a schedule function.

Storage NONE

Scheduled Functions

CCTKxSTARTUP x pughxstartup x startup routine x

11 x 11 Language: 11 c
11 x 11 Type:  11 function

11

CCTKxSTARTUP x pughxregisterpughp2lroutines x register physical to logical process mapping routines x

11 x 11 Language: 11 c
11 x 11 Type:  11 function

CCTKxSTARTUP x pughxregisterpughtopologyroutines x register topology generation routines routines x

11 x 11 Language: 11 c
11 x 11 Type:  11 function

11

CCTKxBASEGRID x pughxreport x report on pugh set up x

11 x 11 Language: 11 c
11 x 11 Type:  11 function

11

CCTKxTERMINATE (conditional) x pughxprinttiminginfo x print time spent in communication x

11 x 11 Language: 11 c
11 x 11 Type:  11 function

11

CCTKxTERMINATE (conditional) x pughxprintfinalstoragereport x print storage information x

11 x 11 Language: 11 c
11 x 11 Type:  11 function

11

CCTKxPOSTSTEP (conditional) x pughxprintstoragereport x print storage information x

11 x 11 Language: 11 c
11 x 11 Type:  11 function

11

CCTKxTERMINATE x pughxterminate x termination routine x

11 x 11 Language: 11 c
11 x 11 Type:  11 function

Aliased Functions x

11 Alias Name: xxxxxxx 11 Function Name:
11 PUGHxStartup  11 DriverxStartup
11 PUGHxTerminate  11 DriverxTerminate