xx Generic Hyperslabbing

Generic Hyperslabbing

Erik Schnetter <eschnetter@perimeterinstitute.ca>

Date

Abstract

The Slab thorn provides generic slabbing facilities. A slab is a sub-array of another array. Both the array and the slab can be multidimensional, and the slab can have a non-unit stride. The Slab thorn provides a routine to copy a slab from one array into a slab of another array, while possibly transposing or mirroring the slab. The processor distribution of the arrays can be specified freely, so that the Slab thorn can be used to interface to non-Cactus libraries with different data layouts.

The Slab thorn is driver independent, i.e., not tied to PUGH or Carpet, and does not require MPI for single-processor configurations.

1 Introduction

A Slab is a subarray of another array. This concept is used in many places with many different names. Fortran has so-called “array subscript triplets”, which represent the same thing. In BLAS, the “leading dimension” arguments are used to described slabs. Slabs are sometimes also called “array views” in object oriented applications. Slabs are rectangular in shape, and can be rotated with regard to their containing array by multiples of 90 degrees. They can also be mirrored, i.e., the direction of axes can be inverted.

It is often necessary to copy slabs from one array into other arrays, or to copy one slab of an array to another slab of the same array. This can be used to change the processor distribution of some data, or to apply symmetry or periodicity boundary conditions, or to collect data onto a single processor to process it more easily.

2 Using This Thorn

SlabxTransfer

Transfer a slab contained in one array to a (possibly different) slab of another (possibly the same) array

Synopsis

INHERITS:xSlab
USESxINCLUDExHEADER:xSlab.h
#includex"cctk.h"
#includex"Slab.h"
structxslabinfox{
xxintxgsh;
xxintxlbnd,xlsh;
xxintxash;
xxintxlbbox,xubbox,xnghostzones;
xxintxoff,xstr,xlen;
};
structxxferinfox{
xxstructxslabinfoxsrc,xdst;
xxintxxpose;
xxintxflip;
};
intxSlab_Transferx(cGHxxxxxxxxxxxxxxxxxxx*xrestrictxconstxcctkGH,
xxxxxxxxxxxxxxxxxxxintxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxconstxdim,
xxxxxxxxxxxxxxxxxxxstructxxferinfoxconstx*xrestrictxconstxxferinfo,
xxxxxxxxxxxxxxxxxxxintxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxconstxoptions,
xxxxxxxxxxxxxxxxxxxintxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxconstxsrctype,
xxxxxxxxxxxxxxxxxxxvoidxxxxxxxxxxxxconstx*xxxxxxxxxxconstxsrcptr,
xxxxxxxxxxxxxxxxxxxintxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxconstxdsttype,
xxxxxxxxxxxxxxxxxxxvoidxxxxxxxxxxxxxxxxxx*xxxxxxxxxxconstxdstptr);

Result

0 Success

nonzero Failure

Parameters

cctkGH Pointer to the CCTK grid hierarchy dim Number of dimensions of the arrays and slabs. Must be nonnegative.

xferinfo[dim] Describes the layout of the slab transfer, i.e., the shape and distribution of the source and destination arrays, and the locations of the source and destination slabs, and a possible transformation between the slabs. Each dimension is described separately. See the entries xferinfo[d].* below.

xferinfo[d].src Describes the source array and source slab. See the slabinfo entries below.

xferinfo[d].dst Describes the destination array and destination slab. See the slabinfo entries below.

xferinfo[d].xpose Describes how to transpose the slab, i.e., possibly permuting the slab axes, as in . xferinfo[d].xpose contains an integer value in the range 0xdim-1, specifying the source axis corresponding to the destination axis d. Specify xferinfo[d].xpose = d for no transposition. No two values of

xferinfo[*].xpose may be the same.

xferinfo[d].flip Describes how to mirror the slab, i.e., possibly inverting the slab axes, as in . xferinfo[d].flip containes a boolean value specifying whether the axis in direction d should be inverted, i.e., either 0 or 1, where 0 indicates no inversion, and 1 indicates inversion.

When axes are both transposed and inverted while a slab is copied, then the transposing happens first, and the axis inversion later. That is, the sequence of “actions” is: extract slab, transpose, invert, insert slab. For example, when transposing the and axes and inverting the axis, then the destination slab’s axis is the source slab’s flipped axis, while the destination axis is the unflipped source axis.

slab Describes the shape and processor distribution of one dimension of an array, and the location of this dimension of a slab. The shape and distribution is specified in the same manner as in the cGH structure. See the entries slab.* below.

slab.gsh Global shape of the array; the overall number of grid points

slab.lbnd Lower boundary of the array; the global index of the lower boundary of the processor-local part of the array

slab.lsh Local shape of the array; the number of grid points on this processor

slab.ash Local allocated shape of the array; the number of grid points on this processor including padding slab.lbbox Lower bounding box of the array; whether the lower boundary of the array is an outer boundary. This corresponds to the even entries in Cactus’ bbox array. Must be 0 or 1.

slab.ubbox Upper bounding box of the array; whether the upper boundary of the array is an outer boundary. This corresponds to the odd entries in Cactus’ bbox array. Must be 0 or 1.

slab.nghostzones Number of ghost zones of the array; the number of grid points on this processor that are only copied from neighbouring processors

slab.off Slab offset; the global index of the lowest grid point of the slab

slab.str Slab stride; the distance between two grid points making up the slab. Specify slab.str = 1 for a unit stride. Must be positive.

slab.len Slab length; the number of grid points that make up the slab. This does not count the grid points that are skipped if the slab has a non-unit stride. Must be nonnegative.

options Handle of an option table. Two options are understood, CCTKxINT useghosts and CCTKxPOINTER comm. useghosts (see below) specifies whether values in ghost zones may be used in the source array. comm is a pointer to an MPI communicator that should be used instead of MPIxCOMMxWORLD or the communicator returned by the aliased function GetMPICommWorld.

srctype Type of the source array. Pass the corresponding CCTKxVARIABLEx* constant.

srcptr Pointer to the source array

dsttype Type of the destination array. Pass the corresponding CCTKxVARIABLEx* constant.

dstptr Pointer to the destination array

Discussion

SlabxTransfer copies one slab from one array onto a possibly different slab in possibly the same array. The shape and processor distribution of the arrays can be specified freely, as long as each processor holds a rectangular subset of grid points. The location of the slab can also be specified freely, and the slab can be rotated (by multiples of 90 degrees) or inverted before it is copied. The source and destination slab are allowed to overlap.

By default, SlabxTransfer is conservative with regard to synchronisation. It assumes that the ghost zones of the source array are not valid, but will correctly fill in the ghost zones in the destination slab. This can be changed by setting the option useghosts in the option table.

There are currently some restrictions, which can be remedied if the need arises: The dimension must be 3 (lower dimensions can easily be padded). The communication schedule is set up for every slab transfer, which is expensive. The number of ghost zones along the lower and upper boundary is assumed to be the same. There is no Fortran interface.

See Also

CarpetSlab The hyperslabbing thorn of the driver Carpet.

PUGHSlab The hyperslabbing thorn of the driver CactusPUGH.

New hyperslabbing API The web page http://www.cactuscode.org/Development/Current.html contains a

hyperlink to the new proposed hyperslabbing API. This API is slightly different from the one used here.

Examples

C The identity transfer: copy a whole 3D grid function without transforming it

#includex<assert.h>
#includex"cctk.h"
#includex"cctk_Arguments.h"
#includex"Slab.h"
DECLARE_CCTK_ARGUMENTS;
structxxferinfoxinfo[3];
intxd;
intxierr;
/*xSetxupxthexarrayxdescriptorsx*/
assertx(cctk_dimx<=x3);
forx(d=0;xd<cctk_dim;x++d)x{
xxinfo[d].src.gshx=xcctk_gsh[d];
xxinfo[d].src.lbndx=xcctk_lbnd[d];
xxinfo[d].src.lshx=xcctk_lsh[d];
xxinfo[d].src.ashx=xcctk_ash[d];
xxinfo[d].src.lbboxx=xcctk_bbox[2*d];
xxinfo[d].src.ubboxx=xcctk_bbox[2*d+1];
xxinfo[d].src.nghostzonesx=xcctk_nghostzones[d];
xxinfo[d].dst.gshx=xcctk_gsh[d];
xxinfo[d].dst.lbndx=xcctk_lbnd[d];
xxinfo[d].dst.lshx=xcctk_lsh[d];
xxinfo[d].dst.ashx=xcctk_ash[d];
xxinfo[d].dst.lbboxx=xcctk_bbox[2*d];
xxinfo[d].dst.ubboxx=xcctk_bbox[2*d+1];
xxinfo[d].dst.nghostzonesx=xcctk_nghostzones[d];
}
/*xSetxupxthexslabxandxtransformationxdescriptorsx*/
assertx(cctk_dimx<=x3);
forx(d=0;xd<cctk_dim;x++d)x{
xxinfo[d].src.offx=x0;
xxinfo[d].src.lenx=xcctk_gsh[d];
xxinfo[d].src.strx=x1;
xxinfo[d].dst.offx=x0;
xxinfo[d].dst.lenx=xcctk_gsh[d];
xxinfo[d].dst.strx=x1;
xxinfo[d].xposex=xd;
xxinfo[d].flipx=x0;

}
/*xTransferxthexslabx*/
ierrx=xSlab_Transfer
xx(cctkGH,xcctk_dim,xinfo,x-1,
xxxCCTK_VARIABLE_REAL,xgf1,xCCTK_VARIABLE_REAL,xgf2);
assertx(!ierr);

C A complicated transfer: copy a subarray, transposing the and axes, and inverting the axis

#includex<assert.h>
#includex"cctk.h"
#includex"cctk_Arguments.h"
#includex"Slab.h"
DECLARE_CCTK_ARGUMENTS;
structxxferinfoxinfo[3];
intxd;
intxierr;
/*xSetxupxthexarrayxdescriptorsx(samexasxabove)x*/
assertx(cctk_dimx<=x3);
forx(d=0;xd<cctk_dim;x++d)x{
xxinfo[d].src.gshx=xcctk_gsh[d];
xxinfo[d].src.lbndx=xcctk_lbnd[d];
xxinfo[d].src.lshx=xcctk_lsh[d];
xxinfo[d].src.ashx=xcctk_ash[d];
xxinfo[d].src.lbboxx=xcctk_bbox[2*d];
xxinfo[d].src.ubboxx=xcctk_bbox[2*d+1];
xxinfo[d].src.nghostzonesx=xcctk_nghostzones[d];
xxinfo[d].dst.gshx=xcctk_gsh[d];
xxinfo[d].dst.lbndx=xcctk_lbnd[d];
xxinfo[d].dst.lshx=xcctk_lsh[d];
xxinfo[d].dst.ashx=xcctk_ash[d];
xxinfo[d].dst.lbboxx=xcctk_bbox[2*d];
xxinfo[d].dst.ubboxx=xcctk_bbox[2*d+1];
xxinfo[d].dst.nghostzonesx=xcctk_nghostzones[d];
}
/*xSetxupxthexslabxandxtransformationxdescriptorsx*/
info[0].src.offx=x5;
info[1].src.offx=x5;
info[2].src.offx=x1;
info[0].src.strx=x1;
info[1].src.strx=x1;
info[2].src.strx=x1;
info[0].src.lenx=x2;
info[1].src.lenx=x3;
info[2].src.lenx=x2;

info[0].dst.offx=x2;
info[1].dst.offx=x3;
info[2].dst.offx=x4;
info[0].dst.strx=x1;
info[1].dst.strx=x1;
info[2].dst.strx=x1;
info[0].dst.lenx=x2;
info[1].dst.lenx=x3;
info[2].dst.lenx=x2;
info[0].xposex=x2;
info[1].xposex=x1;
info[2].xposex=x0;
info[0].flipx=x1;
info[1].flipx=x0;
info[2].flipx=x0;
/*xTransferxthexslabx*/
ierrx=xSlab_Transfer
xx(cctkGH,xcctk_dim,xinfo,x-1,
xxxCCTK_VARIABLE_REAL,xgf1,xCCTK_VARIABLE_REAL,xgf2);
assertx(!ierr);

3 Obtaining This Thorn

This thorn is available from the CactusNumerical arrangement in the arrangements directory on the svn.cactuscode.org CVS server via anonymous pserver access.

4 Support and Feedback

This thorn was written by Erik Schnetter <eschnetter@perimeterinstitute.ca>xin 2002. Contact me for questions and comments.

5 Parameters

11timerxoutput 10Scope: privatex01  BOOLEAN

11Description: Print slabbing timings at shutdown time

11          10  x01  Default: no

11

11usexalltoallv 10Scope: privatex01  BOOLEAN

11Description: Use MPIxAlltoallv for communication?

11          10  x01  Default: no

11

6 Interfaces

General Implements: slab

Adds header: slab.h to Slab.h slab.inc to Slab.inc

Uses header: loopcontrol.h

7 Schedule This section lists all the variables which are assigned storage by thorn CactusNumerical/Slab. Storage can either last for the duration of the run (Always means that if this thorn is activated storage will be assigned, Conditional means that if this thorn is activated storage will be assigned for the duration of the run if some condition is met), or can be turned on for the duration of a schedule function.

Storage NONE

Scheduled Functions

CCTKxSTARTUP x slabxinitmpidatatypes x create mpi datatypes for complex variables in c x

11x11After: 11driverxstartup
11 x 11Language:11c
11 x 11Type: 11function

11

CCTKxWRAGH x slabxinittimers x initialise timers x

11x11Language:11c
11 x 11Type: 11function

11

CCTKxSHUTDOWN (conditional) x slabxprinttimers x print timers x

11x11Language:11c
11 x 11Type: 11function