Generic Hyperslabbing

Erik Schnetter <eschnetter@perimeterinstitute.ca>

\( \)Date\( \)

Abstract

The Slab thorn provides generic slabbing facilities. A slab is a sub-array of another array. Both the array and the slab can be multidimensional, and the slab can have a non-unit stride. The Slab thorn provides a routine to copy a slab from one array into a slab of another array, while possibly transposing or mirroring the slab. The processor distribution of the arrays can be specified freely, so that the Slab thorn can be used to interface to non-Cactus libraries with different data layouts.

The Slab thorn is driver independent, i.e., not tied to PUGH or Carpet, and does not require MPI for single-processor configurations.

1 Introduction

A Slab is a subarray of another array. This concept is used in many places with many different names. Fortran has so-called “array subscript triplets”, which represent the same thing. In BLAS, the “leading dimension” arguments are used to described slabs. Slabs are sometimes also called “array views” in object oriented applications. Slabs are rectangular in shape, and can be rotated with regard to their containing array by multiples of 90 degrees. They can also be mirrored, i.e., the direction of axes can be inverted.

It is often necessary to copy slabs from one array into other arrays, or to copy one slab of an array to another slab of the same array. This can be used to change the processor distribution of some data, or to apply symmetry or periodicity boundary conditions, or to collect data onto a single processor to process it more easily.

2 Using This Thorn

Slab_Transfer

Transfer a slab contained in one array to a (possibly different) slab of another (possibly the same) array

Synopsis

C

INHERITS: Slab
USES INCLUDE HEADER: Slab.h

#include "cctk.h"
#include "Slab.h"

struct slabinfo {
  int gsh;
  int lbnd, lsh;
  int ash;
  int lbbox, ubbox, nghostzones;
  int off, str, len;
};

struct xferinfo {
  struct slabinfo src, dst;
  int xpose;
  int flip;
};

int Slab_Transfer (cGH                   * restrict const cctkGH,
                   int                              const dim,
                   struct xferinfo const * restrict const xferinfo,
                   int                              const options,
                   int                              const srctype,
                   void            const *          const srcptr,
                   int                              const dsttype,
                   void                  *          const dstptr);


Result

0 Success
nonzero Failure

Parameters

cctkGH Pointer to the CCTK grid hierarchy

dim Number of dimensions of the arrays and slabs. Must be nonnegative.

xferinfo[dim] Describes the layout of the slab transfer, i.e., the shape and distribution of the source and destination arrays, and the locations of the source and destination slabs, and a possible transformation between the slabs. Each dimension is described separately. See the entries xferinfo[d].* below.

xferinfo[d].src Describes the source array and source slab. See the slabinfo entries below.

xferinfo[d].dst Describes the destination array and destination slab. See the slabinfo entries below.

xferinfo[d].xpose Describes how to transpose the slab, i.e., possibly permuting the slab axes, as in \((x,y) \rightarrow (y,x)\). xferinfo[d].xpose contains an integer value in the range 0…dim-1, specifying the source axis corresponding to the destination axis d. Specify xferinfo[d].xpose = d for no transposition. No two values of xferinfo[*].xpose may be the same.

xferinfo[d].flip Describes how to mirror the slab, i.e., possibly inverting the slab axes, as in \((x) \rightarrow (-x)\). xferinfo[d].flip containes a boolean value specifying whether the axis in direction d should be inverted, i.e., either 0 or 1, where 0 indicates no inversion, and 1 indicates inversion.

When axes are both transposed and inverted while a slab is copied, then the transposing happens first, and the axis inversion later. That is, the sequence of “actions” is: extract slab, transpose, invert, insert slab. For example, when transposing the \(x\) and \(z\) axes and inverting the \(x\) axis, then the destination slab’s \(x\) axis is the source slab’s flipped \(z\) axis, while the destination \(z\) axis is the unflipped source \(x\) axis.

slab Describes the shape and processor distribution of one dimension of an array, and the location of this dimension of a slab. The shape and distribution is specified in the same manner as in the cGH structure. See the entries slab.* below.

slab.gsh Global shape of the array; the overall number of grid points

slab.lbnd Lower boundary of the array; the global index of the lower boundary of the processor-local part of the array

slab.lsh Local shape of the array; the number of grid points on this processor

slab.ash Local allocated shape of the array; the number of grid points on this processor including padding

slab.lbbox Lower bounding box of the array; whether the lower boundary of the array is an outer boundary. This corresponds to the even entries in Cactus’ bbox array. Must be 0 or 1.

slab.ubbox Upper bounding box of the array; whether the upper boundary of the array is an outer boundary. This corresponds to the odd entries in Cactus’ bbox array. Must be 0 or 1.

slab.nghostzones Number of ghost zones of the array; the number of grid points on this processor that are only copied from neighbouring processors

slab.off Slab offset; the global index of the lowest grid point of the slab

slab.str Slab stride; the distance between two grid points making up the slab. Specify slab.str = 1 for a unit stride. Must be positive.

slab.len Slab length; the number of grid points that make up the slab. This does not count the grid points that are skipped if the slab has a non-unit stride. Must be nonnegative.

options Handle of an option table. Two options are understood, CCTK_INT useghosts and CCTK_POINTER comm. useghosts (see below) specifies whether values in ghost zones may be used in the source array. comm is a pointer to an MPI communicator that should be used instead of MPI_COMM_WORLD or the communicator returned by the aliased function GetMPICommWorld.

srctype Type of the source array. Pass the corresponding CCTK_VARIABLE_* constant.

srcptr Pointer to the source array

dsttype Type of the destination array. Pass the corresponding CCTK_VARIABLE_* constant.

dstptr Pointer to the destination array

Discussion

Slab_Transfer copies one slab from one array onto a possibly different slab in possibly the same array. The shape and processor distribution of the arrays can be specified freely, as long as each processor holds a rectangular subset of grid points. The location of the slab can also be specified freely, and the slab can be rotated (by multiples of 90 degrees) or inverted before it is copied. The source and destination slab are allowed to overlap.

By default, Slab_Transfer is conservative with regard to synchronisation. It assumes that the ghost zones of the source array are not valid, but will correctly fill in the ghost zones in the destination slab. This can be changed by setting the option useghosts in the option table.

There are currently some restrictions, which can be remedied if the need arises: The dimension must be 3 (lower dimensions can easily be padded). The communication schedule is set up for every slab transfer, which is expensive. The number of ghost zones along the lower and upper boundary is assumed to be the same. There is no Fortran interface.

See Also

CarpetSlab The hyperslabbing thorn of the driver Carpet.

PUGHSlab The hyperslabbing thorn of the driver CactusPUGH.

New hyperslabbing API The web page http://www.cactuscode.org/Development/Current.html contains a hyperlink to the new proposed hyperslabbing API. This API is slightly different from the one used here.

Examples

C
The identity transfer: copy a whole 3D grid function without transforming it

INHERITS: Slab
USES INCLUDE HEADER: Slab.h

#include <assert.h>
#include "cctk.h"
#include "cctk_Arguments.h"

#include "Slab.h"

DECLARE_CCTK_ARGUMENTS;

struct xferinfo info[3];
int d;
int ierr;

/* Set up the array descriptors */
assert (cctk_dim <= 3);
for (d=0; d<cctk_dim; ++d) {
  info[d].src.gsh = cctk_gsh[d];
  info[d].src.lbnd = cctk_lbnd[d];
  info[d].src.lsh = cctk_lsh[d];
  info[d].src.ash = cctk_ash[d];
  info[d].src.lbbox = cctk_bbox[2*d];
  info[d].src.ubbox = cctk_bbox[2*d+1];
  info[d].src.nghostzones = cctk_nghostzones[d];
  info[d].dst.gsh = cctk_gsh[d];
  info[d].dst.lbnd = cctk_lbnd[d];
  info[d].dst.lsh = cctk_lsh[d];
  info[d].dst.ash = cctk_ash[d];
  info[d].dst.lbbox = cctk_bbox[2*d];
  info[d].dst.ubbox = cctk_bbox[2*d+1];
  info[d].dst.nghostzones = cctk_nghostzones[d];
}

/* Set up the slab and transformation descriptors */
assert (cctk_dim <= 3);
for (d=0; d<cctk_dim; ++d) {
  info[d].src.off = 0;
  info[d].src.len = cctk_gsh[d];
  info[d].src.str = 1;
  info[d].dst.off = 0;
  info[d].dst.len = cctk_gsh[d];
  info[d].dst.str = 1;
  info[d].xpose = d;
  info[d].flip = 0;
                                                                                       
                                                                                       
}

/* Transfer the slab */
ierr = Slab_Transfer
  (cctkGH, cctk_dim, info, -1,
   CCTK_VARIABLE_REAL, gf1, CCTK_VARIABLE_REAL, gf2);
assert (!ierr);


C
A complicated transfer: copy a subarray, transposing the \(x\) and \(z\) axes, and inverting the \(x\) axis

INHERITS: Slab
USES INCLUDE HEADER: Slab.h

#include <assert.h>
#include "cctk.h"
#include "cctk_Arguments.h"

#include "Slab.h"

DECLARE_CCTK_ARGUMENTS;

struct xferinfo info[3];
int d;
int ierr;

/* Set up the array descriptors (same as above) */
assert (cctk_dim <= 3);
for (d=0; d<cctk_dim; ++d) {
  info[d].src.gsh = cctk_gsh[d];
  info[d].src.lbnd = cctk_lbnd[d];
  info[d].src.lsh = cctk_lsh[d];
  info[d].src.ash = cctk_ash[d];
  info[d].src.lbbox = cctk_bbox[2*d];
  info[d].src.ubbox = cctk_bbox[2*d+1];
  info[d].src.nghostzones = cctk_nghostzones[d];
  info[d].dst.gsh = cctk_gsh[d];
  info[d].dst.lbnd = cctk_lbnd[d];
  info[d].dst.lsh = cctk_lsh[d];
  info[d].dst.ash = cctk_ash[d];
  info[d].dst.lbbox = cctk_bbox[2*d];
  info[d].dst.ubbox = cctk_bbox[2*d+1];
  info[d].dst.nghostzones = cctk_nghostzones[d];
}

/* Set up the slab and transformation descriptors */
info[0].src.off = 5;
info[1].src.off = 5;
info[2].src.off = 1;
info[0].src.str = 1;
info[1].src.str = 1;
info[2].src.str = 1;
info[0].src.len = 2;
info[1].src.len = 3;
info[2].src.len = 2;

                                                                                       
                                                                                       
info[0].dst.off = 2;
info[1].dst.off = 3;
info[2].dst.off = 4;
info[0].dst.str = 1;
info[1].dst.str = 1;
info[2].dst.str = 1;
info[0].dst.len = 2;
info[1].dst.len = 3;
info[2].dst.len = 2;

info[0].xpose = 2;
info[1].xpose = 1;
info[2].xpose = 0;
info[0].flip = 1;
info[1].flip = 0;
info[2].flip = 0;

/* Transfer the slab */
ierr = Slab_Transfer
  (cctkGH, cctk_dim, info, -1,
   CCTK_VARIABLE_REAL, gf1, CCTK_VARIABLE_REAL, gf2);
assert (!ierr);


3 Obtaining This Thorn

This thorn is available from the CactusNumerical arrangement in the arrangements directory on the svn.cactuscode.org CVS server via anonymous pserver access.

4 Support and Feedback

This thorn was written by Erik Schnetter <eschnetter@perimeterinstitute.ca> in 2002. Contact me for questions and comments.

5 Parameters




timer_output
Scope: private  BOOLEAN



Description: Print slabbing timings at shutdown time



  Default: no






use_alltoallv
Scope: private  BOOLEAN



Description: Use MPI_Alltoallv for communication?



  Default: no



6 Interfaces

General

Implements:

slab

Adds header:

slab.h to Slab.h

slab.inc to Slab.inc

Uses header:

loopcontrol.h

7 Schedule

This section lists all the variables which are assigned storage by thorn CactusNumerical/Slab. Storage can either last for the duration of the run (Always means that if this thorn is activated storage will be assigned, Conditional means that if this thorn is activated storage will be assigned for the duration of the run if some condition is met), or can be turned on for the duration of a schedule function.

Storage

NONE

Scheduled Functions

CCTK_STARTUP

  slab_initmpidatatypes

  create mpi datatypes for complex variables in c

 

 After: driver_startup
 Language:c
 Type: function

CCTK_WRAGH

  slab_inittimers

  initialise timers

 

 Language:c
 Type: function

CCTK_SHUTDOWN (conditional)

  slab_printtimers

  print timers

 

 Language:c
 Type: function