PUGHSlab

Gabrielle Allen, Tom Goodale, Thomas Radke,
with many comments and suggestions from Erik Schnetter and Jonathan Thornburg

October 2001

Abstract

Thorn PUGHSlab implements the generic hyperslab data extraction API for CCTK arrays.

1 Introduction

Many I/O thorns output data from distributed CCTK array variables. If – in a multiprocessor run – output is done by only one processor, it needs to collect the data from the others. This ties the I/O thorn to the driver since it has to know about domain-decomposed data layout, interprocessor ghostzones, etc.

A clean way of separating the I/O code from the driver is to use a thorn which provides a generic interface to get/put the distributed data on/from the I/O processor for writing/reading. This interface can also provide more features such as downsampling, datatype conversions, or hyperslab selections. A hyperslab is defined in this context a subset of a global CCTK array, with its own dimension, origin, direction, and extents.

Another possible use of hyperslabs is the implementation of certain boundary conditions (e.g. reflection). By having the boundary condition code calling generic hyperslab get/put calls, it can be written without special knowledge about driver specifics.

This thorn documentation describes the complete generic hyperslab API. All routines use CCTK data types in their argument lists and as return codes exclusively. This allows actual implementations of this API to be realized as CCTK function aliases. Different hyperslab thorns can then implementing the API using the same function names, and other thorns using the API can be independent of the actual hyperslab thorns which are compiled in.

The current version of thorn PUGHSlab implements only parts of the CCTK hyperslab API. Please refer to section 3 for implementation details.

2 CCTK Hyperslab API Specification

In general, a hyperslab get/put operation is done in a three-level scheme:

  1. In a first step, a hyperslab mapping is defined by calling one of the following routines:
      Hyperslab_LocalMappingByIndex()
      Hyperslab_LocalMappingByPhys()
      Hyperslab_GlobalMappingByIndex()
      Hyperslab_GlobalMappingByPhys()
    There exists two complement sets of routines: one for the definition of local hyperslabs (which apply to a processor’s local patch of a distributed grid variable only), and one for global hyperslabs (which spawn the entire grid).

    A mapping can be defined by either physical coordinates or by grid index points.

    All hyperslab mapping routines return an integer handle which refers to an internally allocated data structure describing the defined hyperslab.

  2. With such a mapping, hyperslabs can then be extracted/distributed by one or more calls to   Hyperslab_Get()
      Hyperslab_GetList()
      Hyperslab_Put()
      Hyperslab_PutList()
    There are routines for getting/putting a hyperslab from/to a single grid variable or from/to a list of variables.

  3. Once all hyperslabs are done, the hyperslab mapping should be freed by a call to
      Hyperslab_FreeMapping().

If the Hyperslab_Get*()/Hyperslab_Put*() get passed a mapping for a global hyperslab, a global, synchronuous operation will be performed (i.e., it must be called in sync by every processor). All input arguments must be consistent between processors.

2.1 Defining a hyperslab mapping

An \(M\)-dimensional hyperslab subspace mapped into an \(N\)-dimensional space can be specified either by coordinates on the physical grid or by index points on the underlying computational grid.

CCTK_INT Hyperslab_GlobalMappingByIndex (
           CCTK_POINTER_TO_CONST GH,
           CCTK_INT              vindex,
           CCTK_INT              hdim,
           const CCTK_INT       *direction         /* vdim*hdim */,
           const CCTK_INT       *origin            /* vdim */,
           const CCTK_INT       *extent            /* hdim */,
           const CCTK_INT       *downsample        /* hdim */,
           CCTK_INT              table_handle,
           CCTK_FPOINTER         conversion_fn,
           CCTK_INT             *hsize             /* hdim */);

CCTK_INT Hyperslab_GlobalMappingByPhys (
           CCTK_POINTER_TO_CONST GH,
           CCTK_INT              vindex,
           CCTK_INT              hdim,
           CCTK_STRING           coord_system_name,
           const CCTK_INT       *direction         /* vdim*hdim */,
           const CCTK_REAL      *origin            /* vdim */,
           const CCTK_REAL      *extent            /* hdim */,
           const CCTK_INT       *downsample        /* hdim */,
           CCTK_INT              table_handle,
           CCTK_FPOINTER         conversion_fn,
           CCTK_INT             *hsize             /* hdim */);


CCTK_INT Hyperslab_LocalMappingByIndex (
           CCTK_POINTER_TO_CONST GH,
           CCTK_INT              vindex,
           CCTK_INT              hdim,
           const CCTK_INT       *direction         /* vdim*hdim */,
           const CCTK_INT       *origin            /* vdim */,
           const CCTK_INT       *extent            /* hdim */,
           const CCTK_INT       *downsample        /* hdim */,
           CCTK_INT              table_handle,
           CCTK_FPOINTER         conversion_fn,
           CCTK_INT             *hsize_local,      /* hdim */
           CCTK_INT             *hsize_global,     /* hdim */
           CCTK_INT             *hoffset_global    /* hdim */);

CTK_INT Hyperslab_LocalMappingByPhys (
           CCTK_POINTER_TO_CONST GH,
           CCTK_INT              vindex,
           CCTK_INT              hdim,
           CCTK_STRING           coord_system_name,
                                                                                       
                                                                                       
           const CCTK_INT       *direction         /* vdim*hdim */,
           const CCTK_REAL      *origin            /* vdim */,
           const CCTK_REAL      *extent            /* hdim */,
           const CCTK_INT       *downsample        /* hdim */,
           CCTK_INT              table_handle,
           CCTK_FPOINTER         conversion_fn,
           CCTK_INT             *hsize_local,      /* hdim */
           CCTK_INT             *hsize_global,     /* hdim */
           CCTK_INT             *hoffset_global    /* hdim */);

Function arguments:

Return codes (according to the Cactus Coding Conventions):

2.2 Extracting/distributing a hyperslab

Each set of hyperslab get/put routines has two functions: one which operates on a single hyperslab, and another which gets/puts hyperslabs for a list of variables. Depending on the actual hyperslab implementation it might be more efficient to operate on a list of grid variables using Hyperslab_GetList()/Hyperslab_PutList() rather than doing sequential calls to Hyperslab_Get()/Hyperslab_Put() with individual grid variables.

CCTK_INT Hyperslab_Get (CCTK_POINTER_TO_CONST GH,
                        CCTK_INT       mapping_handle,
                        CCTK_INT       proc,
                        const CCTK_INT vindex,
                        const CCTK_INT timelevel,
                        const CCTK_INT hdatatype,
                        void          *hdata);

CCTK_INT Hyperslab_GetList (CCTK_POINTER_TO_CONST GH,
                            CCTK_INT        mapping_handle,
                            CCTK_INT        num_arrays,
                            const CCTK_INT *procs      /* num_arrays */,
                            const CCTK_INT *vindices   /* num_arrays */,
                            const CCTK_INT *timelevels /* num_arrays */,
                            const CCTK_INT *hdatatypes /* num_arrays */,
                            void *const    *hdata      /* num_arrays */,
                            CCTK_INT       *retvals    /* num_arrays */);

CCTK_INT Hyperslab_Put (CCTK_POINTER_TO_CONST GH,
                        CCTK_INT              mapping_handle,
                        CCTK_INT              proc,
                        CCTK_INT              vindex,
                        CCTK_INT              timelevel,
                        CCTK_INT              hdatatype,
                        CCTK_POINTER_TO_CONST hdata);

CCTK_INT Hyperslab_PutList (CCTK_POINTER_TO_CONST       GH,
                            CCTK_INT                    mapping_handle,
                            CCTK_INT                    num_arrays,
                            const CCTK_INT             *procs      /* num_arrays */,
                            const CCTK_INT             *vindices   /* num_arrays */,
                            const CCTK_INT             *timelevels /* num_arrays */,
                            const CCTK_INT             *hdatatypes /* num_arrays */,
                            const CCTK_POINTER_TO_CONST hdata      /* num_arrays */,
                            CCTK_INT                   *retvals    /* num_arrays */);

Function arguments:

Return Codes for these routines (according to the Cactus Code conventions:

3 Implementation Details

The current version of thorn PUGHSlab implements only parts of the CCTK hyperslab API as described in section 2:

  1. the definition of local/global hyperslabs based on grid indicices
    Currently, the only additional hyperslab mapping information which can be passed through a key/value table is a CCKT_INT option with the key with_ghostzones. If the value of this key is non-zero PUGHSlab will not strip outer boundary ghostzones for periodic boundary conditions as implemented by PUGH.

    PUGHSlab provides a set of predefined built-in functions for the following classes of data type conversions:

  2. local/global hyperslab extractions
    Global hyperslab get requests will strip off all processor-boundary ghostzones from the returned hyperslab data.

    Local hyperslabs will always include processor-boundary ghostzones. The hsize_local, hoffset_local information returned by the hyperslab mapping routines should be used to locate the locate hyperslab within the global grid (e.g. during a recombination of several local hyperslabs into a single global one).

4 Parameters

5 Interfaces

General

Implements:

hyperslab

Adds header:

PUGHSlab.h to Hyperslab.h

Provides:

Hyperslab_Get to

Hyperslab_GetList to

Hyperslab_GlobalMappingByIndex to

Hyperslab_LocalMappingByIndex to

Hyperslab_FreeMapping to

6 Schedule

This section lists all the variables which are assigned storage by thorn CactusPUGH/PUGHSlab. Storage can either last for the duration of the run (Always means that if this thorn is activated storage will be assigned, Conditional means that if this thorn is activated storage will be assigned for the duration of the run if some condition is met), or can be turned on for the duration of a schedule function.

Storage

NONE