1 Introduction

Many applications which use Cactus will want to make use of coordinate systems. The CoordBase thorn provides a method of registering coordinate systems and their properties. Thorns which implement coordinate systems will register the systems they provide with CoordBase. Since coordinate systems are composed of a collection of coordinates, the coordinates which comprise the systems are registered with CoordBase as well. The data describing coordinate systems are held on Cactus key-value tables. A schema for the format of these tables is provided in section 6. Developers are free to compose and use their own schema, but are encouraged to use the one detailed here, as this will be the standard for all of the core Cactus thorns.

CoordBase specifies an extensible set of coordinate properties. The use of key-value tables to hold these properties makes CoordBase flexible and configurable. The coordinate values themselves can be specified in a number of ways, depending on the nature of the coordinate system. This way symmetries of the coordinates on the computational grid can be exploited to minimize memory consumption. Via a function call, one can easily ascertain what coordinate system is associated with any given grid variable, which is convenient for e.g. I/O. A method of registering default coordinates for all grid variables of a certain dimension makes it simple to specify which coordinate systems should be associated with which grid variables.

The coordinate infrastructure provided by CoordBase will work seamlessly with AMR and multi-model codes.

Thorns which provide coordinates will inherit from CoordBase, and register both the coordinate systems they provide, and the default coordinate systems for the appropriate dimension of grid variables. In addition, one can associate a coordinate with any Cactus grid variable, within a thorn’s interface.ccl. Coordinate systems specified in the interface.ccl override defaults for the dimension.

The coordinate functions in the Cactus flesh are deprecated with the introduction of this thorn. Please use the functions provided here in preference to those in the flesh.

2 Coordinate system symmetries

Since computations performed with Cactus are done on a discrete lattice, only a discrete set of coordinate values are used for any coordinate system. The symmetries of how the coordinate values vary on the grid points make coordinates fall into three types: uniform, nonuniform, and warped. (At least these are the three cases that the CoordBase schema considers.) A uniform coordinate varies from each of its neighbors by a constant. i.e. its value can be determined from the index of the grid point from simply an origin and constant ‘delta’. A nonuniform coordinate has a spatially varying ‘delta’. For both uniform and nonuniform coordinates, the coordinate values do not vary along the other directions of the grid. (e.g. the \(x\) coordinate will be the same regardless of the ‘j’ index of the grid point.) Thus one could completely determine the coordinate values of a 3D system of nonuniform coordinates by providing three 1D arrays. This later assumption is relaxed for a warped coordinate; a warped coordinate will vary across the entire grid. Recall that ‘coordinate lines’ (lines of constant coordinate value) cannot cross (because one n-tuple of coordinate values would specify muliple points in space), so this places a ‘bound’ of sorts on the possible ‘warping’ of the coordinates.

The type of a coordinate system will be the same as that of its coordinates. If there are different types of coordinates within the same system, then the coordinate system is mixed. Note that a warped coordinate system is the most general possible, so any coordinate system could be regarded as warped if one wishes.

2.1 Specifying coordinate values

As mentioned above, for a uniform coordinate system, it is sufficient to specify the origin and spacing for a uniform coordinate. One may also specify a grid variable to hold the coordinate values, if desired. See section 6. A nonuniform coordinate can be specified with a 1D grid variable, if desired, or an nD variable. A warped coordinate system will always need a nD grid variable. FMR and AMR will need an nD grid variable to specify the coordinate values. A mixed coordinate system can use some combination of the above, or simply an nD grid variable.

3 Coordinate Thorns

Generally coordinate thorns will allow the declaration of default coordinate systems for a dimension to be controled by parameters.

If a thorn attempts to register a default for a dimension which already has a default registered, a level 1 warning will result, and the default will be overwritten with the new default. Since the order of execution of the registration calls will in general not be specified, one must be careful when activating more than one coordinate thorn.

Coordinate systems and defaults can be changed at any time throughout a run, as can coordinate properties. In general these are set at CCTK_WRAGH and CCTK_BASEGRID, respectively.

The coordinate thorns are responsible for filling out the portions of the coordinate and coordinate system tables that are not written by the registration routines. See sections 6.1 and 6.2.

4 Application thorns

Application thorns should check at CCTK_PARAMCHECK to see if the correct coordinate system is being used for each relevant grid variable. While the old flesh API is still being used you may want to make this merely a warning. Use Coord_GroupSystem() to find the coordinate system for groups (see section 5).

5 Coordinate APIs

CCTK_INT systemhandle = Coord_SystemRegister(CCTK_POINTER_TO_CONST GH,
                                             CCTK_INT dim,
                                             CCTK_STRING systemname)

registers a coordinate system, along with its dimension, with the CoordBase thorn. This will create a coordinate system table, and return the handle of the table for success, or a negative error code upon error. The possible errors are:

CCTK_INT systemhandle = Coord_SystemHandle(CCTK_POINTER_TO_CONST GH,
                                           CCTK_STRING systemname)

returns the handle for a given coordinate system, or negative on error:

CCTK_INT coordhandle = Coord_CoordRegister(CCTK_POINTER_TO_CONST GH,
                                           CCTK_INT systemhandle,
                                           CCTK_INT direction,
                                           CCTK_STRING coordname)

registers a coordinate within a coordinate system, in the specified ‘direction’. (Direction in this context means the index in the coordinate basis, which ranges from 1 to the dimension of the system.) Coord_CoordRegister() returns the coordinate handle, or negative for an error:

CCTK_INT coordhandle = Coord_CoordHandle(CCTK_POINTER_TO_CONST GH,
                                         CCTK_STRING coordname,
                                         CCTK_STRING systemname)

returns the coordinate handle for a given coordinatate in a coordinate system, or negative on error:

int systemhandle = Coord_GroupSystem(const cGH *GH,
                                     const char *groupname)

returns the handle for the coordinate system associated with a group of grid variables, or negative on error. This can either be the default for coordinate systems of this dimension, or the coordinate system that is specified in the interface.ccl. Coordinate systems specified in interface.ccl will override any defaults. The possible error codes are:

CCTK_INT systemhandle = Coord_SetDefaultSystem(CCTK_POINTER_TO_CONST GH,
                                               CCTK_STRING systemname)

sets this coordinate system to be the default for grid variables of the same dimension. It returns the handle of the system, or negative for errors:

CCTK_INT systemhandle = Coord_GetDefaultSystem(CCTK_POINTER_TO_CONST GH,
                                               CCTK_INT dim)

gets the default coordinate system for grid variables of dimension dim (ranging from 1 to the maximum number of dimensions registered). It returns the handle of the system, or negative for errors:

6 Coordinate Schema

6.1 Coordinate System Tables

Associated with each coordinate system is a table, which should have the following entries:

6.2 Coordinate Tables

Associated with each coordinate of each coordinate system is another table, which should have the following entries:

7 Specifying coordinate systems in the interface.ccl

Coordinate systems associated with grid variable groups can be specified in the group’s tags table, using the key COORDSYSTEM. Below is a grid array which could represent a vector field on a 2-sphere.

CCTK_REAL SphericalVectorField TYPE=ARRAY DIM=2 TAGS=’COORDSYSTEM="sphere2d" TENSORTYPE="vector"’
{
  field_theta, field_phi
}

Even though another thorn has set a default for all 2D grid variables to something else, Coord_GroupSystem() will always return the handle for sphere2d when called on this group.

8 Specifying the extent of the physical domain

CoordBase provides a way for specifying the extent of the simulation domain that is independent of the actual coordinate and symmetry thorns. This is necessary because the size of the physical domain is not necessarily the same as the size of the computational grid, which is usually enlarged by symmetry zones and/or boundary zones.

The physical domain is characterised by the location of its lower and upper boundary and by its grid spacing. These quantities are related to the extent and the number of grid cells that span the domain. The relation between the size of the domain and the size of the computational grid is defined in section 8 below. The domain extent as defined in this section is the continuum extent and is independent of the actual discretisation. This makes it possible to adapt the same domain specification for different resolutions by changing only a single parameter.

The domain is specified in one of the following ways, which is selected by the keyword parameter domainsize:

minmax:

by the location of its lower and upper boundary

extent:

by its extent, i.e. its width

spacing:

by grid spacing and the number of grid cells

The domain specification uses the number of grid cells instead of the number of grid points because the latter can easily lead to one-off errors.

The domain size in each dimension is specificied in equivalent ways. For example, the \(x\)-dimension is specified by a set of some of the following parameters:

zero_origin_x (boolean):

When the domain size is specified by extent or by spacing, then the origin (lower boundary) can either be located at \(x=0\), which leads to the domain \(x \in [0,L]\) with the extent \(L\), or the domain can be symmetric with respect to \(x=0\), which leads to \(x \in [-L/2,L/2]\).

xmin and xmax:

When the domain is specified by the location of its lower and upper boundary, then these specify the locations.

xextent:

When the domain is specified by its extent, then this specifies the extent.

dx (real) and ncells_x (int):

When the domain is specified by grid spacing and the number of grid cells, then these specify the grid spacing and the number of grid cells.

9 Specifying the location of the boundary points

CoordBase also provides a way for specifying the discretisation of the boundary that is independent of the actual boundary thorns. This defines the locations of the boundary points and thus the extent of the computational grid.

Each face of the grid is specified independently. The specification does not depend on the resolution, so that it need not be adapted when the resolution is changed. Figure 1 shows a domain with several interior grid points and one boundary point.

While the physical boundary has a width of zero, the computational grid can have more than one boundary points. This boundary size is defined by the integer parameter boundary_size_x_lower for the lower \(x\) boundary (and similarly for the other boundaries). Figure 2 shows an example where there are two boundary points. When it is necessary to increase the number of boundary points (e.g., to accommodate a larger stencil), then this is the only parameter that needs to be changed.

Depending on the physical setup — and depending on the personal taste — the boundary points should be located either inside or outside of the physical boundary. The boolean parameter boundary_internal_x_lower specifies whether the boundary points extend inwards at the lower \(x\) face. Figure 3 shows the same example, but with internal boundary points.

Depending on the physical setup — and depending on the personal taste — the boundary points should either be staggered about the physical boundary, or the last boundary point should be located exactly on the physical boundary. This is specified by the boolean parameter boundary_staggered_x_lower. Figures 4 and 5 show exampled of external and internal staggered boundary points.

Finally, the integer parameter boundary_shiftout_x_lower can be used to shift the boundary points outwards (or inwards with negative values) by multiples of the grid spacing. Figure 6 shows an example of an exterior, non-staggered boundary with a shiftout of one.

The following table gives examples for common situations:

For other boundary conditions such as Dirichlet or Robin, one can choose these parameters freely.

10 Querying about boundary points

When iterating over grid points, one usually needs to know about the boundary sizes and boundary types present. One convenient way is provided by the aliased function GetBoundarySizesAndTypes:

CCTK_INT FUNCTION GetBoundarySizesAndTypes
  (CCTK_POINTER_TO_CONST IN cctkGH,
   CCTK_INT IN size,
   CCTK_INT OUT ARRAY bndsize,
   CCTK_INT OUT ARRAY is_ghostbnd,
   CCTK_INT OUT ARRAY is_symbnd,
   CCTK_INT OUT ARRAY is_physbnd)

The input argument size describes the size of the four output arrays as allocated by the caller, and should be 2*cctk_dim. The output of this routine describes, for each of the faces of the domain:

bndsize

The number of boundary points

is_ghostbnd

Whether this face is a ghost (inter-process) boundary

is_symbnd

Whether this face is a symmetry boundary

is_physbnd

Whether this face is a physical boundary

11 Driver Issues

When using the CoordBase domain and boundary parameters, the driver (PUGH, Carpet, etc) also needs to be kept informed.

Whichever driver you’re using, if you’re specifying the number of boundary zones using CoordBase, note that this does not automatically carry over to the driver::ghost_size_x, driver::ghost_size_y, and driver::ghost_size_z parameters. If you want the ghost sizes to be anything other than their defaults (currently \(1\)), you need to set them explicitly.

11.1 PUGH

If you’re using PUGH, you must still explicitly set the parameters driver::global_nx, driver::global_ny, and driver::global_nz. If you don’t set these parameters, PUGH will assume a default \(10 \times 10 \times 10\) grid, which is almost certainly not what you want!

11.2 Carpet

If you’re using Carpet, things are nice: just set

Carpet::domain_from_coordbase = true

and Carpet will get all its grid information from CoordBase.

12 Parameters

13 Interfaces

General

Implements:

coordbase

Adds header:

CoordBase.h

Provides:

Coord_SystemRegister to

Coord_SystemHandle to

Coord_CoordRegister to

Coord_CoordHandle to

Coord_GroupSystem to

Coord_SetDefaultSystem to

Coord_GetDefaultSystem to

GetBoundarySpecification to

GetDomainSpecification to

ConvertFromPhysicalBoundary to

ConvertFromInteriorBoundary to

ConvertFromExteriorBoundary to

GetBoundarySizesAndTypes to

14 Schedule

This section lists all the variables which are assigned storage by thorn CactusBase/CoordBase. Storage can either last for the duration of the run (Always means that if this thorn is activated storage will be assigned, Conditional means that if this thorn is activated storage will be assigned for the duration of the run if some condition is met), or can be turned on for the duration of a schedule function.

Storage

NONE

Scheduled Functions

CCTK_STARTUP

  coordbase_startup

  register a gh extension to store the coordinate system handles