Llama is a collection of thorns that defines a set of curvilinear multipatch grids in Cactus. The grids are allowed to overlap. Interpatch boundary data is exchanged via interpolation.
The Llama1 code was implemented to allow one to use multiple curvilinear grid patches within Cactus. The ability to have multiple grid “maps” is provided by Carpet. Patch systems, and their coordinate systems and transformations are defined in Llama.
The core of Llama consists of the following core thorns:
Coordinates: Provides patch systems, coordinate mappings, and transformations.
Interpolate2: Sets up inter-patch interpolation. The interpolation itself is carried out using CarpetInterp2.
In addition, there are a number of optional thorns:
GlobalDerivatives: Provides inlined pointwise finite difference operators in the global Cartesian tensor basis. The differencing is carried out in the local basis. Jacobians are applied to transform to the global basis.
What follows is a brief introduction to using Llama. It assumes that you are familiar with the two papers describing Llama [1, 2].
Llama: Coordinates provides grid functions for Jacobians, inverse Jacobians, and the derivitives of the Jacobian between the local coordinates of a given patch and the global Cartesian coordinates. It also provides a volume form to compute volume integrals in the global frame, taking into account the non-trivial overlap between patches.
The public version of Llama can be found on the website http://www.llama-code.org.
Here, we describe the basic parameter settings for Coordinates and Interpolate2. Parameters that are specific to a given patch system are described in the sections below.
The following parameters must be set for multipatch evolution:
Carpet::domain_from_multipatch must be set to yes for multipatch.
CartGrid3D::type must be set to multipatch for multipatch.
CartGrid3D::set_coordinate_ranges_on must be set to "all maps" for multipatch.
The following basic parameters need to be set
Coordinates::coordinate_system. This selects a particular patch system defined in Llama.
Coordinates::patch_boundary_size. Sets the ghost size of the interpatch boundary. This depends, e.g., on the finite difference stencil that is used. This parameter should be equal to Driver::ghost_size.
Coordinates::additional_overlap_size. Sets the number of additional overlap points. The size of this parameter depends on the interpolation order and type that is used, and on the patch system. One wants this parameters to be as small as possible. Llama will complain if the overlap size is not sufficient for a given patch system and interpolation setup.
Coordinates::outer_boundary_size. Sets the number of ghost zones at outer boundaries (i.e., ghost data is not obtained from inter-patch interpolation) where evolution specific outer boundary conditions must be applied.
Coordinates::store_jacobian. Activate/deactivate storage for Jacobian grid function. Whether storage is required depends on other thorns that may use this function.
Coordinates::store_inverse_jacobian. Activate/deactive storage for inverse Jacobian grid function. Whether storage is required depends on other thorns that may use this function.
Coordinates::store_volume_form. Activate/deactive storage for volume form grid function. Whether storage is required depends on other thorns that may use this function.
Interpolate::interpolator_order. Selects the order of Lagrange inter-patch interpolation.
Interpolate::interpolator_order_matter. Selects the order of ENO interpolation for those variables that contain the “matter” tag. If the default value -1 is used, this tag will be ignored and Lagrange interpolation is used also for “matter” variables.
With cell-centered AMR, the following parameters must be set:
Coordinates::stagger_outer_boundaries = yes.
Coordinates::stagger_patch_boundaries = yes.
Currently, we support the following patch systems:
Cartesian: This is standard Cartesian coordinates.
TwoPatchCartesian: This is two Cartesian patches, with one common face.
TwoPatchDistorted: This is one Cartesian patch and a second patch with distortion. It is useful for testing code.
Thornburg04: This is a 7-patch system consisting of 1 central Cartesian cube capable of AMR which is surrounded by 6 “inflated-cube” spherical grids.
Thornburg04nc: This is a 6-patch system suitable for excision. It consists of 6 “inflated-cube” spherical grids which together cover a spherical shell.
Thornburg13: This is similar to the 7 patch system Thornburg04, but with 13 patches.
CylinderInBox: This is a hollow (spherical) cylinder in a (Cartesian) box.
Sphere+Column: This is another system suitable for excision. It has an excision-type overlapping sphere and column grid.
Cylinder+Column: This is similar to the Sphere+Column system, but with a central cylindrical grid instead of a spherical one.
The following basic parameters need to be set for this patch system:
Coordinates::sphere_outer_radius. The physical radius of the outer boundary defined by the “inflated-cube” spherical grids.
Coordinates::sphere_inner_radius. The radius of the inner boundary of the “inflated-cube” spherical grids to the central cartesian cube. The size of the central Cartesian cube is controlled by this parameter.
Coordinates::h_cartesian. The coarse grid resolution \(\Delta x\) of the central Cartesian patch.
Coordinates::h_radial. The radial resolution \(\Delta r\) of the “inflated-cube” spherical grids.
Coordinates::n_angular. The number of grid cells per angular direction per patch of the “inflated-cube” spherical grids.
Coordinates::radial_stretch. Use radial stretching on the spherical inflated-cube grids yes/no.
Coordinates::stretch_rmin_1. Radius from which we start to decrease radial resolution.
Coordinates::stretch_rmax_1. Radius at which radial resolution becomes Coordinates::h_radial_1 and remains constant again
Coordinates::h_radial_1. Radial stretched target resolution.
This has the same parameters as the Thornburg04 patch system, with the exception of Coordinates::h_cartesian, which has no meaning.
The CarpetHDF5 plugin for VisIt [3] supports reading multi-patch HDF5 files out of the box. It does however require that coordinates were output along with the actual data files. Assuming that your current options for HDF5 output look like this
IOHDF5::out_vars = "HydroBase::rho"
it is sufficient to change them to
CarpetIOHDF5::one_file_per_group = "no" # this is required by multipatch IOHDF5::out_vars = "HydroBase::rho grid::coordinates"
VisIt will present the Cartesian and curvilinear parts of the grid as two different meshes and two different variables.
The approximate time line is something like this:
2009: First version of Llama: Ability to simulate vacuum binary black hole mergers.
2012: Second version of Llama: Ability to simulate general-relativistic hydrodynamics.
[1] D. Pollney, C. Reisswig, E. Schnetter, N. Dorband and P. Diener, Phys. Rev. D 83, 044045 (2011), arXiv:0910.3803.
[2] C. Reisswig, R. Haas, C. D. Ott, E. Abdikamalov, P. Mösta, D. Pollney and E. Schnetter, Phys. Rev. D 87, 064023 (2013), arXiv:1212.1191.
[3] VisIt visualization tool https://wci.llnl.gov/codes/visit/