We are pleased to announce the fourth release (code name "Maxwell") of the Einstein Toolkit,
an open, community developed software infrastructure for relativistic astrophysics. This
release includes substantial changes to the underlying AMR infrastructure Carpet and the
Simfactory tool. GRHydro now officially supports magnetohydrodynamics. In addition, many
errors have been corrected since the previous release in April 2011.
The Einstein Toolkit is a collection of software components and tools for simulating and
analyzing general relativistic astrophysical systems. It builds on numerous software
efforts in the numerical relativity community including CactusEinstein, the Carpet AMR
infrastructure, and the relativistic hydrodynamics code GRHydro. The Cactus Framework is
used as the underlying computational infrastructure providing large-scale parallelization,
general computational components, and a model for collaborative, portable code development.
The toolkit includes modules to build complete codes for simulating black hole spacetimes
as well as systems governed by relativistic hydrodynamics.
The Einstein Toolkit uses a distributed software model, and its different modules are
developed, distributed, and supported either by the core team of Einstein Toolkit
Maintainers, or by individual research groups. Where modules are provided by external
groups, the Einstein Toolkit Maintainers provide quality control for modules for inclusion
in the toolkit and help coordinate support. The Einstein Toolkit Maintainers currently
involve postdocs and faculty from five different institutions, and host weekly meetings
that are open for anyone to join.
Guiding principles for the design and implementation of the toolkit include: open,
community-driven software development; well thought-out and stable interfaces; separation
of physics software from computational science infrastructure; provision of a complete
working production code; and training and education for a new generation of
For more information about using or contributing to the Einstein Toolkit, or to join the
Einstein Toolkit Consortium, please visit our web pages at <http://einsteintoolkit.org>.
The Einstein Toolkit is primarily supported by NSF 0903973/0903782/0904015 (CIGR), and
also by NSF 0701566/0855892 (XiRel), 0721915 (Alpaca), 0905046/0941653 (PetaCactus), and
The Einstein Toolkit thorns contain over 130 regression test cases. On a large portion
of the tested machines, all of these testsuites pass, using both
MPI and OpenMP parallelisation.
The changes between this and the previous release include:
- Einstein Toolkit: All test cases pass on almost all of the tested twenty production
and development machines.
- Significant internal development
- Grid structure is handled in a more efficient manner, leading to improved
- Grid structure output supports multipatch
- Improvements to OpenMP parallelism in Carpet
- Support for cell-centering
- Timers are hierarchical, which makes it much easier to see where the time is
spent in a simulation. (Use the parameter output_timer_tree_every to output the timer
tree to standard output.)
- A backtrace file is written to the output directory when the simulation code
crashes. (Note that you probably need to add the -rdynamic option to CFLAGS and
CXXFLAGS for the backtrace symbols to be interpreted correctly.)
- CarpetIOHDF5: There are now parameters to select whether symmetry, boundary, or
buffer points should be output.
- CarpetRegrid2: Supports full AMR based on a regridding criterion
- Internally completely rewritten
- Machine database and optionlists updated
- Can run the Cactus test suites as part of a job in a queuing system
- Optionlists enable instruction vectorization by default, which can significantly
improve performance. (This affects those thorns that explicitly use vectorisation,
including McLachlan and Carpet.)
- Supports parameter file scripts .rpar, written e.g. in Perl or Python.
- Uses the Intel compiler by default on Kraken and Hopper
- CUDA support added for GPU computing
- Parameters can be used in STORAGE specifications in schedule.ccl files
- Multi-line parameter values can contain comments - this makes it easier to
comment out entries
- Mac OS 10.7 (Lion) is supported
- CCTK_GFINDEX3D checks index against array bounds in debug configurations
- Standard output of Cactus build process is much more compact
- Various performance improvements
- BSSN has instruction vectorisation enabled by default for improved speed
- WeylScal4: parallelised via OpenMP
- TimerReport: "top timers" given as min/max/mean across all processes (instead of just
from the root process)
- ADM variables have flat boundary condition applied
- Shift vector has storage by default
- TwoPunctures: Outputs a BBH metadata file, as used by NINJA / NRAR projects
- Vectors: New thorn for vectorizing code, can significantly improve performance of
codes that use it
- Cauchy Characteristic Extraction and the PITT Null Code are included.
- The Pitt code implements a robust fully nonlinear characteristic evolution scheme
for the Einstein equations for asymptotically flat spacetimes.
- Included in the code is the gauge invariant calculation of the Bondi News
function at future null infinity.
- Included in the code are thorns that implement Cauchy Characteristic extraction,
where Cauchy evolutions (McLachlan) provide boundary data for a characteristic
evolution. This allows for the unambiguous calculation of the gravitational waveform
from merging BBH spacetimes.
- FFTW3 library has been added to the ET
- Thorns can be generated including a Jacobian transformation of all derivatives -
this means they can be used with multi-patch grids
- Improvements to instruction vectorization
- Reduced instruction cache usage, selected via VECTORISE_INLINE = yes/no in the
optionlist. This can result in large speed increases.
- Generated thorns check that there are sufficient ghost and boundary points for
the finite differencing stencils
- Improved error detection
All repositories participating in this release carry a branch ET_2011_10
marking this release. These release branches will be updated if severe errors are
This release has been tested on about twenty systems and architectures, including the
- Workstations (AMD, Linux)
- MacBook Pro notebook (Intel, Mac OS X)
- Lenovo T61p notebook (Intel, Linux)
Datura, AEI (cluster, Linux)
(Cray XT5, Linux)
(Intel cluster, Linux)
Queen Bee, LONI
(Intel cluster, Linux)
(AMD cluster, Linux)
TACC (Intel cluster, Linux)
Trestles, SDSC (AMD cluster,
Hopper, NERSC (Cray XE6,
Orca, Sharcnet (AMD cluster,
The Simulation Factory contains ready-to-use
configuration details for more than 40 additional systems, including most HPC systems at
RZG, and XSEDE
For more details about the new release, including a detailed list of changes, fixed and
remaining issues and how to upgrade to the new version, visit the Detailed
Release Announcement Wiki Page
The Einstein Toolkit web site contains online documentation for its thorns, and pointers
for using it to build your own code. There is also a tutorial that explains how to
download, build, and run the code for a simple binary black hole evolution. We invite you
to join our mailing list <email@example.com>.
On behalf of the Einstein Toolkit Consortium: the "Maxwell" Release Team
Christian D. Ott
October 25, 2011