Release Announcement

We are pleased to announce the fourth release (code name "Maxwell") of the Einstein Toolkit, an open, community developed software infrastructure for relativistic astrophysics. This release includes substantial changes to the underlying AMR infrastructure Carpet and the Simfactory tool. GRHydro now officially supports magnetohydrodynamics. In addition, many errors have been corrected since the previous release in April 2011.

The Einstein Toolkit is a collection of software components and tools for simulating and analyzing general relativistic astrophysical systems. It builds on numerous software efforts in the numerical relativity community including CactusEinstein, the Carpet AMR infrastructure, and the relativistic hydrodynamics code GRHydro. The Cactus Framework is used as the underlying computational infrastructure providing large-scale parallelization, general computational components, and a model for collaborative, portable code development. The toolkit includes modules to build complete codes for simulating black hole spacetimes as well as systems governed by relativistic hydrodynamics.

The Einstein Toolkit uses a distributed software model, and its different modules are developed, distributed, and supported either by the core team of Einstein Toolkit Maintainers, or by individual research groups. Where modules are provided by external groups, the Einstein Toolkit Maintainers provide quality control for modules for inclusion in the toolkit and help coordinate support. The Einstein Toolkit Maintainers currently involve postdocs and faculty from five different institutions, and host weekly meetings that are open for anyone to join.

Guiding principles for the design and implementation of the toolkit include: open, community-driven software development; well thought-out and stable interfaces; separation of physics software from computational science infrastructure; provision of a complete working production code; and training and education for a new generation of researchers.

For more information about using or contributing to the Einstein Toolkit, or to join the Einstein Toolkit Consortium, please visit our web pages at <http://einsteintoolkit.org>.

The Einstein Toolkit is primarily supported by NSF 0903973/0903782/0904015 (CIGR), and also by NSF 0701566/0855892 (XiRel), 0721915 (Alpaca), 0905046/0941653 (PetaCactus), and 0710874 (LONI).

The Einstein Toolkit thorns contain over 130 regression test cases. On a large portion of the tested machines, all of these testsuites pass, using both MPI and OpenMP parallelisation.

The changes between this and the previous release include:

  • Einstein Toolkit: All test cases pass on almost all of the tested twenty production and development machines.
  • Carpet
    • Significant internal development
    • Grid structure is handled in a more efficient manner, leading to improved parallel scalability
    • Grid structure output supports multipatch
    • Improvements to OpenMP parallelism in Carpet
    • Support for cell-centering
    • Timers are hierarchical, which makes it much easier to see where the time is spent in a simulation. (Use the parameter output_timer_tree_every to output the timer tree to standard output.)
    • A backtrace file is written to the output directory when the simulation code crashes. (Note that you probably need to add the -rdynamic option to CFLAGS and CXXFLAGS for the backtrace symbols to be interpreted correctly.)
  • CarpetIOHDF5: There are now parameters to select whether symmetry, boundary, or buffer points should be output.
  • CarpetRegrid2: Supports full AMR based on a regridding criterion
  • SimFactory
    • Internally completely rewritten
    • Machine database and optionlists updated
    • Can run the Cactus test suites as part of a job in a queuing system
    • Optionlists enable instruction vectorization by default, which can significantly improve performance. (This affects those thorns that explicitly use vectorisation, including McLachlan and Carpet.)
    • Supports parameter file scripts .rpar, written e.g. in Perl or Python.
    • Uses the Intel compiler by default on Kraken and Hopper
  • Cactus
    • CUDA support added for GPU computing
    • Parameters can be used in STORAGE specifications in schedule.ccl files
    • Multi-line parameter values can contain comments - this makes it easier to comment out entries
    • Mac OS 10.7 (Lion) is supported
    • CCTK_GFINDEX3D checks index against array bounds in debug configurations
    • Standard output of Cactus build process is much more compact
  • McLachlan
    • Various performance improvements
    • BSSN has instruction vectorisation enabled by default for improved speed
  • GRHydro
    • Supports GR-MHD
  • WeylScal4: parallelised via OpenMP
  • TimerReport: "top timers" given as min/max/mean across all processes (instead of just from the root process)
  • ADMBase
    • ADM variables have flat boundary condition applied
    • Shift vector has storage by default
  • TwoPunctures: Outputs a BBH metadata file, as used by NINJA / NRAR projects
  • Vectors: New thorn for vectorizing code, can significantly improve performance of codes that use it
  • Cauchy Characteristic Extraction and the PITT Null Code are included.
    • The Pitt code implements a robust fully nonlinear characteristic evolution scheme for the Einstein equations for asymptotically flat spacetimes.
    • Included in the code is the gauge invariant calculation of the Bondi News function at future null infinity.
    • Included in the code are thorns that implement Cauchy Characteristic extraction, where Cauchy evolutions (McLachlan) provide boundary data for a characteristic evolution. This allows for the unambiguous calculation of the gravitational waveform from merging BBH spacetimes.
  • FFTW3 library has been added to the ET
  • Kranc
    • Thorns can be generated including a Jacobian transformation of all derivatives - this means they can be used with multi-patch grids
    • Improvements to instruction vectorization
    • Reduced instruction cache usage, selected via VECTORISE_INLINE = yes/no in the optionlist. This can result in large speed increases.
    • Generated thorns check that there are sufficient ghost and boundary points for the finite differencing stencils
    • Improved error detection

All repositories participating in this release carry a branch ET_2011_10 marking this release. These release branches will be updated if severe errors are found.

This release has been tested on about twenty systems and architectures, including the following:

  • Workstations (AMD, Linux)
  • MacBook Pro notebook (Intel, Mac OS X)
  • Lenovo T61p notebook (Intel, Linux)
  • Datura, AEI (cluster, Linux)
  • Kraken, NICS (Cray XT5, Linux)
  • Philip, LSU (Intel cluster, Linux)
  • Queen Bee, LONI (Intel cluster, Linux)
  • Ranger, TACC (AMD cluster, Linux)
  • Lonestar, TACC (Intel cluster, Linux)
  • Trestles, SDSC (AMD cluster, Linux)
  • Hopper, NERSC (Cray XE6, Linux)
  • Orca, Sharcnet (AMD cluster, Linux)

The Simulation Factory contains ready-to-use configuration details for more than 40 additional systems, including most HPC systems at LONI, NERSC, RZG, and XSEDE (TeraGrid).

For more details about the new release, including a detailed list of changes, fixed and remaining issues and how to upgrade to the new version, visit the Detailed Release Announcement Wiki Page

The Einstein Toolkit web site contains online documentation for its thorns, and pointers for using it to build your own code. There is also a tutorial that explains how to download, build, and run the code for a simple binary black hole evolution. We invite you to join our mailing list <users@einsteintoolkit.org>.

On behalf of the Einstein Toolkit Consortium: the "Maxwell" Release Team

   Eloisa Bentivegna
   Tanja Bode
   Peter Diener
   Roland Haas
   Ian Hinder
   Frank Löffler
   Bruno Mundim
   Christian D. Ott
   Erik Schnetter
   Eric Seidel

October 25, 2011