## McLachlan

August 18, 2009

### Abstract

McLachlan is a free Einstein solver that uses the Cactus framework and the Einstein toolkit. This document describes the basic features of the code, and also how to obtain, build, and use the code.

### 1 McLachlan

McLachlan is a free Einstein solver that uses the Cactus framework and the Einstein toolkit. McLachlan was developed by Erik Schnetter and Peter Diener with the help of Jian Tao and Ian Hinder. It was first described in [?], where (to our knowledge) the first fully fourth order accurate black hole evolution with adaptive mesh refinement is presented. The McLachlan web pages are located at [?].

### 2 Obtaining McLachlan

McLachlan uses the Cactus Software Framework and the Einstein Toolkit. Cactus organises applications into thorns (modules) that can be maintained independently of each other. In order to use McLachlan, it is necessary to obtain Cactus as well as a set of supporting thorns.

The subsections below describe how to obtain Cactus and other necessary thorns. McLachlan contains also a shell script checkout.sh that attempts to automate this. However, this script is very basic and does not handle errors well.

#### 2.1 Tools

Cactus thorns are ususall stored in repositories that are managed by version control systems such as CVS [?], SVN (subversion) [?], or git [?].

Before getting the code, you will need to install the following software on your local system:

1. wget (or curl)

2. CVS

3. SVN

4. git

5. Perl

These are standard packages, and they should be easily available for all Linux systems.

#### 2.2 Cactus

Cactus [??] is a software framework that makes it possible to maintain parts of applications independent of each other, and combine them into an efficient code when building the application. Cactus is described at http://www.cactuscode.org/, and a new version of the of Cactus web site is currently being prepared at http://preview.cactuscode.org/download/.

To obtain Cactus itself as well as a set of basic thorns, follow the instructions at http://preview.cactuscode.org/download/. (Additional thorns specific to numerical relativity are also located elsewhere.) Don’t use a particular thorn list for this; instead, download all the basic thorn that Cactus offers.

For reference, here is a brief overview over the commands to do this:

2. chmod a+x GetCactus

3. ./GetCactus

This checks out Cactus itself (the flesh) into a new subdirectory Cactus. When asked, choose the development version of Cactus; use the default answer for all other questions.

4. cd Cactus

5. make checkout

This checks out some arrangements with basic thorns for Cactus, including the important CactusEinstein arrangement. Again, use the default answer for all questions, except when you are asked for the second time whetyer you want to quit. In this case, quit.

#### 2.3 Carpet

Carpet [???] is a driver for Cactus. A driver manages memory, handles parallelism, and performs I/O on behalf of the application. Carpet supports adaptive mesh refinement (AMR) and multi-block methods. Carpet is described at http://www.carpetcode.org/.

To obtain Carpet, follow the instructions at http://www.carpetcode.org/get-carpet.html. Please check out the Development Version, which is currently quite stable. (We are planning to release a new stable version soon.)

In particular, the commands to obtain the development version are:

1. cd Cactus

2. git clone -o carpet git://carpetcode.dyndns.org/carpet.git

3. cd arrangements

4. ln -s ../carpet/Carpet* .

(Don’t miss the dot after the Carpet* in the last line.) Note that Carpet should be checked out into the main Cactus directory, and the arrangements subdirectory needs to contain symbolic links pointing into the carpet directory.

#### 2.4 McLachlan

McLachlan [??] in an Einstein solver. It uses one of the BSSN formulations of the Einstein equations. McLachlan is described at http://www.cct.lsu.edu/~eschnett/McLachlan/, which is where you may have obtained this documentation.

To obtain McLachlan, issue the following commands:

1. cd Cactus

2. cd arrangements

3. git clone git://carpetcode.dyndns.org/McLachlan.git

Note that McLachlan needs to be checked out directly into the arrangements subdirectory.

McLachlan uses the Kranc code generation package [????]. Kranc also contains some thorns that McLachlan needs. (However, it is not necessary to run Kranc in order to use McLachlan. It is only necessary to run Kranc if McLachlan is modified.)

To obtain Kranc, issue the following commands:

1. cd Cactus

2. git clone http://www.aei.mpg.de/~ianhin/kranc.git

3. cd arrangements

4. ln -s ../kranc/Auxiliary/Cactus/KrancNumericalTools .

(Don’t miss the dot at the end of the last line.) Note that Kranc needs to be check out into the main Cactus directory, and the arrangements subdirectory needs to contain symbolic links pointing into the kranc directory.

#### 2.5 Other Thorns

All other thorns, including the public Whisky thorns, can be obtained via the GetCactus script that was downloaded above (see section 2.2):

1. cd Cactus

2. cd ..

3. ./GetCactus Cactus/arrangements/McLachlan/doc/mclachlan-public.th

Use the default answer for all questions.

#### 2.6 Consistency Check

The thorn list mclachlan-public.th lists the thorns that are necessary for a simple spacetime evolution. The thorns are grouped into arrangements. All thorns listed in this file must now be present in the arrangements subdirectory of the main Cactus directory.

### 3 Building McLachlan

Building McLachlan and the other thorns requires C, C++, and Fortran 90 compilers, MPI, as well as the BLAS, GSL, HDF5, and LAPACK libraries.

#### 3.1 Documentation

It is best to begin building Cactus with building documentation:

1. cd Cactus

2. make UsersGuide

This creates the users’ guide as doc/UsersGuide.pdf.

All Cactus commands are listed with

1. make help

#### 3.2 Option List

To build a Cactus application one needs to create an options list. This is a text file containing the configuration options that tell Cactus what compilers and compiler options to use, and where MPI and the auxiliary libraries are installed. This process is very specific to each machine and may require some trial and error.

We distribute options lists for a range of machines that we are using on the Cactus web site at http://preview.cactuscode.org/download/configfiles/. Option lists are also available together with the Simulation Factory [?] at https://svn.cct.lsu.edu/repos/numrel/simfactory/optionlists/.

#### 3.3 Building

To build a Cactus application one starts with an option list and a thorn list. The text below assumes that you have an option list called einstein-redshift-gcc.cfg.

To configure an application

1. cd Cactus

2. make sim-config options=redshift-gcc.cfg \
THORNLIST=arrangements/McLachlan/doc/mclachlan-public.th

3. make sim

This is necessary only once, or when the configuration options change. This will create an application called sim; of course, the name could also be different. Different applications, e.g. with different options or different thorn lists, can exist side by side.

To build the application:

1. cd Cactus

2. make sim -j4

The make option -j4 builds 4 files at the same time. Use this option if you have several processors available; this will speed up building the application. The executable is called cactus_sim and is placed in the exe subdirectory.

#### 3.4 Cleaning

You can also clean the application, removing all object files but keeping the configuration options and thorn list:

1. cd Cactus

2. make sim-realclean

### 4 Running McLachlan

To run the McLachlan code, one needs a parameter file. Parameter files select which thorns are activated at run time, and what values the thorns’ run-time parameters have. They typically have a .par suffix. Below, we use the parameter file ks-mclachlan-public.par.

Cactus applications are started like a regular MPI application. The exact mechanism depends on the particular MPI implementation. On Redshift, the command is

1. cd Cactus

2. mkdir simulations

3. cd simulations

4. env OMP_NUM_THREADS=1 mpirun -np 1 ../exe/cactus_sim \    ../arrangements/McLachlan/doc/ks-mclachlan-public.par

This parameter file simulates a single, stationary, spinning black hole in Kerr-Schild coordinates. It requires about 4 GByte of RAM to run.

Note: If the options list enables OpenMP, then the Cactus application will be multi-threaded. Multi-threading can improve performance and reduce memory consumption, especially when many ($>100$) cores are used. However, it is usually a bad idea to over-subscribe cores by having too many threads per node. It is usually best to choose both the number of MPI processes per node and the number of OpenMP threads per process such that their product equals the number of cores on a node. The way in which these numbers are chosen depend on the MPI implementation.