xx ETBHaHAHA: Einstein Toolkit Interface to the BHaHAHA Apparent Horizon Finder Library

ETxBHaHAHA: Einstein Toolkit Interface to the BHaHAHAxApparent Horizon Finder Library

ETxBHaHAHA Maintainers

March 18, 2026

Contents

1 Introduction

2 What ETxBHaHAHA Needs

3 Obtaining and Compiling ETxBHaHAHA 4 ETxBHaHAHA Parameters 4.1 Overall Thorn Parameters 4.2 Per-Horizon Parameters 4.3 BBH Mode Parameters 5 Provided Variables 6 Scheduling and MPI Synchronization 6.1 Storage 6.2 Scheduled Functions 6.3 MPI Synchronization Details 7 Physics Background and Numerical Methods 7.1 Apparent Horizons and MOTS 7.2 Hyperbolic Relaxation 7.3 Numerical Implementation Details within BHaHAHA 7.3.1 Metric Data Handling and Interpolation 7.3.2 Initial Data for Hyperbolic Relaxation 7.3.3 Pseudo-Time Evolution 7.3.4 Convergence and Stopping Conditions 7.3.5 Convergence Acceleration Techniques 7.4 Binary Black Hole (BBH) Mode in ETxBHaHAHA 7.5 Diagnostics Computation 8 Output Data 8.1 Horizon Diagnostics File 8.2 Horizon Surface Shape File 9 Monitoring ETxBHaHAHA’s Status 10 Visualization 11 Accuracy and Testing 12 Examples 12.1 Single Shifted Kerr Black Hole 12.2 Binary Black Hole Inspiral and Merger (BBH Mode)

12.3 Brill-Lindquist Distorted Horizons 13 Acknowledgements 14 License 15 Parameters 16 Interfaces 16.0.1 PUBLIC GROUPS 17 Schedule

Abstract

The  ETxBHaHAHAxthorn  provides  an  Einstein

Toolkit interface to the BHaHAHAx(BlackHoles@Home Apparent Horizon Algorithm) library for finding apparent horizons in numerical relativity simulations. BHaHAHAximplements a hyperbolic flow-based approach, recasting the elliptic partial differential equation (PDE) for a marginally outer trapped surface (MOTS) as a damped nonlinear wave equation. This method is robust, versatile, and performant, leveraging a multigrid-inspired refinement strategy, an over-relaxation technique, and OpenMP parallelization. The Einstein Toolkit interface thorn, ETxBHaHAHA, allows Cactus-based simulations to utilize these capabilities for black hole identification and characterization. It manages metric data interpolation, MPI synchronization of persistent horizon data using a pack/unpack strategy for Cactus global variables, and diagnostic output.

1 Introduction Apparent horizons (AHs) are crucial for characterizing black holes and excising their interiors in numerical relativity (NR) simulations [6]. Connecting observational signatures to black hole physical parameters necessitates robust methods for identifying and tracking the BHs themselves within NR simulations. This is primarily accomplished using apparent horizon (AH) finding algorithms. An AH corresponds to the outermost marginally outer trapped surface (MOTS) of a BH, defined as a surface on which the expansion of outgoing null geodesics vanishes [2,x3,x4].

The BHaHAHAxlibrary [1] introduces the first open-source, infrastructure-agnostic library for AH finding in NR. It implements a hyperbolic flow-based approach, recasting the elliptic PDE for a MOTS as a damped nonlinear wave equation. This method preserves the full nonlinearity of the expansion equation , avoiding the need for linearization or high-quality initial guesses, and is thus robust in practice.

The ETxBHaHAHAxarrangement hosts the ETxBHaHAHAxthorn, which serves as the Einstein Toolkit (ETK) [8] interface to the standalone BHaHAHAxC library. Key responsibilities of

ETxBHaHAHAxinclude:

Interpolating spacetime metric data (components of the 3-metric and extrinsic curvature ) from the host NR code’s grid (typically Cartesian and managed by Carpet [11,x12]) onto a spherical grid suitable for BHaHAHA.

Managing persistent data for each horizon, such as previous horizon shapes, centroid locations, and search radii. This data is crucial for efficient tracking and for making good initial guesses for subsequent horizon finds.

Orchestrating the horizon finding process on distributed memory architectures, handling MPI synchronization of persistent data. Due to ETK’s Fortran legacy, C structures cannot be directly synchronized via MPI. ETxBHaHAHAxcircumvents this by packing scalar horizon data into global CCTKxREAL variables, indexed by an enumeration, and then using Cactus reduction operations for synchronization. Horizon shape data (arrays) are also synchronized via reductions.

Providing user-configurable parameters to control the behavior of the BHaHAHAxsolver and the ETxBHaHAHAxinterface.

Outputting diagnostic information, similar in style to AHFinderDirect, such as horizon area, irreducible mass, centroid coordinates, and spin estimates derived from proper circumferences.

The BHaHAHAxlibrary itself incorporates several features to enhance performance and robustness, such as a multigrid-inspired refinement strategy, an over-relaxation technique, and OpenMP parallelization for the core computations. These features are leveraged by ETxBHaHAHA. An experimental ”BBH mode” is also available for automatically triggering common horizon searches during binary black hole merger simulations.

The hyperbolic relaxation system, central to BHaHAHA, evolves the horizon shape and an auxiliary ”velocity” in pseudo-time :

where is a damping coefficient and is the relaxation wave speed (set to 1 in BHaHAHA). The system relaxes towards a steady state where , the condition for a MOTS.

This thorn is inspired by AHFinderDirect [4,x6] and aims to provide a similarly robust and efficient tool for the numerical relativity community.

2 What ETxBHaHAHAxNeeds The ETxBHaHAHAxthorn requires a 3+1 decomposed spacetime, providing components of the 3-metric and extrinsic curvature . These are typically provided by any ADM evolution thorn that populates the corresponding ADMBase grid functions.

ETxBHaHAHAxinherits from and depends on several other Einstein Toolkit thorns and arrangements for its operation, summarized in Table 1.

Tablex1: Implementations and Thorns typically inherited by ETxBHaHAHA.

11  Functionality / Data 11 Typically Provided by Thorn(s) or Arrangement
11 ADM 3-metric, Extrinsic Curvature 11 ADMBase (EinsteinBase)
11 Boundary Conditions  11 Boundary (CactusBase)

11 Grid Information  11 Grid (CactusBase), Carpet, CartGrid3D, CoordBase

11 Output Directory (IO::outxdir)  11 IOUtil (CactusBase) via shares
11 Shared Parameters (maxntheta, etc.) 11 SphericalSurface (EinsteinAnalysis) via shares

11 Metric Data Interpolation  11 e.g., AEILocalInterp, LocalInterp, CarpetInterp

11 Driver / Parallel Grid Management 11 e.g., Carpet (CarpetCode)
11 MPI Reductions  11 e.g., CarpetReduce (CarpetCode), PUGHReduce (CactusPUGH)

The metric data is expected to be provided on a Cartesian grid, potentially with adaptive mesh refinement (AMR) as managed by Carpet. ETxBHaHAHAxthen interpolates this data onto its internal spherical grid.

3 Obtaining and Compiling ETxBHaHAHA The ETxBHaHAHAxthorn is part of the ETxBHaHAHAxarrangement for the Einstein Toolkit. The core apparent horizon finding capabilities are provided by the BHaHAHAxlibrary, which is generated using NRPy+ [9,x10] and is included within the thorn’s source tree (src/BHaHAHA/). The thorn ETxBHaHAHAxitself is written in C.

To compile ETxBHaHAHA, ensure that the ETxBHaHAHAxarrangement is part of your Cactus checkout. The compilation process is standard for Einstein Toolkit thorns and uses the main Cactus make system. The README file within the thorn’s main directory provides instructions for regenerating the BHaHAHAxlibrary if needed. The BHaHAHAxlibrary components and the ETK interface code are compiled using gcc by default, as specified in the Makefiles

(src/BHaHAHA/Makefile and src/make.code.defn). These files also indicate support
for OpenMP (CFLAGS include -fopenmp -std=gnu99 -O2 -march=native), which is

enabled if detected by the compiler. No special external libraries beyond a C compiler supporting C99 (GNU extensions are used, e.g. xxtypeofxx) and OpenMP are strictly required by ETxBHaHAHAxitself, as its core dependencies (like GSL for table utilities or ADMBase functionality) are handled by the Einstein Toolkit’s build system when those thorns are active.

4 ETxBHaHAHAxParameters The behavior of the ETxBHaHAHAxthorn is controlled by several parameters defined in its param.ccl file. These parameters can be broadly categorized. Parameters are referred to by their param.ccl names.

4.1 Overall Thorn Parameters

maxxnumxhorizons

(Type: CCTKxINT, Default: 3)

Specifies the maximum number of apparent horizons that the thorn will attempt to find and track. This determines the size of various internal arrays. This parameter must be set to be at least as large as the number of horizons you intend to find. In BBH mode, it must

     be 3. The hardcoded limit is 100 (BHAHAHAxMAXxHORIZONSxSUPPORTED). Range: 0:100

findxevery

(Type: CCTKxINT, Default: 1, Steerable: ALWAYS) This integer parameter controls the execution frequency of the main horizon finding routine BHaHAHAxfindxhorizons. The finder will run when the current Cactus iteration cctkxiteration is an integer multiple of findxevery. If set to 0, ETxBHaHAHAxwill be disabled globally. Individual horizon search cadences can override this via

findxeveryxindividual. Range: 0:*

bahxverbosityxlevel

(Type: CCTKxINT, Default: 1, Steerable: ALWAYS) Controls the amount of diagnostic information printed to standard output during the horizon search.

0: Essential output only (e.g., major errors, final timing).

1: Physical summary of found horizons (default). This includes basic information about each horizon found, like its area and centroid, and BBH mode logging.

2: Full algorithmic details. This includes verbose information about the convergence process for each multigrid level, iteration counts, internal timings, MPI sync messages, and possibly debugging information.

Range: 0:2

maxxNtheta

(Type: CCTKxINT, Default: 32) The maximum number of grid points used to sample the (polar) direction on ETxBHaHAHA’s internal spherical surfaces. This corresponds to the highest resolution in the multigrid hierarchy if multigrid is enabled. It must match the last entry in bahxNthetaxarrayxmultigrid. Range: 2:128 (even values only)

maxxNphi

(Type: CCTKxINT, Default: 64) The maximum number of grid points used to sample the (azimuthal) direction on ETxBHaHAHA’s internal spherical surfaces. This corresponds to the highest resolution in the multigrid hierarchy. It must match the last entry in bahxNphixarrayxmultigrid. Range: 2:256 (even values only)

bahxnumxresolutionsxmultigrid

(Type: CCTKxINT, Default: 3) Specifies the number of multigrid levels used in the horizon search. The actual resolutions for

each level are defined in bahxNthetaxarrayxmultigrid and bahxNphixarrayxmultigrid.

Range: 1:16 (practical limit related to array sizes of bahxNthetaxarrayxmultigrid)

bahxNthetaxarrayxmultigrid[16]

(Type: CCTKxINT ARRAY, Default: -100 for all, Steerable: ALWAYS) An array defining the number of points for each multigrid level, starting from the coarsest. The number of entries actually used is determined by bahxnumxresolutionsxmultigrid. Example: [8, 16, 32]. The last entry must equal maxxNtheta. Poison value -100 indicates an unset entry. Range (for each element used): 2: * (even values only, up to

maxxNtheta)

bahxNphixarrayxmultigrid[16]

(Type: CCTKxINT ARRAY, Default: -100 for all, Steerable: ALWAYS) An array defining the number of points for each multigrid level, starting from the coarsest. The number of entries used is determined by bahxnumxresolutionsxmultigrid. Example: [16, 32, 64]. The last entry must equal maxxNphi. Poison value -100 indicates an unset entry. Range (for each element used): 2: * (even values only, up to

maxxNphi)

bahxenablexetaxvaryingxalgxforxprecisionxcommonxhorizon

(Type: CCTKxINT, Default: 0) Enables a specialized algorithm where the damping parameter varies to achieve high precision, particularly for common horizon finding. For general-purpose use, this should be disabled (0). Range: 0:1

syncxdataxacrossxranksxafterxeachxsearchxxwarningxslow

(Type: CCTKxINT, Default: 0) If set to 1, all persistent horizon data (centers, radii, shapes) will be synchronized across all MPI ranks after each horizon search sequence using MPI reductions. This allows for restarting a simulation with a different number of MPI ranks while maintaining consistent horizon history. However, this can significantly slow down the simulation (potentially by a factor of 2x for ETxBHaHAHA) as all ranks must wait for the slowest horizon search. If 0 (default), only minimal synchronization required for the thorn’s logic (e.g., for BBH mode) is performed at the beginning of a find, and rank 0 might have stale data for horizons owned by other ranks for checkpointing purposes. BHaHAHAxis generally robust enough to recover from a lost horizon if this is disabled. Range:

0:1

interpolatorxname
(Type: CCTKxSTRING, Default: "Hermite polynomial interpolation")

Specifies the name of the Cactus interpolator to be used for interpolating metric data from the simulation grid to ETxBHaHAHA’s internal spherical grid. ”Hermite polynomial interpolation” is highly recommended as it often leads to faster overall convergence in BHaHAHAxfor comparable accuracy. ”Lagrange polynomial interpolation” is an alternative. Range: Any valid Cactus interpolator name.

interpolatorxpars

(Type: CCTKxSTRING, Default: "order=3")

A string containing parameters for the chosen interpolator, passed to UtilxTableCreateFromString(). For ”Hermite polynomial interpolation”, ”order=3” is recommended. For ”Lagrange polynomial interpolation”, ”order=4” is recommended. Range: Any string accepted by the selected interpolator’s parameter table creation.

4.2 Per-Horizon Parameters These parameters are arrays, indexed from 0 to maxxnumxhorizons-1.

findxeveryxindividual[101]

(Type: CCTKxINT ARRAY, Default: -1 for all, Steerable: ALWAYS) Allows overriding the global findxevery for individual horizon i. If set to -1, uses global findxevery. If , specifies the search cadence for this horizon. If 0, finding for this horizon is disabled. Range (for each element): -1, 0, 1:*

bahxmaxxsearchxradius[101]

(Type: CCTKxREAL ARRAY, Default: 1.5 for all, Steerable: ALWAYS) The maximum coordinate radius of the spherical shell where ETxBHaHAHAxwill search for horizon i, relative to its current guessed center. This defines the outer boundary of the interpolation region for metric data. This parameter persists throughout the simulation for each horizon. Range (for each element): 0:* (must be positive)

bahxNrxinterpxmax[101]

(Type: CCTKxINT ARRAY, Default: 48 for all, Steerable: ALWAYS) Maximum number of radial points on ETxBHaHAHA’s internal 3D spherical grid for interpolating metric components for horizon i. The actual number of points used will

be determined adaptively by bahxradialxgridxcellxcenteredxsetxup based on the

current search radii, but will not exceed this value. Recommended range is 48-64. Increasing this value mainly increases the cost of the initial interpolation from the ETK grid. Range (for each element): 4:*

bahxMxscale[101]

(Type: CCTKxREAL ARRAY, Default: 1.0 for all, Steerable: ALWAYS) Characteristic mass scale for horizon i, in code units. Used to make tolerances and damping dimensionless. Use puncture bare masses or expected irreducible masses. Range (for each element): 0:* (must be positive)

bahxcflxfactor[101]

(Type: CCTKxREAL ARRAY, Default: 0.98 for all, Steerable: ALWAYS) CFL factor for the hyperbolic relaxation solver for horizon i. This controls the pseudo-timestep. A value of 0.98 is robust. Range (for each element): 0:1.0 (must be positive, technically for SSPRK3 but is safer)

bahxmaxxiterations[101]

(Type: CCTKxINT ARRAY, Default: 10000 for all, Steerable: ALWAYS) Maximum hyperbolic relaxation steps before the horizon search for horizon i is terminated, even if convergence criteria are not met. For typical tracking, 1000-10000 is sufficient. Range (for each element): 0:*

bahxThetaxL2xtimesxMxtolerance[101]

(Type: CCTKxREAL ARRAY, Default: 1e-2 for all, Steerable: ALWAYS) Convergence criterion based on the area-weighted norm of for horizon i. A value of is recommended to balance speed and accuracy for typical horizon tracking, often deferring to the norm. Convergence requires both L2 and Linf criteria to be met. Range (for each element): 0:*

bahxThetaxLinfxtimesxMxtolerance[101]

(Type: CCTKxREAL ARRAY, Default: 1e-5 for all, Steerable: ALWAYS) Convergence criterion based on the norm (maximum absolute value) of for horizon i. A value of is recommended as a typical target for accuracy. Range (for each element): 0:*

bahxetaxdampingxtimesxM[101]

(Type: CCTKxREAL ARRAY, Default: 7.0 for all, Steerable: ALWAYS) The dimensionless damping parameter for the hyperbolic relaxation for horizon i. A value of 7.0 is highly recommended. Range (for each element): 0:*

bahxKOxstrength[101]

(Type: CCTKxREAL ARRAY, Default: 0.0 for all, Steerable: ALWAYS) Strength of the Kreiss-Oliger dissipation applied during the hyperbolic relaxation for horizon i. This usually hinders convergence or accuracy for apparent horizon finding, so it is disabled (0.0) by default. Range (for each element): 0:*

bahxinitialxgridxxxcenter[101]

(Type: CCTKxREAL ARRAY, Default: 0.0 for all, Steerable: ALWAYS) The initial x-coordinate for the center of ETxBHaHAHA’s spherical search grid for horizon i. This is used at or if a horizon is lost and needs a reset. Subsequently, the center is extrapolated. Range (for each element): *:*

bahxinitialxgridxyxcenter[101]

(Type: CCTKxREAL ARRAY, Default: 0.0 for all, Steerable: ALWAYS) The initial y-coordinate for the center of ETxBHaHAHA’s spherical search grid for horizon i. Range (for each element): *:*

bahxinitialxgridxzxcenter[101]

(Type: CCTKxREAL ARRAY, Default: 0.0 for all, Steerable: ALWAYS) The initial z-coordinate for the center of ETxBHaHAHA’s spherical search grid for horizon i. Range (for each element): *:*

4.3 BBH Mode Parameters These parameters control the behavior when ETxBHaHAHAxis run in its experimental Binary Black Hole (BBH) mode.

bahxBBHxmodexenable

(Type: CCTKxINT, Default: 0) Enable (1) or disable (0) the BBH mode. When enabled, ETxBHaHAHAxwill manage the activation/deactivation of searches for two individual inspiralling black holes and a common horizon. Requires maxxnumxhorizons = 3. Range: 0:1

bahxBBHxmodexinspiralxBHxidxs[2]
(Type: CCTKxINT ARRAY, Default: [0,0])

An array of two integers specifying the zero-offset horizon indices for the two inspiralling black holes. Example: [0,1]. These indices must be unique and different from bahxBBHxmodexcommonxhorizonxidx. Range (for each element): 0:2 (Indices must be distinct and within ; typically if )

bahxBBHxmodexcommonxhorizonxidx

(Type: CCTKxINT, Default: 2) The zero-offset horizon index for the common (merged) horizon. Example: 2. This index must be unique and different from those in bahxBBHxmodexinspiralxBHxidxs. Range: 0:2 (Index must be distinct and within )

5 Provided Variables The ETxBHaHAHAxthorn declares several variables in its interface.ccl to store persistent horizon data. These variables are primarily used internally for tracking horizon evolution across time steps and for MPI synchronization, but advanced users or other thorns might find them useful. All variables are public. The index for horizon-specific variables (e.g., xxcenterxm1[h]) is 0-based in C, ranging from 0

to maxxnumxhorizons-1.

bahxprevxhorizons (Group)
(Type: CCTKxREAL ARRAY, Size: maxxnumxhorizons*maxxNtheta*maxxNphi, Distribution:
CONSTANT)

This group holds the shapes of previously found horizons. It is a single flat 1D array, but logically contains three sets of surfaces for each of the maxxnumxhorizons. For a given horizon index hidx (0 to maxxnumxhorizons-1), the shape data for is accessed with an offset:

hidx * maxxNtheta * maxxNphi + itheta * maxxNphi + iphi (assuming varies

fastest, though the actual C indexing macro IDX2 uses itheta + NUMxTHETA * iphi). prevxhorizonxm1: Surface shape of the most recently found horizon (time ).

prevxhorizonxm2: Surface shape of the second most recently found horizon (time ).

prevxhorizonxm3: Surface shape of the third most recently found horizon (time ).

These are used for extrapolating an initial guess for the current time step. All ranks maintain a

copy, synchronized by BHaHAHAxsyncxpersistentxacrossxranks.

bahxcoordxinfo (Group)

(Type: CCTKxREAL SCALAR, Distribution: CONSTANT, Effectively arrays of size maxxnumxhorizons)

Stores scalar information from previous horizon finds for each horizon. Each member is a Cactus scalar grid function that behaves as an array of size maxxnumxhorizons.

txm1[h], txm2[h], txm3[h]: Simulation times of the last three successful finds for horizon h. Initialized to -1.0 if not found.

rxminxm1[h], rxmaxxm1[h], ... rxminxm3[h], rxmaxxm3[h]: Min/max coordinate

radii (centroid-based) for horizon h from the last three finds.

xxcenterxm1[h], yxcenterxm1[h], zxcenterxm1[h], ... xxcenterxm3[h],

yxcenterxm3[h], zxcenterxm3[h]: Cartesian coordinates of the horizon’s centroid for horizon h from the last three finds. These are global Cactus coordinates.

Synchronized by BHaHAHAxsyncxpersistentxacrossxranks.

bahxguess (Group)

(Type: CCTKxINT SCALAR, Distribution: CONSTANT, Effectively an array of size maxxnumxhorizons)

usexfixedxradiusxguessxonxfullxsphere[h]: Flag (0 or 1) for horizon h. If 1,

ETxBHaHAHAxuses a simple spherical initial guess. If 0, it uses an extrapolated shape. Updated based on solver success.

Synchronized by BHaHAHAxsyncxpersistentxacrossxranks.

bahxisxhorizonxactive (Group)

(Type: CCTKxINT SCALAR, Distribution: CONSTANT, Effectively an array of size maxxnumxhorizons)

bahxhorizonxactive[h]: Flag (0 or 1) for horizon h. If 1, a search is attempted. If 0, the search is skipped. Primarily managed by BBH mode.

Synchronized by BHaHAHAxsyncxpersistentxacrossxranks.

6 Scheduling and MPI Synchronization The ETxBHaHAHAxthorn schedules its main driver function and manages storage for its interface.ccl variables as defined in schedule.ccl.

6.1 Storage The following grid function groups are declared for storage:

bahxprevxhorizons

bahxcoordxinfo

bahxguess

bahxisxhorizonxactive

Memory for these variables is allocated by the Cactus framework.

6.2 Scheduled Functions

BHaHAHAxfindxhorizons
Scheduled at CCTKxANALYSIS with options: global-early.

This is the main C language driver function for ETxBHaHAHA. It is responsible for:

Checking if a horizon search is due based on findxevery and

findxeveryxindividual.

Performing initial setup at cctkxiteration == 0, including initializing persistent variables and BBH mode flags.

Synchronizing  persistent  data

across MPI ranks using BHaHAHAxsyncxpersistentxacrossxranks (see Section 6.3

below).

Populating the BHaHAHAxparameter structure (bhahahaxparamsxandxdataxstruct)

for each active horizon, including reading persistent data from Cactus GFs (via

readxpersistentxdataxstorextoxbhahahaxparamsxandxdata).

Applying robustness improvements (e.g., adjusting CFL, multigrid resolutions for first finds).

Extrapolating initial guesses for horizon centers and search radii using

bahxxyzxcenterxrxminmax.

Executing BBH mode logic for activating/deactivating common and individual horizon searches.

For each active horizon:

Setting  up  the  radial  interpolation  grid  using

        bahxradialxgridxcellxcenteredxsetxup.

        Calling BHaHAHAxinterpolatexmetricxdata to interpolate ADM metric

components from the ETK grid to the BHaHAHAxspherical grid. This is done only on the MPI rank owning the horizon.

Invoking the core BHaHAHAxsolver (bahxfindxhorizon) on the owning MPI rank.

Processing  results:  if  successful,  updating  persistent

        Cactus GFs using writexpersistentxfromxbhahahaxparamsxandxdata and

outputting diagnostics using bahxdiagnosticsxfilexoutput. If failed, setting flags to ensure a robust guess next time.

Optionally, performing a final synchronization of all persistent data if

    syncxdataxacrossxranksxafterxeachxsearchxxwarningxslow is enabled.

The global-early option is chosen to be consistent with AHFinderDirect, aiming to find horizons early in the analysis phase.

6.3 MPI Synchronization Details Persistent horizon data (previous shapes, centers, radii, active flags) needs to be consistent across all MPI ranks for robust tracking and BBH mode operation. ETxBHaHAHAxemploys a custom MPI

synchronization strategy implemented in BHaHAHAxsyncxpersistentxacrossxranks.c:

Data Ownership: Each horizon is ”owned” by a specific MPI rank (typically ). This rank is responsible for performing the horizon find and updating its primary data.

Packing/Unpacking and Reduction: Scalar data for each horizon (from groups

bahxcoordxinfo, bahxguess, bahxisxhorizonxactive) are packed by the owner rank

into a local CCTKxREAL buffer. Non-owner ranks contribute zeros. A global sum reduction (CCTKxReduceLocArrayToArray1D) broadcasts the owner’s data to all ranks.

The BHaHAHAxPackedScalarFields enum (defined in ETxBHaHAHA.h) ensures consistent

indexing into this buffer. Horizon shape arrays (from bahxprevxhorizons) are also synchronized segment by segment using sum reductions.

This synchronization occurs at the beginning of BHaHAHAxfindxhorizons (if running in parallel) and

optionally at the end if syncxdataxacrossxranksxafterxeachxsearchxxwarningxslow is

enabled.

7 Physics Background and Numerical Methods The ETxBHaHAHAxthorn interfaces with the BHaHAHAxlibrary, which implements a specific set of methods for finding apparent horizons. The core physics and numerics are detailed in [1].

7.1 Apparent Horizons and MOTS An apparent horizon is defined as the outermost marginally outer trapped surface (MOTS). A MOTS is a 2-surface where the expansion of outgoing null geodesics vanishes. In the ADM 3+1 formalism [5], where the line element is with the lapse, the shift, and the spatial 3-metric, the expansion for a surface is [1]: where is the unit normal, , , is the extrinsic curvature, and .

7.2 Hyperbolic Relaxation BHaHAHAxrecasts the elliptic PDE (2) into a hyperbolic system by introducing a pseudo-time evolution for the surface shape and an auxiliary ”velocity” field :

where is a damping coefficient (units of , related to bahxetaxdampingxtimesxM), and is the relaxation wave speed (dimensionless, set to 1 in BHaHAHA). As , the system is driven towards a steady state where .

7.3 Numerical Implementation Details within BHaHAHA The BHaHAHAxlibrary’s numerical implementation, leveraged by ETxBHaHAHA, involves several key steps:

7.3.1 Metric Data Handling and Interpolation

Input Data and Transformation: ETxBHaHAHAxinterpolates the ADM 3-metric and extrinsic curvature (Cartesian basis) from the host Cactus grid onto a 3D cell-centered

     spherical grid used by BHaHAHAx(via BHaHAHAxinterpolatexmetricxdata.c). Internally,
     BHaHAHAx(in bahxnumgridxxexternalxinputxsetxup.c) converts these to conformal

BSSN-like variables in a spherical basis using a reference-metric formalism [13,x14,x15] to analytically handle coordinate singularities at the poles.

Precomputation of Metric Derivatives: First spatial derivatives of these BSSN-like variables (both angular and radial) are precomputed on the 3D spherical BHaHAHAx”interpolation source”

grid (interpxsrcxgfs in bahxnumgridxxinterpxsrcxsetxup.c) using sixth-order finite

differencing. This is done once per multigrid level. Angular derivatives are computed

in bahxhDDxdDxandxWxdDxinxinterpxsrcxgridxinterior.c, and radial derivatives at
boundaries in bahxapplyxbcsxrxmaxminxpartialxrxhDDxupwinding.c.

Interpolation to Trial Surface: During the pseudo-time evolution, metric quantities and their derivatives on the current trial surface are obtained via 1D Lagrange interpolation along radial spokes from the precomputed 3D BHaHAHAxgrid data

using bahxinterpolationx1dxradialxspokesxonx3dxsrcxgrid.c. This is a crucial

performance optimization.

7.3.2 Initial Data for Hyperbolic Relaxation The initial guess for the hyperbolic relaxation is determined by ETxBHaHAHAx(within

BHaHAHAxfindxhorizons.c before calling the library’s bahxinitialxdata.c):

If no prior horizon information is available (e.g., first timestep, or after a find failure

indicated by usexfixedxradiusxguessxonxfullxsphere=1), a simple spherical guess is

used, with metric data interpolated onto a full spherical volume.

Otherwise, , the grid center , and search radii are extrapolated (up to quadratic order via bahxxyzxcenterxrxminmax.c) from up to three previous successful finds. This allows the metric interpolation to be restricted to a potentially much smaller spherical shell.

The initial auxiliary field is set to by bahxinitialxdata.c to start near the condition.

7.3.3 Pseudo-Time Evolution The hyperbolic system (Eqs. ??-??) is evolved using a third-order Strong Stability Preserving Runge-Kutta (SSPRK3) scheme [16] by bahxMoLxstepxforwardxinxtime.c. The pseudo-timestep is

determined by a CFL condition (via bahxcflxlimitedxtimestepxbasedxonxhxequalsxr.c) based on

the wave speed and the proper grid spacings on the evolving surface . Angular derivatives of (via bahxrhsxeval.c) needed for are recomputed at each SSPRK3 substep.

7.3.4 Convergence and Stopping Conditions The pseudo-time evolution stops when the dimensionless expansion satisfies user-defined tolerances for both its and area-weighted norms (controlled by parameters bahxThetaxLinfxtimesxMxtolerance

and bahxThetaxL2xtimesxMxtolerance), or when bahxmaxxiterations is reached. is a

user-provided characteristic mass for the horizon.

7.3.5 Convergence Acceleration Techniques

Multigrid-Inspired Refinement: The system is first solved on a coarse angular grid. The solution is then prolonged (interpolated) to serve as the initial guess for a finer grid, and so on, up to the target resolution specified by bahxNthetaxarrayxmultigrid and

bahxNphixarrayxmultigrid.

Over-Relaxation: Periodically, an over-relaxation step (bahxoverxrelaxation.c) attempts to accelerate convergence by extrapolating the horizon shape based on its recent evolution: , where is the current shape, is a prior shape, and is an extrapolation factor. The that minimizes is adopted if it offers significant improvement.

7.4 Binary Black Hole (BBH) Mode in ETxBHaHAHA When the parameter bahxBBHxmodexenable = 1, ETxBHaHAHAxactivates specialized logic for binary black hole merger simulations, typically requiring maxxnumxhorizons = 3.

Initial Phase: Horizon searches are active for two individual inspiralling BHs (whose indices are given by bahxBBHxmodexinspiralxBHxidxs). The search for a common horizon

(index bahxBBHxmodexcommonxhorizonxidx) is initially inactive.

Common Horizon Activation: The thorn monitors the separation distance between the centroids of the two inspiral BHs and their last known maximum radii. If the condition is met, and both individual BHs have been successfully found previously, the common horizon search is activated. The initial guess for the common horizon’s center is set to the center-of-mass of the two inspiral BHs (weighted by their respective bahxMxscale values). A full sphere guess is forced for its first few attempts.

Deactivation of Individual BHs: Once the common horizon is consistently found (i.e.,

     usexfixedxradiusxguessxonxfullxsphere[commonxidx] == 0) and remains active,

searches for the individual inspiral BHs are deactivated.

To improve robustness for initial common horizon finds, if the common horizon has not been found successfully for three consecutive attempts, ETxBHaHAHAxadjusts parameters for the common horizon search: the CFL factor is reduced by 10%, a fixed-radius guess is enforced, and if at least two multigrid levels are used, the coarsest multigrid resolution is temporarily increased to match that of the second level, with the maximum iterations doubled.

7.5 Diagnostics Computation Upon convergence, various physical diagnostics are computed by the BHaHAHAxlibrary from the found MOTS , including:

Area and irreducible mass .

Coordinate centroid (area-weighted).

Minimum, maximum, and mean coordinate radii relative to the centroid.

Proper circumferences () in coordinate planes through the centroid.

Dimensionless spin estimates from ratios of these circumferences, by inverting Kerr metric relations [17].

These are handled by functions such as bahxdiagnosticsxareaxcentroidxandxThetaxnorms.c,
bahxdiagnosticsxminxmaxxmeanxradiixwrtxcentroid.c, and bahxdiagnosticsxproperxcircumferences.c.

8 Output Data

The ETxBHaHAHAxthorn, through the BHaHAHAxlibrary function bahxdiagnosticsxfilexoutput.c,

produces two main types of ASCII output files per found horizon, stored in the directory specified by the Cactus parameter IO::outxdir. These files are designed to be gnuplot-friendly.

8.1 Horizon Diagnostics File

Filename Convention: [IO::outxdir]/BHaHAHAxdiagnostics.ah[H].gp, where [H]
is the 1-based horizon number (from bhahahaxparamsxandxdata.whichxhorizon).

Content: This file contains a time series of diagnostic quantities for the specified horizon. The file is appended to at each successful find. A header is written if the file is new. Columns:

Current simulation iteration (cctkxiteration) Current simulation time (cctkxtime) x-coordinate of the horizon centroid (global grid coordinates) y-coordinate of the horizon centroid z-coordinate of the horizon centroid Minimum coordinate radius of the horizon surface, relative to its centroid Maximum coordinate radius of the horizon surface, relative to its centroid Mean coordinate radius of the horizon surface, relative to its centroid (area-weighted) Proper circumference in the xy-plane (at centroid’s z-height) Proper circumference in the xz-plane (at centroid’s y-height) Proper circumference in the yz-plane (at centroid’s x-height) Apparent horizon area Irreducible mass (max norm of expansion) (L2 norm of expansion) Spin x-component estimate from ratio (or NaN if not applicable) Spin x-component estimate from ratio (or NaN) Spin y-component estimate from ratio (or NaN) Spin y-component estimate from ratio (or NaN)

Spin z-component estimate from ratio (or NaN)

Spin z-component estimate from ratio (or NaN)

8.2 Horizon Surface Shape File

Filename Convention: [IO::outxdir]/h.t[ITER].ah[H].gp, where [ITER] is the

7-digit zero-padded Cactus iteration number, and [H] is the 1-based horizon number.

Content: This file contains the Cartesian coordinates of points on the found apparent horizon surface. The coordinates are relative to the global grid origin (i.e., the input center for BHaHAHAxplus the found shape transformed to Cartesian). Data is organized for easy plotting with gnuplot’s splot command using set pm3d map. Theta rings are separated by blank lines.

Columns:

x-coordinate

y-coordinate

z-coordinate

9 Monitoring ETxBHaHAHA’s Status The verbosity of ETxBHaHAHAxis controlled by the bahxverbosityxlevel parameter:

bahxverbosityxlevel = 0: Only essential messages are printed, primarily error messages via CCTKxVError or CCTKxVWarn, and a summary message upon successful completion of all horizon finds for an iteration (e.g., ETxBHaHAHA total elapsed time...).

bahxverbosityxlevel = 1 (Default): Prints a summary for each horizon found by the BHaHAHAxlibrary after its final multigrid level. This includes information about final expansion norms, area, irreducible mass, centroid location (global), mean/min/max radii, and estimated spin components. Also includes timing information for the BHaHAHAxsearch for each horizon. For BBH mode, it logs activation/deactivation of horizon searches.

bahxverbosityxlevel = 2: Provides the most detailed output. This includes everything from level 1, plus:

Detailed  convergence  information  from  the  BHaHAHAxlibrary

for each multigrid resolution level (iteration number, norms of , etc., printed every

    outputxdiagnosticsxeveryxnn internal iterations of BHaHAHA).

Timings for various sub-stages within BHaHAHAxfindxhorizons and BHaHAHAxinterpolatexmetricxdata, such as time spent in MPI synchronization or metric interpolation for each horizon.

Information  printed  by

    readxpersistentxdataxstorextoxbhahahaxparamsxandxdata when DEBUGxMODE

is enabled in ETxBHaHAHAxC code.

Pre-interpolation state information: guess coordinates, active flags, etc.

This level is primarily intended for debugging or detailed performance analysis.

Error messages from BHaHAHAxare converted to descriptive strings by bahxerrorxmessage() and reported via CCTKxVWarn or CCTKxVInfo, often indicating failure to find a horizon and the reason.

10 Visualization The output files from ETxBHaHAHAxare formatted for easy visualization, primarily with gnuplot.

Diagnostic Time Series: Files named BHaHAHAxdiagnostics.ah[H].gp contain time series data. For example, to plot the irreducible mass of horizon 1 over time:

xxxxgnuplot>xplotx"BHaHAHA_diagnostics.ah1.gp"xusingx2:13xwithxlinesxtitlex"M_{irr}xH1"
xxxx

Refer to Section 8.1 for column definitions.

Horizon Shapes: Files named h.t[ITER].ah[H].gp contain the (x,y,z) coordinates of the horizon surface at a specific iteration. These can be plotted as a 3D surface in gnuplot:

xxxxgnuplot>xsplotx"h.t0000000.ah1.gp"xwithxlinesxtitlex"AH1xatxt=0"
xxxx#xForxaxcolorxmapxview
xxxxgnuplot>xsetxviewxmap
xxxxgnuplot>xsplotx"h.t0000000.ah1.gp"xwithxpm3dxtitlex"AH1xatxt=0"
xxxx

Other 3D visualization tools like VisIt or ParaView can also be used by importing the ASCII data.

11 Accuracy and Testing The BHaHAHAxlibrary and its ETxBHaHAHAxinterface have undergone several tests to assess accuracy and robustness, as detailed in [1].

Comparison with AHFinderDirect: For dynamical binary black hole spacetimes (e.g., GW150914-like inspiral and merger), ETxBHaHAHAxwith default tolerances () achieves results (horizon area, centroid trajectories) consistent with AHFinderDirect. The dominant source of discrepancy at these tolerance levels is typically the error from interpolating metric data from the host NR code’s grid to the horizon finder’s internal grid (around for typical ETK setups). ETxBHaHAHAxtends to be significantly faster (approx. 2.1x) than AHFinderDirect when achieving comparable (interpolation-limited) accuracy in such dynamical scenarios.

Scale Invariance: BHaHAHAxis designed with scale-invariant convergence criteria (tolerances are relative to bahxMxscale). Tests on Brill-Lindquist two black hole initial data, where the total mass was varied over 16 orders of magnitude while keeping dimensionless quantities fixed, showed that BHaHAHAxmaintained consistent accuracy in normalized horizon areas (). In contrast, AHFinderDirect (with absolute tolerances) showed deviations at very small or very large mass scales.

Precision Tests (Three-BH Critical Radius): For the demanding Thornburg three-black-hole ”critical radius” test [4], BHaHAHAx(when run within its native BlackHoles@Home environment with high-order finite differencing/interpolation and strict tolerances, e.g., ) reproduces to at least 8 significant digits, matching results from AHFinderDirect and other specialized solvers [18]. This demonstrates the intrinsic accuracy of the underlying hyperbolic relaxation algorithm.

The default tolerance for ETxBHaHAHAx(bahxThetaxLinfxtimesxMxtolerance = 1e-5) is typically

sufficient for most production NR simulations, as accuracy is often limited by metric interpolation error rather than the horizon finder’s intrinsic precision.

12 Examples Example parameter files for using ETxBHaHAHAxcan be found in the ETxBHaHAHA/par/ directory.

12.1 Single Shifted Kerr Black Hole The parameter file shiftedxkerrxschild.par sets up a single, spinning Kerr black hole using the ShiftedKerrSchild initial data thorn. Key ETxBHaHAHAxparameters in this file:

ActiveThornsx=x"ET_BHaHAHA"
ET_BHaHAHA::max_num_horizonsx=x1
ET_BHaHAHA::bah_verbosity_levelx=x2
ET_BHaHAHA::bah_max_search_radius[0]x=x1.5
#xBHaHAHA::bah_M_scale[0]x=x1.0x(MatchesxShiftedKerrSchild::BH_mass)

12.2 Binary Black Hole Inspiral and Merger (BBH Mode) The parameter file GW150914xbaikalvacuum.par simulates a GW150914-like binary black hole system using TwoPunctures initial data and evolved with BaikalVacuum. This example demonstrates the BBH mode of ETxBHaHAHA. Key ETxBHaHAHAxparameters for BBH mode:

ActiveThornsx=x"ET_BHaHAHA"
BHaHAHA::max_num_horizonsx=x3
BHaHAHA::bah_BBH_mode_enablex=x1
BHaHAHA::bah_BBH_mode_common_horizon_idxx=x2
BHaHAHA::bah_BBH_mode_inspiral_BH_idxsx=x[0,x1]
BHaHAHA::find_everyx=x44x#xSearchxcadence
#xPer-horizonxparametersx(indicesx0,x1xforxinspiral;x2xforxcommon)
BHaHAHA::bah_M_scalex=x[0.5538,x0.4461,x1.0]
BHaHAHA::bah_initial_grid_x_center[0]x=xx4.461538
BHaHAHA::bah_initial_grid_x_center[1]x=x-5.538461
BHaHAHA::bah_max_search_radiusx=x[0.6,x0.5,x1.4]

This setup tracks the two inspiralling black holes. When they get close enough, ETxBHaHAHAxautomatically starts searching for the common horizon.

12.3 Brill-Lindquist Distorted Horizons

Files like brillxlindquistxhighlyxdistortedxhorizon.par (and variants with different mass

scales, e.g., -1e-5.par, -1e8.par) set up two Brill-Lindquist punctures with a small separation, resulting in a highly distorted common apparent horizon. These test robustness and scale invariance. BBH mode is disabled, so three horizons (common, and two small individual ones near punctures) are sought independently.

13 Acknowledgements The ETxBHaHAHAxthorn was developed by Zachariah B. Etienne. The underlying BHaHAHAxlibrary, also primarily developed by Zachariah B. Etienne, benefitted from input and contributions from Thiago Assumpxxo, Leonardo Rosa Werneck, and Samuel D. Tootle, as detailed in [1]. Special thanks to Jonathan Thornburg, whose AHFinderDirect thorn has been a cornerstone of apparent horizon finding in the numerical relativity community and served as a template and high standard for BHaHAHAxand ETxBHaHAHA. Thanks also to Roland Haas for valuable input during the development of the BHaHAHAxlibrary.

This research made use of the resources of the High Performance Computing Center at Idaho National Laboratory, which is supported by the Office of Nuclear Energy of the U.S. Department of Energy and the Nuclear Science User Facilities under Contract No. DE-AC07-05ID14517. In addition, it made use of the Falcon supercomputer, operated by the Idaho C3+3 Collaboration. Z.B.E. gratefully acknowledges support from NSF awards PHY-2110352/2508377, PHY-2409654, OAC-2004311/2227105, OAC-2411068, and AST-2108072/2227080, as well as NASA awards ISFM-80NSSC18K0538, TCAN-80NSSC18K1488, and ATP-80NSSC22K1898. TA is thankful for support from NSF grants OAC-2229652 and AST-2108269. L.R.W. gratefully acknowledges support from NASA award LPS-80NSSC24K0360. Z.B.E. and ST received additional support from NASA award ATP-80NSSC22K1898 and the University of Idaho P3-R1 Initiative.

14 License The ETxBHaHAHAxthorn is licensed under the BSD 2-Clause License:

BSDx2-ClausexLicense
Copyrightx(c)x2025,xZachariahxEtienne
Allxrightsxreserved.
Redistributionxandxusexinxsourcexandxbinaryxforms,xwithxorxwithout
modification,xarexpermittedxprovidedxthatxthexfollowingxconditionsxarexmet:
*xRedistributionsxofxsourcexcodexmustxretainxthexabovexcopyrightxnotice,xthis
xxlistxofxconditionsxandxthexfollowingxdisclaimer.
*xRedistributionsxinxbinaryxformxmustxreproducexthexabovexcopyrightxnotice,
xxthisxlistxofxconditionsxandxthexfollowingxdisclaimerxinxthexdocumentation
xxand/orxotherxmaterialsxprovidedxwithxthexdistribution.
THISxSOFTWARExISxPROVIDEDxBYxTHExCOPYRIGHTxHOLDERSxANDxCONTRIBUTORSx"ASxIS"
ANDxANYxEXPRESSxORxIMPLIEDxWARRANTIES,xINCLUDING,xBUTxNOTxLIMITEDxTO,xTHE
IMPLIEDxWARRANTIESxOFxMERCHANTABILITYxANDxFITNESSxFORxAxPARTICULARxPURPOSExARE
DISCLAIMED.xINxNOxEVENTxSHALLxTHExCOPYRIGHTxHOLDERxORxCONTRIBUTORSxBExLIABLE
FORxANYxDIRECT,xINDIRECT,xINCIDENTAL,xSPECIAL,xEXEMPLARY,xORxCONSEQUENTIAL
DAMAGESx(INCLUDING,xBUTxNOTxLIMITEDxTO,xPROCUREMENTxOFxSUBSTITUTExGOODSxOR
SERVICES;xLOSSxOFxUSE,xDATA,xORxPROFITS;xORxBUSINESSxINTERRUPTION)xHOWEVER
CAUSEDxANDxONxANYxTHEORYxOFxLIABILITY,xWHETHERxINxCONTRACT,xSTRICTxLIABILITY,
ORxTORTx(INCLUDINGxNEGLIGENCExORxOTHERWISE)xARISINGxINxANYxWAYxOUTxOFxTHExUSE
OFxTHISxSOFTWARE,xEVENxIFxADVISEDxOFxTHExPOSSIBILITYxOFxSUCHxDAMAGE.

References

[1]xxxZ. B. Etienne, T. Assumpxxo, L. R. Werneck, S. D. Tootle, BHAHAHA: A Fast, Robust Apparent Horizon Finder Library for Numerical Relativity, arXiv:2505.15912 [gr-qc] (2025).

[2]xxxR. Penrose, Gravitational Collapse and Space-Time Singularities, Phys. Rev. Lett., 14 (1965), 57–59.

[3]xxxJ. M. M. Senovilla, Trapped Surfaces, Int. J. Mod. Phys. D, 20 (2011), 2139.

    arXiv:1107.1344 [gr-qc]

[4]xxxJ. Thornburg, A Fast Apparent-Horizon Finder for 3-Dimensional Cartesian Grids in Numerical Relativity, Class. Quantum Grav., 21 (2004), 743–766. arXiv:gr-qc/0306056

[5]xxxR. Arnowitt, S. Deser, C. W. Misner, The Dynamics of General Relativity, In: Witten L. (eds) Gravitation: An Introduction to Current Research. John Wiley & Sons, New York (1962). Reprinted in Gen. Rel. Grav. 40 (2008), 1997–2027. arXiv:gr-qc/0405109

[6]xxxJ. Thornburg, Event and Apparent Horizon Finders for 3+1 Numerical Relativity, Living Rev. Relativ., 10 (2007), 3. arXiv:gr-qc/0512169

[7]xxxThe Cactus Code, http://cactuscode.org/.

[8]xxxF. Lxffler, J. Faber, E. Bentivegna, T. Bode, P. Diener, R. Haas, I. Hinder, B. C. Mundim, C. D. Ott, E. Schnetter, G. Allen, M. Campanelli, and P. Laguna, The Einstein Toolkit: A community computational infrastructure for relativistic astrophysics, Class. Quantum Grav., 29 (2012), 115001. arXiv:1111.3344 [gr-qc].

[9]xxxNRPy: Numerical Relativity in Python, https://nrpyplus.net/.

[10]xxxZ. B. Etienne, et al., NRPy+ GitHub Repository, https://github.com/nrpy/nrpy.

[11]xxxE. Schnetter, S. H. Hawley, I. Hawke, Evolutions in 3D numerical relativity using fixed mesh refinement, Class. Quantum Grav., 21 (2004), 1465. arXiv:gr-qc/0310042

[12]xxxCarpet: Adaptive Mesh Refinement for the Cactus Framework,

http://www.carpetcode.org/.

[13]xxxJ. D. Brown, Covariant formulations of spherical harmonic P N formalisms in numerical relativity, Phys. Rev. D, 80 (2009), 084042. arXiv:0908.3814 [gr-qc].

[14]xxxP. J. Montero and I. Cordero-Carrixn, Reference-metric subsystem for BSSN-inspired equations in spherical coordinates, Phys. Rev. D, 85 (2012), 124037. arXiv:1204.5377

    [gr-qc].

[15]xxxT. W. Baumgarte, P. J. Montero, I. Cordero-Carrixn, and E. Mxller, Numerical relativity in spherical coordinates: A new reference-metric BSSN formulation, Phys. Rev.

    D, 87 (2013), 044026. arXiv:1211.6632 [gr-qc].

[16]xxxS. Gottlieb, C.-W. Shu, and E. Tadmor, Strong Stability-Preserving High-Order Time Discretization Methods, SIAM Review, 43 (2001), 89–112.

[17]xxxM. Alcubierre, J. A. Gonzxlez, D. Nxxez, and J. A. Percival, Quasi-local characterization of a Kerr black hole’s spin, Phys. Rev. D, 72 (2005), 044004. arXiv:gr-qc/0411149.

[18]xxxH. K. Hui and L.-M. Lin, A multigrid apparent horizon solver for binary black hole spacetimes, Class. Quantum Grav., 42 (2025), 055008. arXiv:2404.16511 [gr-qc].

15 Parameters

11 bahxbbhxmodexcommonxhorizonxidx 10 Scope: restrictedx01 INT

11 Description: BBH mode: Horizon indices (zero-offset) for two inspiralling BHs. E.g., if the common horizon has index 2, then set bahxBBHxmodexcommonxhorizonxidx=2

11  Range  10  x01  Default: 2
11 0:2  10 0, 1, or 2.

11

11 bahxbbhxmodexenable  10 Scope: restrictedx01  INT

11 Description: 0 = disable; 1 = enable

11 Range 10 x01 Default: (none)
11 0:1  10 0=disable, 1=enable

11

11 bahxbbhxmodexinspiralxbhxidxs  10 Scope: restrictedx01  INT
11 Description: BBH mode: Horizon indices (zero-offset) for two inspiralling BHs. E.g., if the  inspiralling  BHs  have  indices  0,1  then  set BHaHAHA::bahxBBHxmodexinspiralxBHxidxs = [0,1]

11  Range  10  x01 Default: (none)
11 0:2  10 0, 1, or 2.

11 bahxcflxfactor  10 Scope: restrictedx01  REAL

11 Description: Courant-Friedrichs-Lewy (CFL) factor to set timestep within BHaHAHA hyperbolic relaxation. 0.98 robust default, even for new horizon formation (e.g., common horizons).

11  Range  10  x01 Default: 0.98
11 0:1.0  10 Positive values up to 1.0 accepted.

11

11 bahxenablexetaxvaryingxalgxforxprecisionxcommonxhorizon 10 Scope: restrictedx01  INT

11 Description: Enable eta-varying algorithm for high-precision common horizon finding. For general-purpose horizon finding, leave this disabled.

11  Range  10  x01 Default: (none)
11 0:1  10 0=disable; 1=enable

11

11 bahxetaxdampingxtimesxm  10 Scope: restrictedx01  REAL

11 Description: Hyperbolic relaxation exponential damping parameter.

11 Range 10 x01 Default: 7.0
11 0:*  10 7.0 highly recommended, seems best in GW150914 gallery example.

11

11 bahxinitialxgridxxxcenter  10 Scope: restrictedx01  REAL

11 Description: Initial x locations of horizon grid centers

11 Range 10 x01 Default: 0.0
11 *:*  10 Any value allowed

11

11 bahxinitialxgridxyxcenter  10 Scope: restrictedx01  REAL

11 Description: Initial y locations of horizon grid centers

11 Range 10 x01 Default: 0.0
11 *:*  10 Any value allowed

11

11 bahxinitialxgridxzxcenter  10 Scope: restrictedx01  REAL

11 Description: Initial z locations of horizon grid centers

11 Range 10 x01 Default: 0.0
11 *:*  10 Any value allowed

11

11 bahxkoxstrength  10 Scope: restrictedx01  REAL

11 Description: Hyperbolic relaxation Kreiss-Oliger dissipation parameter. This usually hurts more than it helps, so is disabled (0.0) by default.

11  Range  10  x01 Default: 0.0
11 0:*  10 0.0 strongly recommended

11

11 bahxmxscale  10 Scope: restrictedx01  REAL

11 Description: Mass scale for each horizon.

11  Range  10  x01 Default: 1.0
11 0:*  10 ”Estimate of the mass of each horizon, used  for bahxThetaxLinfxtimes xMxtolerance AND bahxetaxdampingxtime sxM. I would use e.g., the puncture bare masses or expected irreducible masses here.”

11 bahxmaxxiterations  10 Scope: restrictedx01  INT

11 Description: Maximum hyperbolic relaxation steps before ending the horizon search.

11  Range  10  x01 Default: 10000
11 0:*  10 Zero (disable) or positive integer; recommended:  1000–10000. Once horizon found, shouldn’t need more than ˜ 100 at lowest resolution. For  precision  horizon finding (3-horizon Brill-Lindquist for example), this parameter might need to be set in the 1e9 range

11

11 bahxmaxxsearchxradius  10 Scope: restrictedx01  REAL

11 Description: Input spherical grid(s): maximum radius to search, for all time.

11 Range 10 x01 Default: 1.5
11 0:*  10 All positive values accepted.

11

11 bahxnphixarrayxmultigrid  10 Scope: restrictedx01  INT

11 Description: Multigrid Nphi array.

11 Range 10 x01 Default: -100
11 -100  10 poison value
11 2:*:2  10 only allow even values >=2

11

11 bahxnrxinterpxmax  10 Scope: restrictedx01  INT

11 Description: Maximum number of radial points for grid(s): Number of points sampling radial direction.

11  Range  10  x01  Default: 48
11 4:*  10 All values > 4 accepted. Recommended: 48 – 64. Increasing this will increase ET interpolation overhead (usually the largest cost), but BHaHAHA itself will remain about the same speed.

11

11 bahxnthetaxarrayxmultigrid  10 Scope: restrictedx01  INT

11 Description: Multigrid Ntheta array.

11 Range 10 x01 Default: -100
11 -100  10 poison value
11 2:*:2  10 only allow even values >=2

11

11 bahxnumxresolutionsxmultigrid  10 Scope: restrictedx01  INT

11 Description: Number of resolutions provided in bahxN{theta,phi}xarrayxmultigrid[] arrays

11  Range  10  x01  Default: 3
11 -100  10 poison value
11 1:*  10 anything else

11

11 bahxthetaxl2xtimesxmxtolerance  10 Scope: restrictedx01  REAL

11 Description: Primary convergence criterion: area-weighted L2 norm of residual function Theta on horizon.

11  Range  10  x01 Default: 1e-2
11 0:*  10 1e-2 recommended, for speed vs accuracy balance

11 bahxthetaxlinfxtimesxmxtolerance  10 Scope: restrictedx01  REAL

11 Description: Secondary convergence criterion: L-infinity norm of Theta, time mass scale.

11 Range 10 x01 Default: 1e-5
11 0:*  10 1e-5 recommended

11

11 bahxverbosityxlevel  10 Scope: restrictedx01  INT

11 Description: 0 = essential only; 1 = physical summary of found horizon; 2 = full algorithmic details

11  Range  10  x01  Default: 1
11 0:2  10 0, 1, or 2.

11

11 findxevery  10 Scope: restrictedx01  INT

11 Description: Search cadence for all horizons.

11 Range 10 x01 Default: 1
11                                             10 disable BHaHAHA
11 1:*  10 any integer >= 1

11

11 findxeveryxindividual  10 Scope: restrictedx01  INT

11 Description: Search cadence for individual apparent horizons. (overrides findxevery if >= 1).

11  Range  10  x01  Default: -1
11 -1  10 use findxevery
11                                             10 disable BHaHAHA on this horizon
11 1:*  10 any integer >= 1

11

11 interpolatorxname  10 Scope: restrictedx01  STRING

11 Description: Which interpolator should I use

11 Range 10 x01Default: Hermite polynomial interpolation
11 .+  10 Any nonempty string

11

11 interpolatorxpars  10 Scope: restrictedx01  STRING

11 Description: Parameters for the interpolator

11  Range  10  x01Default: order=3
11 .*  10 ”Any  string  that UtilxTableSetFromStr ing() will take”

11

11 maxxnphi  10 Scope: restrictedx01  INT

11 Description: Maximum number of points that will sample the phi direction. For typical horizon finding, leave this to default value of 64.

11  Range  10  x01  Default: 64
11 2:256:2  10 only allow even values between 2 and 256

11

11 maxxntheta  10 Scope: restrictedx01  INT

11 Description: Maximum number of points that will sample the theta direction. For typical horizon finding, leave this to default value of 32.

11  Range  10  x01  Default: 32
11 2:128:2  10 only allow even values between 2 and 128

11

11 maxxnumxhorizons  10 Scope: restrictedx01  INT

11 Description: Maximum number of horizons for this simulation.

11 Range 10 x01 Default: 3
11 0:*  10 Zero or any positive number.

11 syncxdataxacrossxranksxafterxeachxsearchxxwarningxslow 10 Scope: restrictedx01  INT

11 Description: Enables restart from a different number of MPI ranks. WARNING: Slows BHaHAHA by >2x

11  Range  10  x01 Default: (none)
11 0:1  10 Zero (no) or one (yes).

11

11 maxnphi  10 Scope: shared from SPHERICALSURFACEx01INT

11

11 maxntheta  10 Scope: shared from SPHERICALSURFACEx01INT

11

11 nsurfaces  10 Scope: shared from SPHERICALSURFACEx01INT

11

16 Interfaces

General Implements:

bhahaha

Inherits:

admbase

boundary

grid

Grid Variables

16.0.1 PUBLIC GROUPS

11 x Group Names x 10  x Variable Names x  x01 Details x 11 x
11 bahxprevxhorizons 10  x01  compact 11 0
11                    10  prevxhorizonxm1  x01 dimensions 11 1
11                    10  prevxhorizonxm2  x01 distribution 11 CONSTANT
11                    10  prevxhorizonxm3  x01 group type 11 ARRAY
11                    10  x01  size 11 MAXxNUMxHORIZONS*MAXxNTHETA*MAXxNPHI
11                    10  x01  timelevels 11 1
11                    10  x01 vararrayxsize 11 maxxnumxhorizons
11                    10  x01 variable type 11 REAL
11  bahxcoordxinfo  10  x01  compact 11 0
11                    10  txm1  x01 dimensions 11 0
11                    10  rxminxm1  x01 distribution 11 CONSTANT
11                    10  rxmaxxm1  x01 group type 11 SCALAR
11                    10  xxcenterxm1  x01  timelevels 11 1
11                    10  yxcenterxm1  x01 vararrayxsize 11 maxxnumxhorizons
11                    10  zxcenterxm1  x01 variable type 11 REAL
11  bahxguess  10  x01  compact 11 0
11                    10 usexfixedxradiusxguessxonxfullxspherex01 dimensions 11 0
11                    10  x01 distribution 11 CONSTANT
11                    10  x01 group type 11 SCALAR
11 10 x01 timelevels 11 1
11                    10  x01 vararrayxsize 11 maxxnumxhorizons
11                    10  x01 variable type 11 INT
11 bahxisxhorizonxactive 10  x01  compact 11 0
11                    10  bahxhorizonxactive  x01 dimensions 11 0
11                    10  x01 distribution 11 CONSTANT
11                    10  x01 group type 11 SCALAR
11 10 x01 timelevels 11 1
11                    10  x01 vararrayxsize 11 maxxnumxhorizons
11                    10  x01 variable type 11 INT

11

17 Schedule This section lists all the variables which are assigned storage by thorn EinsteinAnalysis/ETxBHaHAHA. Storage can either last for the duration of the run (Always means that if this thorn is activated storage will be assigned, Conditional means that if this thorn is activated storage will be assigned for the duration of the run if some condition is met), or can be turned on for the duration of a schedule function.

Storage x

11 Always:  11 x
11 bahxprevxhorizons 11 x
11 bahxcoordxinfo  11 x
11 bahxguess  11 x
11 bahxisxhorizonxactive 11 x

11 x  11 x

11

Scheduled Functions

CCTKxANALYSIS x bhahahaxfindxhorizons x driver function for bhahaha x

11 x 11 Language: 11 c
11 x 11 Options: 11 global-early
11 x 11 Type:  11 function