xx EllPETSc

EllPETSc

Paul Walker, Gerd Lanfermann

Date

Abstract

EllPETSc provides 3D elliptic solvers for the various classes of elliptic problems defined in EllBase. EllPETSc using the “Portable, Extendable Toolkit for Scientific computation” (PETSc) by Argonne National Lab. PETSc is a suite of routines and data structures that can be employed for solving partial differential equations in parallel. EllPETSc t is called by the interfaces provided in EllBase.

1 Purpose

This thorns provides sophisticated solvers based on PETSc libraries. It supports all the interfaces defined in EllBase. At this point is not optimized for performance. Expect improvements as we develop the elliptic solver arrangement.

This thorn provides No Pizza

No Wine

peace

2 Technical Details

This thorn supports three elliptic problem classes: LinFlat for a standard 3D cartesian Laplace operator, using the standard 7-point computational molecule. LinMetric for a Laplace operator derived from the metric, using 19-point stencil. LinConfMetric for a Laplace operator derived from the metric and a conformal factor, using a 19-point stencil. The code of the solvers differs for the classes and is explained in the following section.

2.1 Installing PETSc

PETSc needs to be installed on the machine and the environment variables PETSCxARCH and PETSCxDIR have to be set to compile EllPETSc. PETSc can obtained for free at http://www-fp.mcs.anl.gov/petsc/. Cactus needs to be compiled with MPI. While PETSc can be compiled for single processor mode (without MPI), Cactus has only been tested and used with the parallel version of PETSc requiring MPI. For detailed information in how to install PETSc refer to the documentation.

2.2 LinFlat

For this class we employ the the 7-point stencil based on only. These values are constant at each gridpoint.

2.3 LinMetric

For this class the standard 19-point stencil is initialized, taken the underlying metric into account. The values for the stencil function differ at each gridpoints.

2.4 LinConfMetric

For this class the standard 19-point stencil is initialized, taken the underlying metric and its conformal factor into account. The values for the stencil function differ at each gridpoints.

2.5 Interfacing PETSc

The main task when interfacing PETSc consists of transferring data from the Cactus parallel data structures (gridfunctions) to the parallel structures provided by PETSc.

Here we explain the main steps, to be read with the code at hands.

The indices imin,imax x are calculated and describe the starting/ending points in 3D local index space: ghostzones are not included here.

A linear global index is calculated describing the starting/ending points in linear global index space. Ghostzone are not included here.

A lookup gridfunction wsp is loaded identifying the 3D local index with the linear global index. Values of zero indicate boundaries.

PETSc matrices/vectors are created specifying the linear size: global endpoint - global startpoint. For the elliptic class LinFlat the stencil functions are initialized with the standard 7-point stencil, the class LinMetric and LinConfMetric require a more sophisticated treatment described later. Looping over the processor local grid points (in 3D local index space) the PETSc vectors and coefficient matrix is loaded if no boundary is present (wsp[i,j,k] not equal zero.); Starting the PETSc vector and matrix assembly, nested for performance as recommended by PETSc. Creation of the elliptic solver context and setting of options, followed by the call to the PETSc solver. Upon completion of the solve, the PETSc solution has to transferred to the Cactus data structures.

3 Comments The sizes of the arrays Mlinear for the coefficient matrix and Nsource are passed in the solver. A storage flag is set if these variables are of a sized greater 1. In this case, the array can be accessed.

4 General remarks: PETSc within Cactus 4.1 PETSc in src code

Use PETSc as normal, Use the PUGH communicator if a routine needs a communictor. On first pass, you need to make a call to PETScSetCommWorld() and PetscInitialize() to set the PETSc communicator and initialize PETSc.

This could be a seperate routine scheduled early in schedule.ccl at BASEGRID eg. PetscInitialize() requires the commandline parameters as input. It allows you to pass through the flags, etc. (I have not ried this feature.) Initialize the PETSc communicator with the Cactus communicator. You end up having code like this:

xx/*xThexpughxExtensionxhandlex*/x
xxpGHx*pughGH;
xx/*xGetxthexlinkxtoxpughxExtensionx*/
xxpughGHx=x(pGH*)GH->extensions[CCTK_GHExtensionHandle("PUGH")];
xxifx(first_trip==0)
xx{
xxxxintxargc;
xxxxcharx**argv;
xxxx/*xGetxthexcommandlinexargumentsx*/
xxxxargcx=xCCTK_CommandLine(&argv);
xxxx/*xSetxthexPETScxcommunicatorxtoxsetxof
xxxxxxxPUGHxandxinitializexPETScx*/
xxxxierrx=xPetscSetCommWorld(pughGH->PUGH_COMM_WORLD);xCHKERRA(ierr);
xxxxPetscInitialize(&argc,&argv,NULL,NULL);
xxxxCCTK_INFO("PETScxinitialized");
xx}

4.2 make.code.defn

You need to tell Cactus to look for the PETSc includes: In the file make.code.defn define the SRCS (sources) as explained in the dcoumentation and add a lien for SUSxINCxDIR which lets Cactus look for additional includes, eg.:

SYS_INC_DIRSx+=x$(PETSC_DIR)x$(PETSC_DIR)/includexx\
xxxxxxxxxxxxxxxx$(PETSC_DIR)/bmake/$(PETSC_ARCH)

4.3 make.configuration.defn

This file is not created by the when you use Cactus to create a new thorn by ”gmake newthorn”. For a template PETSc configuration file, have a look in ./CactusElliptic/EllPETSc/src/make.configuration.defn.

The first section checks if PETSCxDIR/PETSCxLIB are set. If they are not, the configuration process will be interrupted (otherwise you have to wait to the end of the compilation to find out that your program won’t link).

Second section specifies the standard PETSc libs. eg.:

PETSC_LIB_DIRx=x$(PETSC_DIR)/lib/libg/$(PETSC_ARCH)
PETSC_LIBSxxxx=xpetsctsxpetscsnesxpetscslesxpetscdm

Third section adds platform dependent file, by checking PETSCxARCH and assigning the appropriate libs.

In the end the variables are assigned to the variables that Cactus make process is using (note the incremental assignment ”+=”)

LIBDIRSxxxx+=x$(PETSC_LIB_DIR)x$(X_LIB_DIR)
LIBSxxxxxxx+=x$(PETSC_LIBS)x$(PLATFORM_LIBS)xX11
EXTRAFLAGSx+=x-I$(PETSC_DIR)/include

5 Parameters

11 petscxcoeffxtoxone 10 Scope: private01  BOOLEAN

11 Description: Divide each line of the matrix by the central value?

11                10              01  Default: no

11

11 petscxkspxtype 10 Scope: private01  STRING

11 Description: Which Krylov subspace method to use

11 Range 10 01 Default: KSPBCGS
11 KSPCR  10 pcr
11 KSPCG  10 cg
11 KSPCGS  10 cgs
11 KSPBCGS  10 bcgs
11 KSPLSQR  10 lsqr
11 KSPGMRES 10 gmres
11 KSPTCQMR 10 tcqmr
11 KSPTFQMR 10 tfqmr

11 KSPCHEBYCHEV

10 chebyshev

11 KSPCHEBYSHEV

10 chebyshev

11 KSPRICHARDSON

10 richardson

11

11 petscxnablaform10 Scope: private01  KEYWORD

11 Description: PETSC nabla form

11  Range  10              01  Default: down

11 up  10
11 down  10

11

11 petscxpcxtype 10 Scope: private01  KEYWORD

11 Description: Which preconditioner method to use

11 Range 10 01 Default: PCJACOBI
11 PCNONE  10 none
11 PCJACOBI  10 jacobi
11 PCBJACOBI 10 bjacobi
11 PCICC  10 icc
11 PCILU  10 ilu
11 PCASM  10 asm
11 PCLU  10 lu

11

11 petscxreuse  10 Scope: private01  BOOLEAN

11 Description: Reuse parts of the PETSc structure

11                10              01  Default: no

11 petscxverbose 10 Scope: private01  KEYWORD

11 Description: PETSc verbose output

11 Range 10 01 Default: yes
11 no  10 No output
11 yes  10 Some output
11 debug  10 Tons of output

11

11 domain  10 Scope: shared from GRID01KEYWORD

11

6 Interfaces

General Implements: ellpetsc

Inherits: ellbase driver

Grid Variables

6.0.1 PRIVATE GROUPS

11 x Group Names x 10 x Variable Names x01 Details x 11 x
11 petscworkspace 10                    01  compact 11 0
11 10 wsp 01 description 11 Workspace for the elliptic PETSc solver
11                    10                    01 dimensions 11 3
11                    10                    01 distribution 11 DEFAULT
11                    10                    01 group type 11 GF
11                    10                    01 timelevels 11 1
11                    10                    01 variable type 11 REAL

11

Uses header: EllBase.h 7 Schedule This section lists all the variables which are assigned storage by thorn CactusElliptic/EllPETSc. Storage can either last for the duration of the run (Always means that if this thorn is activated storage will be assigned, Conditional means that if this thorn is activated storage will be assigned for the duration of the run if some condition is met), or can be turned on for the duration of a schedule function.

Storage x

11 Always:  11 x
11 petscworkspace 11 x

11 x  11 x

11

Scheduled Functions

CCTKxBASEGRID x ellpetscxregister x register the petsc solvers x

11 x 11 Language: 11 c
11 x 11 Type:  11 function