xx WaveToyOpenCL

WaveToyOpenCL

Erik Schnetter <eschnetter@perimeterinstitute.ca>

May 15, 2012

Abstract

This thorn implements WaveToy, solving the scalar wave equation (in a Euclidean, i.e.xtrivial geometry). The thorn is implemented in OpenCL, with some wrapper code in C++.

1 Introduction

This thorn WavetoyOpenCL solves the scalar wave equation, the same equation solved in thorn WaveToy and its companions written in other languages. Its major purpose is to serve as high-level example of using OpenCL in Cactus. It is purposefully written to be simple and easy to understand; for example, there are no parameters to choose different types of initial or boundary conditions.

2 Thorn Structure

We assume the reader is familiar with the structure of a Cactus thorn written e.g.xin C or C++. An OpenCL thorn is slightly more complex because it (1) has to describe when and what data are moved between host and device, and (2) Cactus does not (yet?) support calling OpenCL code directly; some boilerplate code is necessary.

2.1 Schedule Declarations

Thorn WaveToyOpenCL relies on thorn Accelerator to handle data movement between host and device. This does not need to be managed explicitly; instead, the file schedule.ccl describes which routines are executed where (host or device), and which variables or groups are read or written.

The location where a scheduled routine is ultimately executed needs to be described in a Device= schedule tag. The set of variables that are read and/or written needs to be declared in READS and WRITES schedule statements. For example, this is the schedule item for the evolution routine of thorn

WaveToyOpenCL:

SCHEDULExWaveToyOpenCL_EvolxATxevol
{
xxLANG:xxxC
xxTAGS:xxxDevice=1
xxWRITES:xWaveToyOpenCL::Scalar
}x"Evolvexscalarxwave"

This indicates that this routine executes on the device, i.e.xits kernel is implemented in OpenCL. Note that, in OpenCL, both CPU and GPU count as devices (thus every routine written in OpenCL counts as executing on a device, even if the device happens to be the CPU).

This also indicates that this routine writes (i.e.xdefines) the grid function group Scalar, without looking at (the current timelevel of) this group.

2.2 Schedule Routines

Executing OpenCL code requires some boilerplate: One needs to choose an OpenCL platform and device, needs to compile the code (from a C string), needs to pass in arguments, and finally needs to execute the actual kernel code. Thorn OpenCLRunTime provides a simple helper routine for these tasks that can be used e.g.xas follows:

xxcharxconstx*constxgroups[]x=x{
xxxx"WaveToyOpenCL::Scalar",
xxxxNULL};
xxintxconstximin[]x=x{cctk_nghostzones[0],
xxxxxxxxxxxxxxxxxxxxxxcctk_nghostzones[1],
xxxxxxxxxxxxxxxxxxxxxxcctk_nghostzones[2]};
xxintxconstximax[]x=x{cctk_lsh[0]x-xcctk_nghostzones[0],
xxxxxxxxxxxxxxxxxxxxxxcctk_lsh[1]x-xcctk_nghostzones[1],
xxxxxxxxxxxxxxxxxxxxxxcctk_lsh[2]x-xcctk_nghostzones[2]};
xxstaticxstructxOpenCLKernelx*kernelx=xNULL;
xxcharxconstx*constxsources[]x=x{"",xOpenCL_source_WaveToyOpenCL_evol,xNULL};
xxOpenCLRunTime_CallKernel(cctkGH,xCCTK_THORNSTRING,x"evol",
xxxxxxxxxxxxxxxxxxxxxxxxxxxsources,xgroups,xNULL,xNULL,xNULL,x-1,
xxxxxxxxxxxxxxxxxxxxxxxxxxximin,ximax,x&kernel);

The function OpenCLRunTimexCallKernel performs the following steps:

Choose a platform and device, compile the kernel code, and memoise (remember) the kernel for the next call

Pass a set of grid functions to the kernel routine

Parallelise the kernel over a certain set of grid points (this is similar e.g.xto an OpenMP parallelisation, except that OpenCL devices may offer much more parallelism)

Call the kernel Consequently, one needs to define the set of grid functions to be passed to the OpenCL kernel (groups, a C array terminated by NULL), needs to define the iteration bounds (imin and imax), and needs to provide the actual source code (sources, a C array terminated by NULL).

Note that the first element of sources contains declarations (it is empty here), while the second element contains the actual kernel code (see thorn OpenCLRunTime). See thorn OpenCL how the string

OpenCLxsourcexWaveToyOpenCLxevol is generated from .cl files).

The actual kernel is contained in the file evol.cl, and should be readable with some C knowledge. LCxLOOP3 is a macro that parallelises a loop, similar to the macros provided by thorn LoopControl.

3 Parameters

11 verbose 10 Scope: private01 BOOLEAN

11 Description: Output progress information

11            10              01  Default: no

11

11 wavelength 10 Scope: private01  REAL

11 Description: Wavelength of initial data

11 Range 10              01  Default: 1.0

11 *:*  10

11

4 Interfaces

General Implements: wavetoyopencl

Grid Variables 4.0.1 PRIVATE GROUPS

11 x Group Names x 10 x Variable Names x01 Details x 11 x
11  scalar  10                    01  compact 11 0
11                    10  u  01 description 11 Scalar
11                    10                    01 dimensions 11 3
11                    10                    01 distribution 11 DEFAULT
11                    10                    01 group type 11 GF
11                    10                    01 timelevels 11 3
11                    10                    01 variable type 11 REAL

11

Uses header: OpenCLRunTime.h

5 Schedule This section lists all the variables which are assigned storage by thorn CactusExamples/WaveToyOpenCL. Storage can either last for the duration of the run (Always means that if this thorn is activated storage will be assigned, Conditional means that if this thorn is activated storage will be assigned for the duration of the run if some condition is met), or can be turned on for the duration of a schedule function.

Storage x

11 Always: 11 x
11 Scalar[3] 11 x

11 x  11 x

11

Scheduled Functions

CCTKxINITIAL x wavetoyopenclxinit x initialise scalar wave x

11 x 11 Language: 11 c
11 x 11 Tags:  11 device=1
11 x 11 Type:  11 function
11 x 11 Writes: 11 wavetoyopencl::scalar

CCTKxEVOL x wavetoyopenclxevol x evolve scalar wave x

11 x 11 Language: 11 c
11 x 11 Tags:  11 device=1
11 x 11 Type:  11 function
11 x 11 Writes: 11 wavetoyopencl::scalar

11

CCTKxEVOL x wavetoyopenclxboundary x boundary conditions for scalar wave x

11 x 11 After:  11 wavetoyopenclxevol
11 x 11 Language: 11 c
11 x 11 Reads:  11 wavetoyopencl::scalar
11 x 11 Sync:  11 scalar
11 x 11 Tags:  11 device=1
11 x 11 Type:  11 function
11 x 11 Writes: 11 wavetoyopencl::scalar