xx OpenCLRunTime

OpenCLRunTime

Erik Schnetter <eschnetter@perimeterinstitute.ca>

May 17, 2012

Abstract

Executing OpenCL kernels requires some boilerplate code: One needs to choose an OpenCL platform and device, needs to compile the code (from a C string), needs to pass in arguments, and finally needs to execute the actual kernel code. This thorn OpenCLRunTime provides a simple helper routine for these tasks.

1 Overview

Thorn OpenCLRunTime performs the following tasks:

At startup, it outputs a description of the hardware (platforms and devices) available via OpenCL, i.e.xCPUs and GPUs that have OpenCL drivers installed.

At startup, it selects one device of one platform that will be used later on.

It provides an API that can be used to compile kernels for this device, and which remembers previously compiled kernels so that they don’t have to be recompiled.

It disassembles the compiled kernels so that they can be examined (which would otherwise be difficult in an environment using dynamic compilation).

It allocates storage for grid functions on the device, and handles copying data to and from the device (in interaction with thorn Accelerator).

It offers a set of convenience macros similar to CCTKxARGUMENTS and CCTKxPARAMETERS to access grid structure information. (Parameter values are expanded when compiling kernels, often enabling additional optimisations.)

It offers looping macros similar to those provided by thorn LoopControl, which parallelise loops via OpenCL’s multithreading.

It offers datatypes and macros for easy vectorisation, based on OpenCL’s vector types.

At the moment, OpenCLRunTime only supports unigrid simulations; adaptive mesh refinement or multi-block methods are not yet supported. (The main reason for this is that the cctkGH entries on the device are not updated; however, this should be straightforward to implement.)

2 Example

An OpenCL compute kernel may be called as follows:

xxcharxconstx*constxgroups[]x=x{
xxxx"WaveToyOpenCL::Scalar",
xxxxNULL};
xxintxconstximin[]x=x{cctk_nghostzones[0],
xxxxxxxxxxxxxxxxxxxxxxcctk_nghostzones[1],
xxxxxxxxxxxxxxxxxxxxxxcctk_nghostzones[2]};
xxintxconstximax[]x=x{cctk_lsh[0]x-xcctk_nghostzones[0],
xxxxxxxxxxxxxxxxxxxxxxcctk_lsh[1]x-xcctk_nghostzones[1],
xxxxxxxxxxxxxxxxxxxxxxcctk_lsh[2]x-xcctk_nghostzones[2]};
xxstaticxstructxOpenCLKernelx*kernelx=xNULL;
xxcharxconstx*constxsources[]x=x{"",xOpenCL_source_WaveToyOpenCL_evol,xNULL};
xxOpenCLRunTime_CallKernel(cctkGH,xCCTK_THORNSTRING,x"evol",
xxxxxxxxxxxxxxxxxxxxxxxxxxxsources,xgroups,xNULL,xNULL,xNULL,x-1,
xxxxxxxxxxxxxxxxxxxxxxxxxxximin,ximax,x&kernel);

The array groups specifies which grid functions are to be available in the OpenCL kernel. This is a C array terminated by NULL. (This information could instead also be gathered from the respective schedule.ccl declarations.)

The integer arrays imin and imax specify the iteration bounds of the kernel. This information is necessary so that OpenCL can properly map this iteration space onto the available OpenCL groups (threads).

The array sources (a C array terminated by NULL) specifies the actual source code for the kernel. The first string (here empty) can contain declarations and definitions that should be available outside the kernel function. The second string specifies the actual kernel code, excluding the actual function declaration which is inserted automatically. This is an example for such a kernel code:

//xGridxpointsxarexindexxinxthexsamexwayxasxforxaxCPU
//xUsingxptrdiff_txinsteadxofxintxisxmorexefficientxonx64-bit
//xarchitectures
ptrdiff_txconstxdix=x1;
ptrdiff_txconstxdjx=
xxCCTK_GFINDEX3D(cctkGH,0,1,0)x-xCCTK_GFINDEX3D(cctkGH,0,0,0);
ptrdiff_txconstxdkx=
xxCCTK_GFINDEX3D(cctkGH,0,0,1)x-xCCTK_GFINDEX3D(cctkGH,0,0,0);
//xCoordinatesxarexcalculatedxinxthexsamexasxasxforxaxCPU
CCTK_REALxconstxidx2x=x1.0x/xpown(CCTK_DELTA_SPACE(0),x2);
CCTK_REALxconstxidy2x=x1.0x/xpown(CCTK_DELTA_SPACE(1),x2);
CCTK_REALxconstxidz2x=x1.0x/xpown(CCTK_DELTA_SPACE(2),x2);
CCTK_REALxconstxdt2xx=xpown(CCTK_DELTA_TIME,x2);
//xNote:xThexkernelxbelowxisxnotxvectorisedx(sincexitxdoesn'txuse
//xCCTK_REAL_VEC).xTherefore,xvectorisationxmustxbexswitchedxoffxin
//xthexparamterxfilex(viaxOpenCLRunTime::vector_size_xx=x1).
//xThisxloopxmacroxautomaticallyxparallelizesxthexcode
//ximin[]xandximax[]xarexpassedxfromxthexhost
LC_LOOP3(evol,
xxxxxxxxxi,j,k,ximin[0],imin[1],imin[2],ximax[0],imax[1],imax[2],
xxxxxxxxxcctk_lsh[0],cctk_lsh[1],cctk_lsh[2])
{
xx//xCalculatexindexxofxcurrentxpoint
xxptrdiff_txconstxijkx=xdi*ix+xdj*jx+xdk*k;
xxCCTK_REALxconstxdxxux=xidx2x*x(u_p[ijk-di]x-x2.0x*xu_p[ijk]x+xu_p[ijk+di]);
xxCCTK_REALxconstxdyyux=xidy2x*x(u_p[ijk-dj]x-x2.0x*xu_p[ijk]x+xu_p[ijk+dj]);
xxCCTK_REALxconstxdzzux=xidz2x*x(u_p[ijk-dk]x-x2.0x*xu_p[ijk]x+xu_p[ijk+dk]);
xxCCTK_REALxconstxuvalx=
xxxx+2.0x*xu_p[ijk]x-xu_p_p[ijk]x+xdt2x*x(dxxux+xdyyux+xdzzu);
xxu[ijk]x=xuval;
}xLC_ENDLOOP3(evol);

The last argument kernel is used to store the compiled kernel and associated information, so that kernels do not have to be recompiled for every call.

In this case, the actual kernel source code is contained in a source file evol.cl in thorn WaveToyOpenCL. The C string OpenCLxsourcexWaveToyOpenCLxevol is generated automatically by Cactus as described in the users’ guide and/or thorn OpenCL.

3 Details

3.1 Hardware information

At startup, this thorn outputs a description of the hardware (platforms and devices) available via OpenCL, i.e.xCPUs and GPUs that have OpenCL drivers installed. Platforms correspond to vendors (AMD, Apple, Intel, Nvidia), devices to actual hardware (a CPU, a GPU, etc.). This information is written into a file opencl.txt in the output directory.

3.2 Device selection

At startup, this thorn selects one device of one platform that will be used later on. It chooses the first device of the first platform that matches the parameter openclxdevicextype, which can be CPU, GPU, accelerator, or

any.

3.3 Compiling kernels

This thorn provides an API that can be used to compile kernels for this device, and which remembers previously compiled kernels so that they don’t have to be recompiled. The compiler options for OpenCL are specified by the parameter openclxoptions and enable aggressive optimisations by default, as one would want for floating-point intensive code that is not too susceptive to round-off errors.

Cactus parameter values are expanded at compile time, enabling further optimisations. (However, when a parameter value changes, the kernel is not automatically recompiled – steerable parameters are not yet supported. This would be straightforward to implement.)

Typically, OpenCL compilers can optimise more than e.g.xC or Fortran compilers. The reason is that an OpenCL compiler knows the complete code – it is not possible to call routines that are defined elsewhere, or to be influenced by changes originating elsewhere (e.g.xin another thread). Unfortunatelly, this does not mean that all OpenCL compilers are good at optimising – OpenCL is a fairly young language, and some of the technology is still immature.

A file containing the exact source code passed to the compiler is placed into the output directory with a name KERNELNAME.cl. A log file containing all compiler output including error messages is placed into the output directory with a name KERNELNAME.log. Both are indispensable for debugging OpenCL code.

3.4 Disassembling kernels

This thorn disassembles the compiled kernels so that they can be examined (which would otherwise be difficult in an environment using dynamic compilation). The disassembled output is placed into the output directory with a name KERNELNAME.s, if disassembling is supported and makes sense. (For example, object files for Nvidia GPUs contain PTX, which is essentially a high-level assembler code, and are thus not disassembled.)

By default, kernels are disassembled in the background.

3.5 Memory management

This thorn allocates storage for grid functions on the device, and handles copying data to and from the device (in interaction with thorn Accelerator).

OpenCL devices have memory that is independent of the host memory. This is the case even when using CPUs – a particular memory region cannot be accessed by host code and by OpenCL kernels at the same time. This thorn offers several memory models (memory allocation strategies):

always-mapped:

Host and device access the memory simultaneously. This may work, but violates the OpenCL standard. Do not use this, unless you know that your implementation supports this. If this does not work, some values in memory will randomly change.

copy:

Host and device memory are allocated independently. Data will be copied. This makes sense e.g.xfor GPUs that have their own memory. This model also allows memory layout optimisation such as aligning grid functions with vector sizes or cache lines. Such layout optimisations are currently not supported by the Cactus flesh (but work is in progress to implement this there).

map:

Device memory is allocated such that it (likely) coincides with the memory already allocated on the host. However, either only the host or only the device can access this memory at a time; the OpenCL run-time needs to be notified to switch between these. This memory model will save space, but may be slower if host memory cannot efficiently be accessed from the device. This memory model is also not yet fully tested.

Routines may execute either on the host (regular routines) or on a device (OpenCL routines). Variables accessed (read or written) by routines may need to be copied between host and device. Thorn Accelerator keeps track of this, and notifies thorn OpenCLRunTime when data need to be copied.

Data also need to be available on the host for inter-processor synchronisation and I/O.

The parameter syncxcopyxwholexbuffer determines whether the whole grid function or only values on/near the boundary are copied for synchronisation.

3.6 Grid structure

This thorn offers a set of convenience macros similar to CCTKxARGUMENTS and CCTKxPARAMETERS to access grid structure information. Currently, only a subsect of the information in cctkGH is available:

xxptrdiff_txcctk_lbnd[]
xxptrdiff_txcctk_lsh[]
xxptrdiff_tximin[]
xxptrdiff_tximax[]
xxCCTK_REALxcctk_time
xxCCTK_REALxcctk_delta_time
xxCCTK_REALxcctk_origin_space[]
xxCCTK_REALxcctk_delta_space[]
xxCCTK_DELTA_TIME
xxCCTK_ORIGIN_SPACE()
xxCCTK_DELTA_SPACE()
xxCCTK_GFINDEX3D()

cctkxlbnd and cctkxlsh have the same meaning as on the host. imin and imax contain the values specified when calling OpenCLRunTimexCallKernel, and determine the loop bounds used in this kernel. The real-valued variables and their macro counterparts have the same meaning as on the host. The type of the integer fields has been changed from int to ptrdiffxt, which is a 64-bit type on 64-bit platforms, and leads to more efficient code since it avoids type conversions.

3.7 Loops

This thorn offers looping macros similar to those provided by thorn LoopControl, which parallelise loops via OpenCL’s multithreading.

The loop macros LCxLOOP3 and LCxENDLOOP3 should be called as in the example above. The first argument defines a name for the loop, the next three arguments define the names of the iteration indices. The remaining arguments describe the loop bounds and the grid function size.

These macros need to be used. Each OpenCL thread will loop only over a part of the region described by imin and imax. If this macro is not used, OpenCL’s multithreading may be used in an inconsistent manner (unless you use OpenCL’s API to distribute the workload yourself).

3.8 Vectorisation

OpenCL supports vector data types. Using such vector data types is important to achieve good performance on CPUs. This thorn provides macros, in particular CCTKxREALxVEC, that can be used for this. Unfortunately, vectorisation has to be performed explicitly by the kernel writer, and is not performed by this thorn. (However, note that some OpenCL compilers can vectorise code automatically.)

When vectorising code explicitly, one needs to use special instructions to load and store values from and to memory. This is not (yet) described here; however, the macros are similar to those offered by thorn Vectors. At the moment, these vectorisation capabilities are targeted for automated code generation (e.g.xby Kranc) rather than for manual programming.

4 Parameters

11 disassemblexinxbackground10 Scope: private01  BOOLEAN

11 Description: Disassemble in the background (using fork)

11                         10              01  Default: yes

11

11 disassemblexkernels  10 Scope: private01  BOOLEAN

11 Description: Disassemble kernels

11                         10              01  Default: yes

11

11 groupxsizexx  10 Scope: private01  INT

11 Description: Group size in x direction

11  Range  10              01  Default: 1

11 1:*  10

11

11 groupxsizexy  10 Scope: private01  INT

11 Description: Group size in y direction

11  Range  10              01  Default: 1

11 1:*  10

11

11 groupxsizexz  10 Scope: private01  INT

11 Description: Group size in z direction

11  Range  10              01  Default: 1

11 1:*  10

11

11 memoryxmodel  10 Scope: private01  KEYWORD

11 Description: Memory model

11 Range 10 01 Default: copy
11 always-mapped  10 CPU and GPU use the same memory (may violate standard)
11 copy  10 Copy buffers
11 map  10 Map buffers (requires same layout)

11 openclxdevicextype  10 Scope: private01  KEYWORD

11 Description: Device type

11  Range  10              01  Default: CPU

11 CPU  10
11 GPU  10

11 accelerator  10

11 any  10

11

11 openclxoptions  10 Scope: private01  STRING

11 Description: OpenCL compiler options

11  Range  10              01Default: -cl-mad-enable -cl-no-signed-zeros -cl-fast-relaxed-math

11                         10

11

11 syncxcopyxwholexbuffer 10 Scope: private01  BOOLEAN

11 Description: Copy whole buffer before/after syncing

11                         10              01  Default: no

11

11 tilexsizexx  10 Scope: private01  INT

11 Description: Tile size in x direction

11  Range  10              01  Default: 1

11 1:*  10

11

11 tilexsizexy  10 Scope: private01  INT

11 Description: Tile size in y direction

11  Range  10              01  Default: 1

11 1:*  10

11

11 tilexsizexz  10 Scope: private01  INT

11 Description: Tile size in z direction

11  Range  10              01  Default: 1

11 1:*  10

11

11 unrollxsizexx  10 Scope: private01  INT

11 Description: Unroll size in x direction

11  Range  10              01  Default: 1

11 1:*  10

11

11 unrollxsizexy  10 Scope: private01  INT

11 Description: Unroll size in y direction

11  Range  10              01  Default: 1

11 1:*  10

11

11 unrollxsizexz  10 Scope: private01  INT

11 Description: Unroll size in z direction

11  Range  10              01  Default: 1

11 1:*  10

11 vectorxsizexx  10 Scope: private01  INT

11 Description: Vector size in x direction

11 Range 10 01 Default: (none)
11                         10 use preferred vector size

11 1:*  10

11

11 vectorxsizexy  10 Scope: private01  INT

11 Description: Vector size in y direction

11  Range  10              01  Default: 1

11 1  10

11

11 vectorxsizexz  10 Scope: private01  INT

11 Description: Vector size in z direction

11  Range  10              01  Default: 1

11 1  10

11

11 verbose  10 Scope: private01  BOOLEAN

11 Description: Output detailed device information

11                         10              01  Default: no

11

11 veryverbose  10 Scope: private01  BOOLEAN

11 Description: Output even more detailed information

11                         10              01  Default: no

11

11 outxdir  10 Scope: shared from IO01STRING

11

5 Interfaces

General Implements: openclruntime Adds header: OpenCLRunTime.h

Uses header: carpet.hh vectors.h

Provides: DevicexCreateVariables to DevicexCopyCycle to DevicexCopyFromPast to DevicexCopyToDevice to DevicexCopyToHost to DevicexCopyPreSync to DevicexCopyPostSync to LinearCombination to

6 Schedule This section lists all the variables which are assigned storage by thorn CactusUtils/OpenCLRunTime. Storage can either last for the duration of the run (Always means that if this thorn is activated storage will be assigned, Conditional means that if this thorn is activated storage will be assigned for the duration of the run if some condition is met), or can be turned on for the duration of a schedule function.

Storage NONE

Scheduled Functions

CCTKxSTARTUP x openclruntimexsetup x set up opencl device x

11 x 11 Language: 11 c
11 x 11 Type:  11 function

11

CCTKxWRAGH x openclruntimexdeviceinfo x output opencl device information

11 x 11 Language: 11 c
11 x 11 Type:  11 function

11

CCTKxWRAGH x openclruntimexautoconf x determine whether certain features are supported x

11 x 11 After:  11 openclruntimexdeviceinfo
11 x 11 Language: 11 c
11 x 11 Type:  11 function

11

CCTKxBASEGRID x openclruntimexsetupdevicegh x set up device grid structure x

11 x 11 After:  11 spatialspacings
11 x 11 x  11 temporalspacings
11 x 11 Before: 11 spatialcoordinates
11 x 11 Language: 11 c
11 x 11 Type:  11 function

11

CCTKxTERMINATE x openclruntimexstatistics x output profiling information x

11 x 11 Language: 11 c
11 x 11 Options: 11 global
11 x 11 Type:  11 function