A suite of benchmark runs which assess the throughput of the Euler Finite Volume kernels.

ExaHyPE 2's implementation of the Euler equations is shipped with a benchmark to assess the speed of the compute kernels. The benchmark solely computes the throughput and thus provides key information how efficient these core kernels perform on your system. It neglects other effects such as multithreading orchestration overhead or MPI data exchange, grid management, time stepping behaviour, halo data exchange, and so forth. While this might be of limited relevance for large-scale runs, where scalability, IO overheads and other factors play a dominant role, it tells us what the core compute routines could, in theory, deliver.

Prepare the environment

Prior to a build, ExaHyPE has to be configured and built correctly. For this, please follow the instructions on ExaHyPE 2 (scroll down to the rubric Building the ExaHyPE 2 extension). Notably ensure that the toolboxes are in place. You don't need the multithreading or GPU extensions. If you enable them, the kernel benchmarks benchmark these features. Otherwise, they benchmark the single core setup.

Once built, change into the benchmark's directory. The directory hosts a set of benchmarks which assess the sole performance of the individual kernels on the CPU and GPUs. These benchmarks are shipped through Python scripts which should run out-of-the-box. Besides the benchmarking scripts themselves, we also ship a plotting script which help you to produce fancy graphs.

Build benchmarks

There are different benchmarks for the different solver types (e.g., Finite Volumes (FV) and Discontinuous Galerkin (DG)) and you have to build different executables for different patch sizes, orders or dimensions.

Finite Volumes Rusanov

To build the executables you can run following commands:

export PYTHONPATH=../../../../python
 
python3 kernel-benchmarks-fv-rusanov-dsl.py --dim 2 --patch-size 4 --num-patches 512

The extension dsl here is misleading in the sense that all of our kernels currently transition into a DSL-based formalism. In that sense, the extension is obsolete. However, the DSL unfolds its full potential if and only you use xDSL. Without it, the kernel benchmarks fall back to some default kernel generations which are plain C++ with SYCL or OpenMP, respectively.

If you wish to also benchmark the MLIR versions of the solver, add –enable-mlir to the compile command. Currently this is only supported for host kernels, so also add –cpu to the command.

Cleanup

You can throw away all of the glue code after a successful build. This is triggered by:

make clean

make distclean

If you want to keep the executable, you will need to rename it.

Run benchmarks

All the benchmarks run out of the box without any user interaction once built. For most runs, you don't have to specify anything. Depending on the build configuration, your file's name is similar to

./KernelBenchmarksFVRusanovDSL.Release

which obviously checks the Rusanov Finite Volume scheme with a release build. Nothing stops you from setting up a build with assertions or even in debug mode (type in --help for the Python script to see options).

OpenMP, SYCL and other back-ends might want you to configure your environment in specific ways. You might have to set some environment variables specifying the GPUs to use or core counts accordingly. The exact usage obviously depends strongly on your software stack.

If you encounter issues with a particular backend, it makes sense to study the page on FAQs and Troubleshooting.

Outputs

All of them produce outputs in a similar way, so it is easier to interpret the outcomes. All outcome is piped onto the terminal. It looks similar to

Dimensions: 3
Number of threads launching compute kernels: 1
Number of patches per thread/compute kernel launch to study: [1024]
Number of compute threads: 32
Number of MPI ranks: 1
Number of GPU devices: 4
Number of finite volumes per axis per patch: 8
Number of samples per measurement: 10
Evaluate max. eigenvalue (reduction step): 1
Number of patches per rank: 1024
------------------------------------------------------------------------------------------------------------------------------------------------------
Kernel ID:
    Compute Kernel Time |
    Compute Kernel Time (Normalised) |
    Compute Kernel String |
    Kernel Launch Time |
    Kernel Launch Time (Normalised) |
    Kernel Launch String
 00:00:15     ------------------------------------------------------------------------------------------------------------------------------------------------------
host, functors, batched, AoS, sequential:
    0.864157 |
    1.64825e-06 |
    (avg=0.864157,#measurements=10,max=0.867656(value #0),min=0.862205(value #2),+0.404973%,-0.225896%,std-deviation=0.0014239) |
    0.868278 |
    1.65611e-06 |
    (avg=0.868278,#measurements=10,max=0.871799(value #0),min=0.866315(value #2),+0.405557%,-0.226038%,std-deviation=0.00142904)
 00:00:24     ------------------------------------------------------------------------------------------------------------------------------------------------------
host, functors, batched, AoS, parallel-for:
    0.863212 |
    1.64645e-06 |
    (avg=0.863212,#measurements=10,max=0.864114(value #5),min=0.862669(value #3),+0.104399%,-0.0629067%,std-deviation=0.000462938) |
    0.867338 |
    1.65432e-06 |
    (avg=0.867338,#measurements=10,max=0.868259(value #5),min=0.866779(value #3),+0.106173%,-0.0643882%,std-deviation=0.000468994)
 00:00:33     ------------------------------------------------------------------------------------------------------------------------------------------------------
host, functors, batched, AoS, subtasks:
    0.863837 |
    1.64764e-06 |
    (avg=0.863837,#measurements=10,max=0.86558(value #5),min=0.862646(value #8),+0.201852%,-0.13781%,std-deviation=0.000818006) |
    0.867955 |
    1.65549e-06 |
    (avg=0.867955,#measurements=10,max=0.869699(value #5),min=0.866756(value #8),+0.200937%,-0.138144%,std-deviation=0.000815395)
 00:00:37     ------------------------------------------------------------------------------------------------------------------------------------------------------
host, functors, patch-wise, AoS, sequential:
    0.273316 |
    5.21308e-07 |
    (avg=0.273316,#measurements=10,max=0.273409(value #0),min=0.273277(value #5),+0.0340361%,-0.0140717%,std-deviation=3.9666e-05) |
    0.277728 |
    5.29725e-07 |
    (avg=0.277728,#measurements=10,max=0.277896(value #0),min=0.277672(value #9),+0.0603929%,-0.0201329%,std-deviation=6.12112e-05)
 00:00:40     ------------------------------------------------------------------------------------------------------------------------------------------------------
host, functors, patch-wise, AoS, parallel-for:
    0.273104 |
    5.20904e-07 |
    (avg=0.273104,#measurements=10,max=0.273143(value #1),min=0.273023(value #3),+0.0144253%,-0.0293911%,std-deviation=3.27078e-05) |
    0.277518 |
    5.29323e-07 |
    (avg=0.277518,#measurements=10,max=0.277556(value #1),min=0.277446(value #3),+0.0136391%,-0.0259851%,std-deviation=3.3277e-05)
 00:00:44     ------------------------------------------------------------------------------------------------------------------------------------------------------
host, functors, patch-wise, AoS, subtasks:
    0.273305 |
    5.21287e-07 |
    (avg=0.273305,#measurements=10,max=0.273578(value #1),min=0.273225(value #7),+0.100017%,-0.0291667%,std-deviation=9.38979e-05) |
    0.277729 |
    5.29726e-07 |
    (avg=0.277729,#measurements=10,max=0.277993(value #1),min=0.277629(value #8),+0.0949597%,-0.0360712%,std-deviation=9.59726e-05)
 00:00:45     ------------------------------------------------------------------------------------------------------------------------------------------------------
host, functors, volume-wise, AoS, sequential:
    0.0671029 |
    1.27989e-07 |
    (avg=0.0671029,#measurements=10,max=0.0681471(value #0),min=0.0667223(value #6),+1.5561%,-0.567194%,std-deviation=0.000379433) |
    0.0671064 |
    1.27995e-07 |
    (avg=0.0671064,#measurements=10,max=0.0681511(value #0),min=0.0667256(value #6),+1.55681%,-0.567535%,std-deviation=0.000379594)
 00:00:46     ------------------------------------------------------------------------------------------------------------------------------------------------------
host, functors, volume-wise, AoS, parallel-for:
    0.0669354 |
    1.27669e-07 |
    (avg=0.0669354,#measurements=10,max=0.0671566(value #4),min=0.0667106(value #9),+0.330462%,-0.335768%,std-deviation=0.000122576) |
    0.0669387 |
    1.27675e-07 |
    (avg=0.0669387,#measurements=10,max=0.0671597(value #4),min=0.0667136(value #9),+0.330165%,-0.336318%,std-deviation=0.000122623)
 00:00:48     ------------------------------------------------------------------------------------------------------------------------------------------------------
host, functors, volume-wise, AoS, subtasks:
    0.0669391 |
    1.27676e-07 |
    (avg=0.0669391,#measurements=10,max=0.0671185(value #7),min=0.0666028(value #1),+0.268017%,-0.50245%,std-deviation=0.000137354) |
    0.0669425 |
    1.27683e-07 |
    (avg=0.0669425,#measurements=10,max=0.0671218(value #7),min=0.0666066(value #1),+0.26783%,-0.501842%,std-deviation=0.000137297)

We get sets of experiments for different combinations of number of threads that launch compute tasks and number of patches handled by launch. In the simple one-thread case, we make thread 0 of a multithreading environment submit different kernels realisations for a certain number of patches. Some of these kernels are parallelised internally, others are not.

If configured, two threads issue the same type of kernel in parallel. This way, we can stress test the system.

Per realisation, we get six measurements:

The first measurement gives the time spent solely within the compute kernels.
The second measurement gives the compute kernel time per degree of freedom.
The fourth measurement is the total time of a single kernel execution including overhead such as temporal memory allocation and freeing.
The fith measurement is the update time per degree of freedom within a single kernel including overhead.
The third and sixth measurement provide the raw data, so we can see the standard deviation of the measurements. We also see that we usually sample each value multiple times to eliminate noise.

If a single kernel invocation (for multiple patches) from one thread is able to exploit all cores already, then multiple launches of a kernel should either deliver the same throughput or a worse one, as there's some overhead. If a single kernel launch (for multiple patches) is not able to exploit all threads, launching multiple kernels in parallel should improve the throughput.

Plotting

For any postprocessing, we recommend that the test's output is piped into a file:

./KernelBenchmarksFVRusanovDSL.Release > output.txt

The directory hosts a script called plot.py which solely takes the output piped into a file as input.

These scatter plots report the throughput for particular combinations of threads which launch kernels simultaenously and patches per kernel launch. We distinguish the throughput for one single kernel (considering the second measurement) (x-axis) and the total throughput (considering the fith measurement) (y-axis).

Plots like the one above highlight that some kernel realisations deliver outcomes for their patches particularly quickly. They exhibit a small algorithmic latency. Other realisations are not that fast for their patches, but are fairer and hence allow Peano to pipe more kernels simultatenously through the system.