This is a short benchmark which assesses the kernel throughputs of the ExaParty project.
Design
The benchmark design is very simple. We first create a simple SPH test through the full-blown ExaParty software stack. Before we compile, we however replace the main file with our bespoke benchmark main, which is really just a sequence of loops.
The benchmark is guided by few key parameters
- Number of particles. The benchmark driver iterates over those particles internally. You can alter the maximum particles count however (invoke the Python script with –help for details).
- The whole test suite is repeated
sample times. The default is 16, but you can alter it through the Python script.
- The tutal number of particles is split up into work items, i.e. chunks of continuous particles. For a real setup, we can think of these work items as the number of particles associated typically with a vertex while we kick them, or we can read them as the average number of particles per cell which have to be compared to each other. Lacking a better name, we use the generic term work item here and make its size configurable.
- The benchmark runs over a sequence of work items and invokes the kernels over all particles per work item. If you have a linear kernel (e.g. the drift or kick), the script runs through all particles per work item. If you have a quadratic kernel (force and density) the benchmark runs over all particles of a work item checked against all other particles of this work item. That is, the kernel benchmark works with quadratic complexity over the work item size, but linear complexity overall. We do not write these loops here ourselves, but we directly invoke the ExaParty loop routines, i.e. everything we measure is 1:1 from the ExaParty code base.
What the benchmark does
The benchmark takes the total number of particles and splits it up into chunks of work items. Per hydro kernel operator, we loop over the work items in parallel and issue the compute kernel per chunk. We then report on the outcome similar to
===============================
Experiment: 256 particles
No of work items: 4
===============================
density kernel: 0.000752679 (avg=0.000752679,#measurements=16,max=0.0119854(value #15),min=3.257e-06(value #12),+1492.37%,-99.5673%,std-deviation=0.00290028)
force kernel: 4.20344e-06 (avg=4.20344e-06,#measurements=16,max=8.392e-06(value #0),min=3.695e-06(value #11),+99.6461%,-12.0958%,std-deviation=1.10753e-06)
kick1 kernel: 1.41364e-05 (avg=1.41364e-05,#measurements=16,max=5.0735e-05(value #3),min=1.0868e-05(value #14),+258.897%,-23.1203%,std-deviation=9.60516e-06)
kick2 kernel: 1.25493e-05 (avg=1.25493e-05,#measurements=16,max=1.6944e-05(value #10),min=1.1706e-05(value #4),+35.0193%,-6.71999%,std-deviation=1.16347e-06)
drift kernel: 2.10832e-05 (avg=2.10832e-05,#measurements=16,max=0.000168095(value #2),min=1.1007e-05(value #12),+697.294%,-47.7925%,std-deviation=3.79588e-05)
===============================
Experiment: 512 particles
No of work items: 8
===============================
density kernel: 0.00024182 (avg=0.00024182,#measurements=16,max=0.0034722(value #0),min=3.836e-06(value #1),+1335.86%,-98.4137%,std-deviation=0.000835668)
force kernel: 0.000694633 (avg=0.000694633,#measurements=16,max=0.0088532(value #15),min=4.288e-06(value #10),+1174.51%,-99.3827%,std-deviation=0.00217165)
kick1 kernel: 4.28743e-05 (avg=4.28743e-05,#measurements=16,max=0.000353835(value #2),min=1.9147e-05(value #6),+725.286%,-55.3415%,std-deviation=8.07341e-05)
kick2 kernel: 2.21336e-05 (avg=2.21336e-05,#measurements=16,max=4.2598e-05(value #2),min=1.9444e-05(value #10),+92.4589%,-12.1515%,std-deviation=5.47072e-06)
drift kernel: 4.49467e-05 (avg=4.49467e-05,#measurements=16,max=0.000418756(value #1),min=1.956e-05(value #10),+831.673%,-56.4818%,std-deviation=9.65175e-05)
This way, we can assess different kernel speeds against each other. Depending on the build (OpenMP on, GPU offloading on), we assess different variants of the kernel and compare them against each other.
Build a demo
The build can all be triggered through the command line using the Python interface.
export PYTHONPATH=../../../../python
python3 kernel-throughput.py --dimensions 2
This is the baseline version using some default arguments. You might want to tailor the settings. The build yields an executable which you should be able to run without further user interference.
Postprocessing
There's a postprocessing script in the directory, which expects the naming convention as of above. It operates on a per-directory basis:
python3 ./print-kernel-throughput.py file
Assessing individual kernels
It is possible to assess individual kernels. First, we can narrow down the measurement to individual particles counts:
python3 kernel-throughput.py --dimensions 2 --min-particle-number 65536 --max-particle-number 65536
Further to that, we can pick only certain kernels. If you type in –help, you see all kernels that we offer as part of the benchmark. By default, we test all of them. If you use the –enable-kernel argument, all kernels are, by default, switched off and you pick one or multiple kernels from the list.
python3 kernel-throughput.py --dimensions 2 --enable-kernel drift
1.9.8