Runtime analysis

ExaParty comes along with a set of Python scripts to analyse the code's runtime behaviour.

These tools can be found in the namespace exaparty.api.postprocessing. The core ingredient in there is a parser which analyses the standard outputs of ExaParty. Once you pipe the standard outputs into a file, you can apply these parser, i.e. use them within your own postprocessing scripts. There are however also a couple of command line tools which help you to explore the data. These tools use the same parsers, i.e. are a good starting point to familiarise yourself with the parser utilities.

Analyse tool feedback

Start by gathering vectoriser output. If you use LLVM/Clang, recompile the appropriate object file with the flags

-Rpass=loop-vectorize -Rpass-missed=loop-vectorize -Rpass-analysis=loop-vectorize`

Find the desired loop in the output. For templates, the reported file/line is of the template, not the template instantiation location. When working with nested loops, usually only the innermost loop gets vectorised, so focus on this one. Sample expected output:

../../../../src/exaparty/kernels/ParticleParticleInteraction.cpph:481:11: remark: loop not vectorized [-Rpass-missed=loop-vectorize]
         for (int activeParticleNumberInThisChunk = 0; activeParticleNumberInThisChunk < activeParticlesChunkSize;

Sometimes the vectoriser only mentions that the loop isn't vectorised without mentioning why. This is usually the case where the are multiple issues. A safe point in this case is it to start to try to inline any major functions called inside the loop. If your kernel consists of a loop that calls a single interaction function, manually inline this function.

The Intel toolchain is a little bit more convenient: Include '''-O3 -xHost -qopt-report=3''' in the compile call. This make the compiler yield a .optrpt text file, which contains the actual optimisation feedback. Often, you don't want to configure with the optimisation report and get one optimisation report per file. If you know where your routines are used, generate your code plus the Makefile through Python and, afterwards, manipulate the Makefile:

solver:  $(CXX_OBJS) $(FORTRAN_OBJS)
    rm observers/StephydroPartForceLoop_sweep62peano4_toolbox_particles_UpdateParticle_MultiLevelInteraction_StackOfLists_ContiguousParticles4.o
    $(CXX) -qopt-report=3 $(CXXFLAGS) -c -o observers/StephydroPartForceLoop_sweep62peano4_toolbox_particles_UpdateParticle_MultiLevelInteraction_StackOfLists_ContiguousParticles4.o observers/StephydroPartForceLoop_sweep62peano4_toolbox_particles_UpdateParticle_MultiLevelInteraction_StackOfLists_ContiguousParticles4.cpp
    $(CXX) $(FORTRAN_MODULE_OBJS) $(FORTRAN_OBJS) $(CXX_OBJS) $(LDFLAGS) $(GPU_OBJS) $(CU_OBJS) $(LIBS) -o noh2D

where the observer file names have to be replaced with the name of the file which uses your altered kernel invocation. This tweaked makefile builds all object files. After that, it deletes the object file behind the action set StephydroPartForceLoop. This is the step where the force kernel is actually used. We then retranslate this single object file with Intel's -opt-report flag which provides ample evidence which vectorisation optimisation has been performed by the compiler.

You should get a file observers/StephydroPartForceLoop_sweep62peano4_toolbox_particles_UpdateParticle_MultiLevelInteraction_StackOfLists_ContiguousParticles4.optrpt which summarises if the compiler has vectorised your code and even how much speedup you might expect. Please note that every rerun of the Python script overwrites the Makefile. However, there's no need to do this while you tweak the kernels. Just re-invoke make.

Peano data

As ExaParty is built on top of Peano 4, all the generic Peano scripts work for ExaParty, too.

See also

Domain decomposition

python/exahype2/postprocessing/plot-scaling.py

python/exahype2/postprocessing/TimeseriesPlots.py