Domain decomposition (parallelisation with MPI or multithreading)

Table of Contents

• Adding your own parallel features to an ExaHyPE solver (programmer view)

Peano's domain decomposition is used for both MPI and multithreading.

It is used as is by ExaHyPE. The one thing ExaHyPE adds on top is some bespoke/abstracted load-balancing, while its solvers themselves obviously are made domain decomposition-aware.

This page is all about how the domain decomposition is realised within ExaHyPE, i.e. what specific aspects an ExaHyPE solver has to take into account. There is other domain decomposition material available on the pages:

• The tutorials Domain decomposition with an off-the-shelf heuristic and Offline load balancing discuss how to use the domain decomposition.
• The page Load Balancing (Domain Decomposition) discusses various load balancing variants as offered through the load balancing toolbox.
• The page Domain decomposition in Peano's documentation demonstrates how the domain decomposition is technically realised on the mesh level.
• The page Runtime analysis touches upon domain decomposition from a performance point of view.

Adding your own parallel features to an ExaHyPE solver (programmer view)

Most users that write extensions to ExaHyPE which need some global data exchange are users which add new fields to their solver. Examples are global statistics that they want to track over the solution or global variables that they control.

Each MPI rank holds one instance of the solver. All threads per MPI rank share the same instance of the solver. Peano splits up the domain into chunks along the space-filling curve. If you use an enclave solver, it then splits up these chunks further into tasks. That is, operations per cell such as the Riemann solves and the patch postprocessing can run in parallel on each thread.

If you add new attributes to your solver, this attribute exists on each solver on each rank. You therefore have to

• ensure that no two threads access the attribute at the same time;
• ensure that the attributes of the individual solver instances per rank remain consistent.

The first item is relatively simple: You might, for example, decide to protect all accesses to your own variables via a semaphore. There is no specific place to take care of the protection: Just protect the actual data access.

The second item requires slightly more work, as you have to globally reduce the attribute: After each time step, you have to ensure that all ranks exchange their attributes with each other, and you then have to merge these data. I recommend not to use MPI's data exchange routine directly, but to use Peano's wrappers around these routines. See also Peano's domain decomposition (data exchange events).

The actual data exchange should be done at the end of each time step. For this, you add

void finishTimeStep() override;

to your solver. The implementation should first invoke the superclass and then add the additional data exchange.

void mynamespace::MyFancySolver::finishTimeStep() {
  MyFancyAbstractSolver::finishTimeStep();
  #ifdef Parallel
  // additional data exchange
  #endif
}

ExaHyPE internally realises these two steps for all of its solver attributes such as the maximum mesh size or the admissible time step sizes. This is the reason why you have to call the superclass implementation, too.

See also

• Domain decomposition tutorial
• Offline load balancing tutorial
• Load balancing API
• Load Balancing (Domain Decomposition)
• Runtime analysis
• Offline extraction of hardcoded parameters (see Load balancing API) reconstructs split operations per mesh level, i.e. it emits an instruction for each level where a subtree is introduced across a parent–child edge (plus potential level-0 presence). The reported numberOfCells therefore counts “introduced” cells per level rather than only the earliest appearance.