This file is part of the ExaHyPE 2 project within Peano 4. More...

Namespaces
namespace	api

namespace	blockstructured

namespace	dg

namespace	elastic

namespace	enumerator

namespace	fd

namespace	fv

namespace	internal

Data Structures
class	CellAccess
	Cell access class. More...

struct	CellData
	Representation of a number of cells which contains all information that's required to process the stored patches or polynomial shape functions (given you know the semantics of the data). More...

class	CheckpointReader
	Read a checkpoint back. More...

struct	CopyCellDataGPU
	Cousing to CellData which administers copies on the accelerator. More...

class	LoadBalancingConfiguration
	ExaHyPE 2-specific load balancing configuration. More...

class	PlotFilter

class	RefinementControl
	Manage refine/erase requests within ExaHyPE 2. More...

class	RefinementControlService

class	Solver

struct	VolumeIndex
	A lot of the loop routines expect a voxel (or face) index which is a vector over integers. More...

struct	VolumeIndex< 0, Args... >

Enumerations
enum class	PrecisionCommand { Long_double = sizeof(long double) , Double = sizeof(double) , Single = sizeof(float) , Float = Single , Half = sizeof(float) / 2 , Float16_t = Half , FP16 = Half , Bfloat = -1 , Bfloat16_t = Bfloat , BF16 = Bfloat , Custom }

enum class	RefinementCommand { Keep , Refine , Erase }

Functions
PrecisionCommand	getDefaultPrecisionCommand ()
	The default is double as this is the highest precision available per default.

RefinementCommand	getDefaultRefinementCommand ()
	The default is coarsen as this is the lowest priority command.

bool	configureRuntime (int timeout, int numberOfThreads, const std::set< int > &gpus, tarch::multicore::orchestration::Strategy *orchestration)
	Configure the runtime of an ExaHyPE 2 simulation.

void	runSimulation (peano4::RunMultipleSteps &setupSimulation, peano4::RunMultipleSteps &plot, peano4::RunMultipleSteps &timeStep, peano4::RunMultipleSteps &meshAdaptation, peano4::RunMultipleSteps::Report &report, std::function< bool()> wantsToPlot, std::function< double()> getMinTimeStamp, std::function< double()> getMaxTimeStamp, std::function< double()> getMaxTimeStepSize, double minTerminalTime, double maxTerminalTime)
	Run an ExaHyPE 2 simulation.

bool	isMeshAdaptationRequired ()
	Does the mesh have to be adapted or the load to be rebalanced?

void	shutdownRuntime (std::function< void()> shutdownDatatypes)
	Shut down the runtime of an ExaHyPE 2 simulation.

template<typename CellLabel , typename FaceLabel >
double	getMinTimeStampOfNeighboursAhead (const CellLabel &cellLabel, const peano4::datamanagement::FaceEnumerator< FaceLabel > &faceLabelEnumerator)
	Similar to getMinTimeStampOfNeighbours(), but we minimise only over those neighbours that are actually ahead.

template<typename CellLabel , typename FaceLabel >
bool	runTimeStepOnCell (const CellLabel &cellLabel, const peano4::datamanagement::FaceEnumerator< FaceLabel > &faceLabelEnumerator, double globalMinTimeStamp)
	Determine whether to run a time step on a cell by analysing the neighbouring cells' timestamp.

template<typename CellLabel , typename FaceLabel >
double	removeTimeStepAccumulationErrorsFromCell (const CellLabel &cellLabel, const peano4::datamanagement::FaceEnumerator< FaceLabel > &faceLabelEnumerator, double timeStepSize)
	Remove accumulation errors from cell time stamps.

std::pair< double, double >	getInterpolationWeights (double oldTimeStampOnFace, double newTimeStampOnFace, double cellTimeStamp)

double	discretiseAndTruncateTimeStepSizes (double cellTimeStepSize, double maxGlobalTimeStepSize, int discretisationStepsSize)
	Discretise (bucket) time step sizes and truncate it.

tarch::tests::TestCase *	getUnitTests ()
	Please destroy after usage.

void	setDefaultLogStatements ()

template<class... Args>
auto	volumeIndex (Args... args)

Variables
str	ASCII_ART_HEADER

str	COPYRIGHT_HEADER = "ExaHyPE 2x extension"

bool	havenumpy = False

bool	havempmath = False

bool	havesympy = False

bool	havematplotlib = False

Detailed Description

This file is part of the ExaHyPE 2 project within Peano 4.

For the generic kernels that I use here most of the time.

For conditions of distribution and use, please see the copyright notice at www.peano-framework.org

Enumeration Type Documentation

◆ PrecisionCommand

enum class exahype2::PrecisionCommand

strong

Enumerator
Long_double
Double
Single
Float
Half
Float16_t
FP16
Bfloat
Bfloat16_t
BF16
Custom

Definition at line 6 of file PrecisionControl.h.

◆ RefinementCommand

enum class exahype2::RefinementCommand

strong

Enumerator
Keep
Refine
Erase

Definition at line 13 of file RefinementControl.h.

Function Documentation

◆ configureRuntime()

bool exahype2::configureRuntime	(	int	timeout,
		int	numberOfThreads,
		const std::set< int > &	gpus,
		tarch::multicore::orchestration::Strategy *	orchestration
	)

Configure the runtime of an ExaHyPE 2 simulation.

The main() generated by exahype2.api.generators.DefaultSimulationMain calls this routine once the parallel environment, the data types and Peano's singletons are initialised, and before it sets the load balancing. The routine

sets the deadlock time out of tarch::mpi::Rank (if timeout>=0),
configures the number of threads (tarch::multicore::Core) and the GPUs (tarch::accelerator::Device),
installs the multicore orchestration, and
checks that MPI runs use more than one thread, as MPI runs without multithreading are not supported.

Parameters

timeout	Deadlock time out in seconds. A negative value keeps the default.
numberOfThreads	Number of threads, or tarch::multicore::Core::UseDefaultNumberOfThreads.
gpus	Devices to use. Empty if no GPU is to be used.
orchestration	Multicore orchestration. The routine hands it over to tarch::multicore::setOrchestration(), which takes ownership.

Returns: false if the configuration is invalid, i.e. if this is an MPI run without multithreading. main() then has to terminate.

◆ discretiseAndTruncateTimeStepSizes()

double exahype2::discretiseAndTruncateTimeStepSizes	(	double	cellTimeStepSize,
		double	maxGlobalTimeStepSize,
		int	discretisationStepsSize
	)

Discretise (bucket) time step sizes and truncate it.

This routine is used by any local time stepping. The idea is that we first mimick subcycling: Let there be a global max time step size. We assume this one arises on the coarsest mesh level. So patches either do this time step or they do a time step that is \( 3^{-k} \) that size.

Pitfalls

We expect a min time step size that we use globally. We find the biggest \( discretisationSteps^k \cdot minGlobalTimeStepSize < cellTimeStepSize \) value through k which still meets the stability of cellTimeStepSize. We then return this value.

Truncation

If the eigenvalues become very small within a cell, we end up with huge time step sizes. This should not happen. So I expect the global time step size (largest value) and truncate the time step size in any case by this one.

Decreasing time step sizes

I use the global minimal time step size to kick off the analysis. This fails if the admissible global time step size shrinks over time. Therefore, the cell's time step size can be smaller than the globally admissible time step size. It simply means that the global time step size is shrinking and that the argument we get is lagging behind as we haven't finished the current time step yet.

Parameters

discretisationSteps	Pass in zero to allow a totally anarchic time stepping. Use one to construct subcycling without a re-adoption of the time step sizes or something negative to switch the bucketing off.
maxGlobalTimeStepSize	Maximum global time step size (of previous time step). I use this one to truncate too big time step sizes. Overall, I expect the time step size not to grow by more than 10 percent.
cellTimeStepSize	Time step size the patch would like to pick.

◆ getDefaultPrecisionCommand()

PrecisionCommand exahype2::getDefaultPrecisionCommand ( )

The default is double as this is the highest precision available per default.

◆ getDefaultRefinementCommand()

RefinementCommand exahype2::getDefaultRefinementCommand ( )

The default is coarsen as this is the lowest priority command.

◆ getInterpolationWeights()

std::pair< double, double > exahype2::getInterpolationWeights	(	double	oldTimeStampOnFace,
		double	newTimeStampOnFace,
		double	cellTimeStamp
	)

A posteriori fixes of time stamps

I'm afraid of accumulation errors in the time stamps for the small cells which might end up lacking behind or running ahead. I can recognise such situations by very small weights for the old time snapshot. However, I don't fix it here. For fixing such a divergence, I rely on an explicit cleanup later on.

Parameters

cellTimeStamp This is an in-out parameter, as we can adopt the stamp size to avoid the accummulation of rounding errors.

Returns: Weight of (old,new) data.

◆ getMinTimeStampOfNeighboursAhead()

template<typename CellLabel , typename FaceLabel >

double exahype2::getMinTimeStampOfNeighboursAhead	(	const CellLabel &	cellLabel,
		const peano4::datamanagement::FaceEnumerator< FaceLabel > &	faceLabelEnumerator
	)

Similar to getMinTimeStampOfNeighbours(), but we minimise only over those neighbours that are actually ahead.

If no neighbour is ahead or one lags behind, we return the time stamp of cellLabel.

Using code/algorithms

Local timestepping codes can run into situations where the eigenvalue per cell is zero. This means that nothing happens in such cell. Therefore, the cell cannot advance in time. We don't know how far we can/should march. In this case, ExaHyPE offers two strategies: We can use the global admissible time step size, i.e. the global maximum eigenvalue, to determine a time step size; or we can see whether one of the neighbours is ahead and catch up.

The former method leads to a staircase pattern, as some cells where nothing happens race forward in time. A description of this behaviour is found in the Python routine referenced below.

Deadlocks

I've seen multiple cases where

See also: exahype2.solvers.fv.kernels.create_compute_new_time_step_size_kernel_for_local_time_stepping

Definition at line 61 of file TimeStepping.h.

References assertion, tarch::la::greater(), and tarch::la::smaller().

Here is the call graph for this function:

◆ getUnitTests()

tarch::tests::TestCase * exahype2::getUnitTests ( )

Please destroy after usage.

◆ isMeshAdaptationRequired()

bool exahype2::isMeshAdaptationRequired ( )

Does the mesh have to be adapted or the load to be rebalanced?

runSimulation() asks this predicate right after each time step. It holds if

exahype2::RefinementControlService holds committed refinement or erase events, i.e. a time step has asked for a mesh change which nobody has realised yet, or
a load balancing strategy is installed and has split recently (toolbox::loadbalancing::AbstractLoadBalancing::hasSplitRecently()), i.e. it has triggered a rebalancing which Peano still has to carry out.

These are the same predicates which the repeat conditions of the mesh construction use. As the predicate is evaluated on the global master only, both refer to this rank's view of the simulation.

◆ removeTimeStepAccumulationErrorsFromCell()

template<typename CellLabel , typename FaceLabel >

double exahype2::removeTimeStepAccumulationErrorsFromCell	(	const CellLabel &	cellLabel,
		const peano4::datamanagement::FaceEnumerator< FaceLabel > &	faceLabelEnumerator,
		double	timeStepSize
	)

Remove accumulation errors from cell time stamps.

If we add time step sizes to the time stamp over and over again, it is very likely that the time stamps at resolution boundaries do not match at one point. If we use subcycling, 3xdt/3 usually does not equal the dt of a big cell. So it makes sense to amend the time step sizes slightly to help us to synchronise time stamps between cells, i.e. to avoid that they are off by a tiny bit and hence mess up the time stepping.

So what I do first of all is to check if the current (update) cell time stamp does equal a neighbour. If there are multiple of matching neighbours, I pick the one with the biggest time stamp. This check for equality is a double precision check which should use relative quantities, as time stamps can be arbitrarily large. So once I find a matching time stamp, I return this time stamp rather than the current time step size and, hence, sync the current time step size with the neighbours. This can mean that I reduce or increase the time step size slightly, i.e. I assume that enough slack is built into the chosen time step size such that a tiny increase does not blow up the result.

Relative accuracy

If two time stamps are equal depends on the magnutide of the time step size. It does not matter what the time stamps are, as any "rounding" decision should be time translation invariant. However, if I have a large time step size, then I'd be happy to alter it more rigorously than for a stiff problem with tiny time step sizes. Overall, I accept around one percent of the time step sizes as acceptable alteration of the outcome.

Usage of routine within local time stepping

If we have subcycling, then we know that timeStepSize is never zero, as we use the largest (original) eigenvalue within the system and derive the levels' time step sizes from there. Things are different with local time stepping, where each cell can have its own time step size. If a cell has a size of zero, then we always should derive the admissible dt from the neighbours.

See also: getMinTimeStampOfNeighboursAhead() for a discussion how to make zero eigenvalue patches catch up over time.

Parameters

timeStepSize Time step size the code would wish to use

Returns: Time step size the code should use eventually

Definition at line 200 of file TimeStepping.h.

References assertion2, tarch::la::equals(), tarch::la::greaterEquals(), logDebug, and tarch::la::NUMERICAL_ZERO_DIFFERENCE.

Here is the call graph for this function:

◆ runSimulation()

void exahype2::runSimulation	(	peano4::RunMultipleSteps &	setupSimulation,
		peano4::RunMultipleSteps &	plot,
		peano4::RunMultipleSteps &	timeStep,
		peano4::RunMultipleSteps &	meshAdaptation,
		peano4::RunMultipleSteps::Report &	report,
		std::function< bool()>	wantsToPlot,
		std::function< double()>	getMinTimeStamp,
		std::function< double()>	getMaxTimeStamp,
		std::function< double()>	getMaxTimeStepSize,
		double	minTerminalTime,
		double	maxTerminalTime
	)

Run an ExaHyPE 2 simulation.

Every ExaHyPE 2 project consists of four runners: one setting up the simulation (mesh construction and initial conditions), one plotting the solution, one running a time step, and one adapting the mesh and rebalancing the load between two time steps. The routine realises the main loop over these runners:

The global master runs setupSimulation once. Afterwards, it alternates between plotting and time stepping as long as the minimum time stamp has not reached minTerminalTime, the maximum time stamp has not reached maxTerminalTime, and the maximum time step size is positive. A plot is only triggered if wantsToPlot() holds, and never twice in a row, as a plot does not advance the simulation. Once the loop has terminated, the master plots a final time if wantsToPlot() holds.
Right after each time step, the global master runs meshAdaptation if isMeshAdaptationRequired() holds. It never runs meshAdaptation if this runner has no mesh sweeps at all, i.e. if no solver has contributed to the mesh adaptation phase. If the mesh adaptation has changed the mesh, it resets the time step size to zero (exahype2::Solver::resetTimeStepSize()), so the subsequent time step does not advance the time stamps. This time step hence is not taken into account by the stagnation check below.
After each time step, the global master checks whether the minimum and the maximum time stamp have changed. If neither has changed, it writes a warning, as a single step might legitimately not advance in time (a solver might for example reject and rerun a step). If they do not change in the subsequent time step either, the simulation is stuck. The master writes an error and leaves the loop as if it had reached the terminal time, i.e. it plots a final time if wantsToPlot() holds and then terminates regularly. Any time step in which a time stamp changes resets this check.
All other ranks (Peano workers) run whatever program step the master has selected until the master tells them to terminate.

Each runner which has run is reported to report, and report's final results are written once the loop has terminated. With OpenMP, the whole loop runs within the master thread of a parallel region, so Peano can spawn tasks.

The routine replaces the main loop which exahype2.api.generators.DefaultSimulationMain used to generate. The time stamps and step sizes stem from the project's solver repository, so the generated main() hands them over as functors.

Parameters

setupSimulation	Runner setting up the simulation.
plot	Runner plotting the solution.
timeStep	Runner running one time step.
meshAdaptation	Runner adapting the mesh and rebalancing the load. Its getFirstSubstep() is peano4::parallel::Node::UndefProgramStep if it has no mesh sweeps.
report	Report to which each run of a runner is reported.
wantsToPlot	Does any solver want to plot?
getMinTimeStamp	Minimum time stamp over all solvers.
getMaxTimeStamp	Maximum time stamp over all solvers.
getMaxTimeStepSize	Maximum time step size over all solvers.
minTerminalTime	The simulation terminates once the minimum time stamp reaches this value.
maxTerminalTime	The simulation terminates once the maximum time stamp reaches this value.

◆ runTimeStepOnCell()

template<typename CellLabel , typename FaceLabel >

bool exahype2::runTimeStepOnCell	(	const CellLabel &	cellLabel,
		const peano4::datamanagement::FaceEnumerator< FaceLabel > &	faceLabelEnumerator,
		double	globalMinTimeStamp
	)

Determine whether to run a time step on a cell by analysing the neighbouring cells' timestamp.

These timestamps are stored within the face labels. We update a cell if all the neighbours are at the same time stamp or ahead.

See also: removeTimeStepAccumulationErrorsFromCell() for a discussion of precisions we use for the checks above

For some reason that I haven't understood completely yet, the above decision pattern can still lead to a deadlock where no cell in the global domain does advance in time. Which is really weird, as there always should be a global minimum. My guess is that something along the AMR boundaries does not work properly.

Parameters

cellLabel

Definition at line 132 of file TimeStepping.h.

References tarch::la::equals(), exahype2::internal::getMinTimeStampOfNeighbours(), and tarch::la::greaterEquals().

Here is the call graph for this function:

◆ setDefaultLogStatements()

void exahype2::setDefaultLogStatements ( )

◆ shutdownRuntime()

void exahype2::shutdownRuntime ( std::function< void()> shutdownDatatypes )

Shut down the runtime of an ExaHyPE 2 simulation.

Counterpart of configureRuntime(), plus the shutdown of everything main() has initialised: the routine writes the statistics, shuts down the load balancing, Peano's singletons, the project's data types, SmartMPI, the non-critical assertions, and finally the parallel environment.

Parameters

shutdownDatatypes Shuts down the project's data types, i.e. typically the project's repositories::DataRepository::shutdownDatatypes(). It is called after Peano's singletons are down and before SmartMPI and the parallel environment are shut down.

◆ volumeIndex()

template<class... Args>

auto exahype2::volumeIndex ( Args... args )

Definition at line 54 of file VolumeIndex.h.

References exahype2::VolumeIndex< N, Args >::generate().

Referenced by exahype2::fv::riemann::loopbodies::copySolution(), exahype2::enumerator::AoSLexicographicEnumerator::operator()(), exahype2::enumerator::AoSoALexicographicEnumerator::operator()(), and exahype2::enumerator::SoALexicographicEnumerator::operator()().

Here is the call graph for this function:

Here is the caller graph for this function:

Variable Documentation

◆ ASCII_ART_HEADER

str exahype2.ASCII_ART_HEADER

Initial value:

=  r"""
 ___          _  _      ___ ___
| __|_ ____ _| || |_  _| _ \ __|
| _|\ \ / _` | __ | || |  _/ _|
|___/_\_\__,_|_||_|\_, |_| |___|
                   |__/
"""

Definition at line 15 of file __init__.py.

◆ COPYRIGHT_HEADER

str exahype2.COPYRIGHT_HEADER = "ExaHyPE 2x extension"

Definition at line 22 of file __init__.py.

◆ havematplotlib

bool exahype2.havematplotlib = False

Definition at line 62 of file __init__.py.

◆ havempmath

bool exahype2.havempmath = False

Definition at line 37 of file __init__.py.

◆ havenumpy

bool exahype2.havenumpy = False

Definition at line 28 of file __init__.py.

◆ havesympy

bool exahype2.havesympy = False

Definition at line 51 of file __init__.py.

Namespaces

Data Structures

Enumerations

Functions

Variables

Detailed Description

Enumeration Type Documentation

◆ PrecisionCommand

◆ RefinementCommand

Function Documentation

◆ configureRuntime()

◆ discretiseAndTruncateTimeStepSizes()

Pitfalls

Truncation

Decreasing time step sizes

◆ getDefaultPrecisionCommand()

◆ getDefaultRefinementCommand()

◆ getInterpolationWeights()

A posteriori fixes of time stamps

◆ getMinTimeStampOfNeighboursAhead()

Using code/algorithms

Deadlocks

◆ getUnitTests()

◆ isMeshAdaptationRequired()

◆ removeTimeStepAccumulationErrorsFromCell()

Relative accuracy

Usage of routine within local time stepping

◆ runSimulation()

◆ runTimeStepOnCell()

◆ setDefaultLogStatements()

◆ shutdownRuntime()

◆ volumeIndex()

Variable Documentation

◆ ASCII_ART_HEADER

◆ COPYRIGHT_HEADER

◆ havematplotlib

◆ havempmath

◆ havenumpy

◆ havesympy