Peano
Loading...
Searching...
No Matches
Additional solver steps: postprocessing the solution after each update
This documentation is autogenerated from ./tutorials/exahype2/20_additional-solver-steps.ipynb. Once you clone the Peano repository, please work against this notebook or study Peano's generic description how to generate web docu from Jupyter notebooks.

Many applications have to touch the solution after each time step: They clip unphysical values such as negative densities, project the solution back onto a constraint manifold, or enforce algebraic constraints of the PDE. ExaGRyPE's CCZ4 formulation of the Einstein equations, for example, requires the conformal metric to have unit determinant and the conformal extrinsic curvature to be trace-free. Such operations are no part of the PDE terms. They are additional steps of the solver, which have to run after each update of the solution.

In this tutorial, we add such a step to the Euler setup of A first advection problem with the Euler equations and Higher-order solvers: Runge Kutta and Finite Differences. We run through the following steps:

  • how ExaHyPE describes a time step as a task graph;
  • how to find the tasks which update the solution, and how to insert a step after them;
  • where the step ends up for the Finite Volumes and for Runge-Kutta schemes;
  • which contract the numerical schemes and the inserted steps have to obey.

We assume that you have completed Higher-order solvers: Runge Kutta and Finite Differences, as we reuse its PDE terms without further explanation.

Time steps as task graphs

ExaHyPE does not hard-code what happens within a time step. Instead, the numerical scheme of each solver describes a time step as a task graph (peano4.api.taskgraph.Graph): Each task is one logical step such as "reconstruct the patch with its halo" or "update the cell", and the tasks' dependencies say which step has to finish before another one may start. The ExaHyPE project hands this graph over to a task graph compiler, which maps it onto as few mesh traversals as possible. If you run a script with the logging verbosity TEXT, ExaHyPE prints the task graphs that it hands over to the compiler.

An additional solver step hence is an additional task. It has to be wired into the graph such that it reads the new solution once this solution has been computed, and before anybody else reads it. For the Finite Volumes, this means after the cell update, but before the solution is projected onto the faces, where the neighbours pick it up, and before the CFL condition of the next time step is evaluated.

Configuring the environment

As in the previous tutorials, we add Peano's Python API to the search path:

import sys, os
sys.path.insert(0, os.path.abspath("../../src"))
import peano4.api
import exahype2
try:
import ipywidgets, matplotlib
except ImportError:
!{sys.executable} -m pip install ipywidgets matplotlib

Project, solver and PDE terms

The project and the solver are those of Higher-order solvers: Runge Kutta and Finite Differences. The only difference is that the numerical scheme is an instance of a subclass of one of ExaHyPE's schemes, which adds the additional step. We start with the Finite Volumes.

namespace = ["tutorials", "exahype2", "additionalsteps"],
project_name = "AdditionalSteps",
directory = "20_additional-solver-steps",
executable = "additional-steps",
)
ExaHyPE 2 project.
Definition Project.py:15

Adding a step after each solution update

Our additional step is a density floor: Whenever the density in a volume drops below a minimum density, we raise it to this minimum and keep the velocity and the pressure. This is a crude version of the positivity fixes that many codes use to keep densities and pressures positive. To see an effect, we pick an unrealistically high floor of 0.9. Our density wave oscillates between 0.8 and 1.2, i.e. the floor cuts off its troughs.

To insert the step, we have to find the tasks which update the solution. We do not want to know how a particular numerical scheme builds its graph, though. The schemes therefore tag the relevant tasks. Every task (peano4.api.taskgraph.Task) carries a set of tags, i.e. of plain strings, which the task graph compilers ignore. Each numerical scheme adds the tag exahype2.api.Solver.SolutionUpdateTaskTag to every task which writes a new solution. Runge-Kutta schemes add exahype2.api.Solver.IntermediateSolutionUpdateTag to every task which writes the input of a further stage, i.e. an intermediate solution.

Inserting a step then always follows the same pattern:

  1. find_tasks() of the time stepping graph (peano4.api.taskgraph.Graph) returns all tasks which carry at least one of the tags passed.
  2. For each of them, we create a new task.
  3. insert_after(anchor, task) adds the new task to the graph, makes it depend on the tagged task, and redirects all tasks which have depended on the tagged task so far to the new task.

The place to do this is the numerical scheme: Its routine construct_output() builds the time stepping graph. We subclass the scheme, call the superclass' construct_output() to get the original graph, and then insert our step. The scheme is the right place, as it decides on the data structures: It knows which patch its tagged tasks write and which cells they update. We write the insertion as a plain function, so we can reuse it for further schemes, and let the subclass pass in the patch:

density_floor = '''
constexpr double gamma = 1.4;
constexpr double minDensity = 0.9;
if (Q[0] < minDensity) {
const double u = Q[1] / Q[0];
const double v = Q[2] / Q[0];
const double p = (gamma - 1.0) * (Q[3] - 0.5 * Q[0] * (u * u + v * v));
Q[0] = minDensity;
Q[1] = minDensity * u;
Q[2] = minDensity * v;
Q[3] = p / (gamma - 1.0) + 0.5 * minDensity * (u * u + v * v);
}
'''
def insert_density_floor(solver, time_step, tag, patch):
for anchor in time_step.find_tasks(tag):
if not anchor.name.startswith(solver.name + "_"):
continue
name = f"{solver.name}_Density floor after {anchor.name[len(solver.name) + 1:]}",
entity = peano4.api.taskgraph.Task.GridEntity.CELL,
)
task.use_cell(patch, peano4.api.taskgraph.Task.DataAccess.READ_AND_WRITE)
task.add_action_set(
toolbox.blockstructured.api.ApplyPointWiselyToPatch(
patch = patch,
implementation = density_floor,
guard = solver.grid_selector.createCellGuard(),
additional_includes = solver._get_default_includes() + solver.user_solver_includes,
)
)
time_step.insert_after(anchor, task)
class RusanovWithDensityFloor(exahype2.api.fv.RusanovWithExplicitEuler):
def construct_output(self, exahype2_project, peano_project, time_step):
super().construct_output(exahype2_project, peano_project, time_step)
insert_density_floor(self._solver, time_step, exahype2.api.Solver.SolutionUpdateTaskTag, self._solver._patch)
if peano4.api.Logger().is_text:
time_step.log_as_text(title=f"Time stepping task graph for {self._solver.name} with density floor")
Finite Volumes with a Rusanov Riemann solver and explicit Euler time stepping, realised over a block-...
Singleton logger for the Peano 4 Python API.
Definition Logger.py:37
One task within a peano4.api.taskgraph.Graph.
Definition Task.py:8
The task graph abstraction allows users to formalise their solver steps over small task graphs which ...
Definition __init__.py:1

A few details are worth a closer look:

  • The tags say what a task does, but not which data it writes. The scheme knows this: The Finite Volumes write the new solution into the patch _patch, which they have created themselves and attached to the solver. Manipulate an existing solver's steps (e.g. inserting additional sweeps) lists the tagged tasks of all numerical schemes.
  • The task names carry the solver's name as prefix. A project with several solvers holds the tasks of all of them in one graph, so we skip the tasks of other solvers.
  • toolbox.blockstructured.api.ApplyPointWiselyToPatch runs our C++ snippet for each volume of the patch. Within the snippet, Q points to the unknowns of one volume, which the snippet may read and write, and x holds the position of the volume's centre. The guard restricts the step to the cells of our solver.
  • We declare that the new task reads and writes the patch. The task graph compiler relies on these declarations when it decides which tasks it may fuse into one mesh traversal.
  • We insert the task after the anchor rather than adding a plain dependency. A plain dependency would allow the projection onto the faces to run before our task, i.e. the neighbours could see data which has not been postprocessed yet.

If you want to insert several steps after the same anchor, chain them: Use the task which you have just inserted as the anchor of the next one.

With the class at hand, we create the solver as in the previous tutorials, but use RusanovWithDensityFloor as its numerical scheme:

"EulerFV",
4, # unknowns
0, # auxiliary variables
)
solver.add_user_solver_includes("#include <cmath>\n#include <algorithm>\n")
solver.add_pde_term(exahype2.api.Flux(
implementation = '''
const double irho = 1.0 / Q[0];
constexpr double gamma = 1.4;
const double p = (gamma - 1.0) * (Q[3] - 0.5 * irho * (Q[1] * Q[1] + Q[2] * Q[2]));
switch (normal) {
case 0:
F[0] = Q[1];
F[1] = irho * Q[1] * Q[1] + p;
F[2] = irho * Q[1] * Q[2];
F[3] = irho * Q[1] * (Q[3] + p);
break;
case 1:
F[0] = Q[2];
F[1] = irho * Q[2] * Q[1];
F[2] = irho * Q[2] * Q[2] + p;
F[3] = irho * Q[2] * (Q[3] + p);
break;
}
''',
stateless = False,
))
solver.add_pde_term(exahype2.api.MaxEigenvalue(
implementation = '''
const double irho = 1.0 / Q[0];
constexpr double gamma = 1.4;
const double p = (gamma - 1.0) * (Q[3] - 0.5 * irho * (Q[1] * Q[1] + Q[2] * Q[2]));
const double c = std::sqrt(gamma * p * irho);
const double u = Q[1 + normal] * irho;
lambdaMax[0] = std::max(std::abs(u + c), std::abs(u - c));
''',
stateless = False,
))
solver.add_pde_term(exahype2.api.InitPointWisely(
implementation = '''
constexpr double gamma = 1.4;
constexpr double p = 1.0;
constexpr double u = 1.0;
const double rho = 1.0 + 0.2 * std::sin(2.0 * M_PI * (x[0] + 0.1) / 0.2);
Q[0] = rho;
Q[1] = rho * u;
Q[2] = 0.0;
Q[3] = p / (gamma - 1) + 0.5 * rho * u * u;
''',
stateless = False,
))
patch_size = 9
volume_size = 0.0075
solver.numerical_scheme = RusanovWithDensityFloor(solver, patch_size)
solver.grid_construction = exahype2.api.fv.SetupSimulationOverRegularGrid(solver, volume_size=volume_size)
solver.plotter = exahype2.api.fv.PlotAsVTKFile(solver, first_plot_time_stamp=0.0, time_in_between_plots=0.05)
The conservative flux F(Q) of a hyperbolic PDE.
Definition Flux.py:8
Point-wise initial condition: given the physical position of a single degree of freedom,...
The maximum absolute eigenvalue of the flux Jacobian in a given direction, i.e.
Abstract base class for all ExaHyPE 2 solvers.
Definition Solver.py:17
Plotting counterpart to RusanovWithExplicitEuler: maps the patch data of a Finite Volume scheme with ...
Setup phase counterpart to RusanovWithExplicitEuler: sets the initial condition for a Finite Volume s...

The remaining steps are the same as in the previous tutorials:

project.add_solver(solver)
project.set_global_simulation_parameters(
dimensions = 2,
offset = [-0.1, -0.1],
size = [ 0.2, 0.2],
min_end_time = 0.2,
max_end_time = 0.2,
periodic_BC = [True, True, True],
)
project.set_Peano4_installation(os.path.abspath("../../"), peano4.api.output.CompileMode.RELEASE)
peano4_project = project.generate_Peano4_project(verbosity=peano4.api.LoggerVerbosity.TEXT)
peano4_project.generate()

Among the output, you find the time stepping task graph twice: once as the numerical scheme has created it, and once with the density floor. The picture below shows the latter:

Time stepping task graph of the Finite Volumes with the density floor

The task "EulerFV_Density floor after Update cell" sits directly behind the cell update. All tasks which used to depend on the cell update, i.e. the projection onto the faces and the evaluation of the CFL condition, now depend on the density floor. Whether the new task costs an additional mesh traversal is up to the task graph compiler. The default compiler maps each task onto a mesh traversal of its own. The new task reads and writes cell data only, though, so a fusing compiler such as peano4.api.taskgraph.compiler.FuseSteps merges it into the mesh traversal of the cell update.

We build and run the code:

!cd 20_additional-solver-steps && make -j4
!cd 20_additional-solver-steps && ./additional-steps
"20_additional-solver-steps/solution-EulerFV.peano-patch-file",
unknown = 0,
unknown_names = ["rho", "rho u_1", "rho u_2", "E"],
)
Interactive visualisation of Peano patch files within Jupyter notebooks.
Definition notebook.py:1
show_solution(file_name, unknown=0, unknown_names=None, clim=None)
Interactive viewer for a patch file within a Jupyter notebook.
Definition notebook.py:159

After the first time step, the troughs of the wave are flat: The density never drops below 0.9.

Runge-Kutta: postprocessing after each stage

A Runge-Kutta scheme does not compute the new solution in one go. It computes a sequence of intermediate solutions, the stages, and evaluates the right-hand side of the PDE for each of them (Higher-order solvers: Runge Kutta and Finite Differences). If we postprocessed the final solution only, the right-hand side would be evaluated for stages which violate our constraints. For a density floor, a stage could even carry a negative density, which the flux cannot handle. The postprocessing therefore has to run after each stage, too.

The Runge-Kutta scheme exahype2.api.rkfd4.CentralDifferencesWithRungeKutta writes the inputs of the further stages into the patch _stage_input and tags these writers with exahype2.api.Solver.IntermediateSolutionUpdateTag. Its subclass hence calls our function twice, once per tag, and passes in the respective patch:

class RungeKuttaWithDensityFloor(exahype2.api.rkfd4.CentralDifferencesWithRungeKutta):
def construct_output(self, exahype2_project, peano_project, time_step):
super().construct_output(exahype2_project, peano_project, time_step)
insert_density_floor(self._solver, time_step, exahype2.api.Solver.SolutionUpdateTaskTag, self._solver._patch)
insert_density_floor(self._solver, time_step, exahype2.api.Solver.IntermediateSolutionUpdateTag, self._solver._stage_input)
Fourth-order central Finite Differences with explicit Runge-Kutta time stepping, realised over a bloc...

If we switch to the Finite Differences with Runge-Kutta time stepping solver.numerical_scheme = RungeKuttaWithDensityFloor(solver, patch_size=patch_size, rk_order=2) solver.grid_construction = exahype2.api.rkfd4.SetupSimulationOverRegularGrid(solver, volume_size=volume_size) solver.plotter = exahype2.api.rkfd4.PlotAsVTKFile(solver, first_plot_time_stamp=0.0, time_in_between_plots=0.05) the task graph below results. We use a Runge-Kutta scheme of order two to keep the picture small:

Time stepping task graph of Runge-Kutta of order two with the density floor

The density floor now runs twice per time step: After "Prepare stage 1", which computes the input of the second stage, and after "Complete time step", which computes the new solution. In both cases, it runs before the outcome is projected onto the faces. The neighbours hence see the postprocessed data only.

The picture below compares the density along \( y=0\) after one period for both schemes, with and without the floor:

Density along y=0 after one period with and without density floor

With Runge-Kutta, the floor cuts off the trough of the wave, while the rest of the wave remains on top of the exact solution. After one period, the smallest density is exactly 0.9: The floor holds for every stage, not only for the final solution of each time step.

The floor adds mass whenever it raises the density, i.e. the scheme is no longer conservative. For the Finite Volumes, this is particularly visible: Their numerical diffusion damps the wave such that it stays above the floor after a few time steps. Until then, however, the floor has added mass, and the whole profile ends up above the profile without floor. A density floor hence is a last resort which keeps a code alive, not a physical model. Constraint enforcement is different: It removes errors which the exact solution would not have in the first place.

The contract

The tags are a contract between the numerical schemes and any code which inserts steps. The numerical schemes have to tag all tasks which write a new solution or an intermediate solution with the respective tag, and they must not read this solution within the same task. This is why the Runge-Kutta scheme splits the computation of each stage into two tasks, "Prepare stage k" and "Project stage k onto faces": the postprocessing has to fit in between. If a scheme computes the new solution in several tasks, as the optimistic Finite Volumes exahype2.api.fv.RusanovWithOptimisticExplicitEuler do, it tags each of them. Manipulate an existing solver's steps (e.g. inserting additional sweeps) discusses the details.

You can use tags for your own purposes, too: Add any string to a task's set tags and search for it via find_tasks(). Prefix your tags, e.g. with the name of your library, so they cannot clash with the tags of others.