Long-running simulations on a cluster have to survive crashes, queue limits and the occasional "let's rerun it with a different end time". A checkpoint is a snapshot which holds the whole state of a solver, so a simulation can continue from it instead of starting from scratch. A checkpoint is different from a plot: A plot writes what you want to look at, with a filter and in single precision if you wish. A checkpoint writes everything the solver needs, without loss.
ExaHyPE 2 writes checkpoints through the same mechanism as plots. It stores them in Peano's patch file format, which exists in two flavours: a human-readable ASCII flavour, and a binary flavour which stores the values as raw doubles. In this tutorial, we
Configuring the environment
As always, we make Peano's Python API available:
import sys, os
sys.path.insert(0, os.path.abspath("../../src"))
import exahype2
The solver
The solver is the one of the Euler tutorial. To keep the notebook self-contained, we do not edit the generated C++ file but inject the flux and the eigenvalue from Python (see the end of A first advection problem with the Euler equations). The initial condition is a sine density profile over the periodic domain, i.e. a density wave which is transported to the right. We use a load balancing which spreads the mesh over all threads once it has been constructed, so we end up with several subdomains which are all processed in parallel.
namespace = ["tutorials", "exahype2", "checkpointing"],
project_name = "Euler",
directory = "36_checkpointing",
executable = "euler",
)
patch_size = 9
def create_solver(create_grid_construction):
"EulerFV",
4,
0,
)
solver.add_user_solver_includes("#include <cmath>\n#include <algorithm>\n")
solver.add_pde_term(
implementation = '''
const double irho = 1.0 / Q[0];
constexpr double gamma = 1.4;
const double p = (gamma - 1.0) * (Q[3] - 0.5 * irho * (Q[1] * Q[1] + Q[2] * Q[2]));
switch (normal) {
case 0:
F[0] = Q[1];
F[1] = irho * Q[1] * Q[1] + p;
F[2] = irho * Q[1] * Q[2];
F[3] = irho * Q[1] * (Q[3] + p);
break;
case 1:
F[0] = Q[2];
F[1] = irho * Q[2] * Q[1];
F[2] = irho * Q[2] * Q[2] + p;
F[3] = irho * Q[2] * (Q[3] + p);
break;
}
''',
stateless = False,
)
)
solver.add_pde_term(
implementation = '''
const double irho = 1.0 / Q[0];
constexpr double gamma = 1.4;
const double p = (gamma - 1.0) * (Q[3] - 0.5 * irho * (Q[1] * Q[1] + Q[2] * Q[2]));
const double c = std::sqrt(gamma * p * irho);
lambdaMax[0] = std::max(std::abs(Q[1 + normal] * irho + c), std::abs(Q[1 + normal] * irho - c));
''',
stateless = False,
)
)
solver.add_pde_term(
implementation = '''
constexpr double gamma = 1.4;
constexpr double p = 1.0;
constexpr double u = 1.0;
const double rho = 1.0 + 0.2 * std::sin(2.0 * M_PI * (x[0] + 0.1) / 0.2);
Q[0] = rho;
Q[1] = rho * u;
Q[2] = 0.0;
Q[3] = p / (gamma - 1) + 0.5 * rho * u * u;
''',
stateless = False,
)
)
solver.grid_construction = create_grid_construction(solver)
return solver
solver = create_solver(
)
The conservative flux F(Q) of a hyperbolic PDE.
Point-wise initial condition: given the physical position of a single degree of freedom,...
The maximum absolute eigenvalue of the flux Jacobian in a given direction, i.e.
Abstract base class for all ExaHyPE 2 solvers.
Finite Volumes with a Rusanov Riemann solver and explicit Euler time stepping, realised over a block-...
Setup phase counterpart to RusanovWithExplicitEuler: sets the initial condition for a Finite Volume s...
One plotter and two checkpoints
A solver can have an arbitrary number of plotters (exahype2.api.Solver.Solver.add_plotter()). Each plotter has a name which is unique per solver, and each plotter writes into files of its own. We add
- the usual exahype2.api.fv.PlotAsVTKFile, which dumps the solution for visualisation every 0.01 time units;
- a checkpoint in ASCII format. exahype2.api.fv.WriteCheckpoint writes the whole state of the solver every 0.02 time units. We pass
binary=False;
- the same checkpoint again, but in binary format. This is the default of WriteCheckpoint, we spell it out for clarity.
The plotter names default to the class names. As we have two checkpoints, we have to name them explicitly. We also give them different file names.
solver.add_plotter(
solver,
first_plot_time_stamp = 0.0,
time_in_between_plots = 0.01,
)
)
solver.add_plotter(
solver,
first_plot_time_stamp = 0.0,
time_in_between_plots = 0.02,
filename = "checkpoint-ascii",
name = "AsciiCheckpoint",
binary = False,
)
)
solver.add_plotter(
solver,
first_plot_time_stamp = 0.0,
time_in_between_plots = 0.02,
filename = "checkpoint-binary",
name = "BinaryCheckpoint",
binary = True,
)
)
Plotting counterpart to RusanovWithExplicitEuler: maps the patch data of a Finite Volume scheme with ...
Write checkpoints of a Finite Volume solver.
A checkpoint writes the patches together with their halo, i.e. it stores more than a plot. It does so with 17 significant digits by default, so reading the checkpoint back is lossless. Peano's plot task graph runs all plotters which are active at a time stamp in one go, i.e. the checkpoints at multiples of 0.02 come together with a plot.
Global parameters and load balancing
We run the code until \( t=0.05\) . With a checkpoint every 0.02 time units, we get checkpoints at \( t \approx 0, 0.02, 0.04\) . Each plotter becomes active as soon as the solver's time stamp has passed the next output time, i.e. the snapshots are written at the first time step after the nominal time stamp.
project.add_solver(solver)
project.set_global_simulation_parameters(
dimensions = 2,
offset = [-0.1, -0.1],
size = [ 0.2, 0.2],
min_end_time = 0.05,
max_end_time = 0.05,
periodic_BC = [True, True, True],
)
project.set_load_balancing(
"toolbox::loadbalancing::strategies::SpreadOutOnceGridStagnates",
"new ::exahype2::LoadBalancingConfiguration()",
)
project.set_Peano4_installation(os.path.abspath("../../"), peano4.api.output.CompileMode.RELEASE)
peano4_project = project.generate_Peano4_project()
peano4_project.generate()
Build and run
We compile and run with four threads. Each thread owns a subdomain, and each subdomain writes its own part of every snapshot. If you built Peano with MPI, mpirun -np 2 ./euler works the same way; the files then are distributed over the ranks.
!cd 36_checkpointing && make -j > make.log 2>&1 && OMP_NUM_THREADS=4 ./euler > run.log 2>&1 && echo done
What got written
Each plotter writes an index file ending with .peano-patch-file, plus one data file per subdomain and snapshot. The VTK plotter writes a snapshot every 0.01 time units, the checkpoints every 0.02 time units:
!cd 36_checkpointing && ls *.peano-patch-file
!cd 36_checkpointing && ls checkpoint-binary-EulerFV-* | head
The index files are always text files. They list the snapshots and carry the solver's state in a meta data block. Let's peek into the head of a data file of the ASCII checkpoint. Besides the format, the header lists the mesh and the meta data (time stamp, admissible time step size, ...) which the restart will use:
!cd 36_checkpointing && f=$(ls checkpoint-ascii-EulerFV-*.peano-patch-file | grep -v "^checkpoint-ascii-EulerFV.peano-patch-file$" | head -1) && head -n 25 $f
ASCII versus binary
The ASCII flavour writes each value with 17 significant digits as text. The binary flavour writes the 8 bytes of the double as they are. Binary files are hence smaller, they are written and read faster, and nothing is rounded. The price: you cannot read them in a text editor. (The patch file readers of Peano's Python API, the probe and render scripts, and the merge tool handle both flavours transparently.) Let's compare the footprint of the two checkpoints and of the VTK plot:
import glob
def size_in_mb(prefix):
total = sum(os.path.getsize(f) for f in glob.glob("36_checkpointing/" + prefix + "*"))
return total / 1024.0 / 1024.0
sizes = {
"VTK plot (binary)": size_in_mb("solution-EulerFV"),
"Checkpoint (ASCII)": size_in_mb("checkpoint-ascii-EulerFV"),
"Checkpoint (binary)": size_in_mb("checkpoint-binary-EulerFV"),
}
for name, mb in sizes.items():
print(f"{name:22s} {mb:9.2f} MB")
print(f"ASCII/binary ratio: {sizes['Checkpoint (ASCII)'] / sizes['Checkpoint (binary)']:.1f}")
import matplotlib.pyplot as plt
plt.bar(list(sizes.keys()), list(sizes.values()), color=["tab:gray", "tab:red", "tab:blue"])
plt.ylabel("MB on disk")
plt.title("Output volume of the whole run")
plt.show()
On the author's machine, the three ASCII checkpoints occupy about 25 MB, while the three binary checkpoints take about 8.5 MB, i.e. the binary files are roughly three times smaller. Keep in mind that this is the volume of the whole run: all snapshots of the VTK plotter (six) and all snapshots of the checkpoints (three). It also depends on the patch size, as a checkpoint stores a halo around each patch. The relative saving of binary vs. ASCII, however, is robust: a double occupies 8 bytes in a binary file, but around 24 bytes in ASCII with 17 digits, which is where the factor of three comes from.
You can also reduce the footprint by writing an ASCII checkpoint with fewer digits (plotter_precision), but then you give up the lossless restart. We recommend binary checkpoints for production runs and ASCII checkpoints for debugging and small experiments where you want to look into the data.
Restarting from a checkpoint
A restart replaces the grid construction. Instead of exahype2.api.fv.SetupSimulationOverRegularGrid, we use exahype2.api.blockstructured.LoadCheckpoint.LoadCheckpoint (available as exahype2.api.fv.LoadCheckpoint). It reads the mesh and the data of the checkpoint and restores the time stamps of the solver. It detects the format of the files on its own, i.e. we can restart from the binary and from the ASCII checkpoint alike. We restart from the binary one, and run the simulation until \( t=0.1\) .
The restart is a project of its own. The solver has to have the same unknowns, auxiliary variables and patch size, and the domain has to be the same. The Euler solver hence is the same code as above; the implementation of the PDE terms is copied over by wrapping the setup into a function.
namespace = ["tutorials", "exahype2", "checkpointing"],
project_name = "Euler",
directory = "36_checkpointing-restart",
executable = "euler",
)
restart_solver = create_solver(
s,
filename = "checkpoint-binary",
output_path = "../36_checkpointing",
)
)
restart_solver,
first_plot_time_stamp = 0.0,
time_in_between_plots = 0.01,
)
restart_project.add_solver(restart_solver)
restart_project.set_global_simulation_parameters(
dimensions = 2,
offset = [-0.1, -0.1],
size = [ 0.2, 0.2],
min_end_time = 0.1,
max_end_time = 0.1,
periodic_BC = [True, True, True],
)
restart_project.set_load_balancing(
"toolbox::loadbalancing::strategies::SpreadOutOnceGridStagnates",
"new ::exahype2::LoadBalancingConfiguration()",
)
restart_project.set_Peano4_installation(os.path.abspath("../../"), peano4.api.output.CompileMode.RELEASE)
restart_project.generate_Peano4_project().generate()
Set up a simulation from a checkpoint.
The checkpoint is read at runtime, i.e. the files only have to exist once we start the code. The plotter of the restarted run starts at \( t=0\) as well; the restart notices that the checkpoint was taken at \( t \approx 0.04\) and moves the plotter on to its next output, so it does not write snapshots that the first run has written already.
!cd 36_checkpointing-restart && make -j > make.log 2>&1 && OMP_NUM_THREADS=4 ./euler > run.log 2>&1 && echo done
!cd 36_checkpointing-restart && grep "CheckpointReader" run.log | cut -c1-300
The log line above shows that the code has picked the latest checkpoint, written at \( t = 0.040069\) , and that it has read all 729 patches. The simulation continues from there until \( t=0.1\) . The plotter has been moved on as well: The first run has written the snapshot at \( t \approx 0.04\) already, so the restarted run continues with the plot at \( t \approx 0.05\) instead of repeating it. The plotter's index file confirms this (the empty snapshot at \( t=0\) is a placeholder that the restart writes into every index file):
!grep "timestamp" 36_checkpointing-restart/solution-EulerFV.peano-patch-file | head -4
Where to go from here
- Checkpoints are written once per plot phase; the interval can be chosen independently of the plot interval.
- The format of the files is described in the visualisation chapter. If you have a run with many ranks and want to inspect a snapshot, the merge tool described there stitches the files of all subdomains together.
- Combine checkpoints with the offline load balancing: a restarted run can use a different decomposition than the run that wrote the checkpoint.
1.9.8