This documentation is autogenerated from ./applications/exahype2/gprdr/GPRDR.ipynb. Once you clone the Peano repository, please work against this notebook or study Peano's generic description how to generate web docu from Jupyter notebooks.

I've written this benchmark to reconstruct work done by Duo Li and Alice-Agnes Gabriel from LMU. Ensure you've set the Python path correct

export PYTHONPATH=../../../python

to Peano's Python directory before you invoke the notebook.

import os
import peano4.api
import exahype2

DaStGen 2 (C) www.peano-framework.org
Peano 4 (C) www.peano-framework.org
ExaHyPE 2 (C) www.peano-framework.org

The sympy submodule is not included by default. But we'll use the homogeneous BC implementation from this package, as I'm too lazy to implement it myself. Even though they are trivial.

import exahype2.sympy

Clean-up

Lets ensure that no left-over files stay in there. Lets in particular delete the executable and the output files.

output_files = [ f for f in os.listdir(".") if f.endswith(".peano-patch-file") or f.endswith(".vtu") or f.startswith("output")]
for f in output_files:
  os.remove(f)

Setup simulation

We create a new simulation. At the moment I can't non-cubic domains. I mean: I haven't tested it. This is something I have to do.

project = exahype2.Project( 
  namespace    = ["examples", "exahype2", "gprdr"], 
  project_name = "gprdr", 
  directory    = "." )

dimensions=2 
project.set_global_simulation_parameters(
  dimensions            = dimensions,
  offset                = [-10000.0,-10000.0], 
  size                  = [20000.0,20000.0],
  end_time              = 8.001,
  first_plot_time_stamp = 0.0, 
  time_in_between_plots = 0.01
)

Create the solver

I have a whole set of different solvers and will definitely create a few more over the next months. This guy is the most basic/primitive one. Lets ensure it works first. I work with patch-based AMR with patches of the size 5x5 volumes. This is, so far, also a regular grid setup.

mesh_size      = 300
time_step_size = 0.001
 
unknowns = {    # yeah, I like self-explaining names ;-)
  "rho":1,
  "v":  3,
  "s":  1,
  "J":  3,
  "A":  9,
  "alpha": 1,
  "lambda": 1,
  "mu": 1,
  "xi": 1,
  "xi1": 1,
  "sliding": 1,
  "sliding_xi": 1,
  "slip": 3
}
 
number_of_unknowns = 0
number_of_auxiliary_variables = 0
for i in unknowns:
  number_of_unknowns += unknowns[i]
 
print( "no of unknowns: ", number_of_unknowns)
 
pde = exahype2.sympy.PDE(unknowns=number_of_unknowns,auxiliary_variables=number_of_auxiliary_variables,dimensions=dimensions)
 
my_solver = exahype2.solvers.fv.GenericRiemannFixedTimeStepSize(
    name                = "GPRDR", 
    patch_size          = 5, 
    unknowns            = number_of_unknowns,
    auxiliary_variables = number_of_auxiliary_variables,
    time_step_size      = time_step_size, 
    min_h               = mesh_size,
    max_h               = mesh_size)
    
my_solver.set_implementation(
  boundary_conditions=pde.implementation_of_homogeneous_Neumann_BC(),
)   
 
project.add_solver( my_solver )    

no of unknowns:  27

Parallelise

Even this simple setup is quite compute-intensive, so it pays off to work with the parallel code base right from the start. If you have configured Peano with

--with-multithreading=omp

or even with

--with-multithreading=omp --with-mpi=mpiicpc

then the line below adds parallelisation:

project.set_load_balancing( "toolbox::loadbalancing::strategies::RecursiveBipartition" )

Configure the build

The idea is that the ExaHyPE2 front-end generates a Peano Python project. We can then ask this Peano project to generate all C++ code that we need. As we don't want to type in all the different configuration parameters (which compiler is used, which environment variables are to be used on this machine, ...) we first tell the ExaHyPE project to parse the setups we've selected through the initial ./configure call. Ah, and we also commit to a build flavour.

build_mode = peano4.api.output.CompileMode.Asserts
#build_mode = peano4.api.output.CompileMode.Release
 
project.set_Peano4_installation("../../..", build_mode)
 
peano4_project = project.generate_Peano4_project()

parse configure outcome ../../../src/Makefile to extract compile settings
used C++ compiler is g++
used Fortran compiler is gfortran

Next, we add all the Fortran routines that we need in our project. Peano 4/ExaHyPE 2 will automatically add it to the build environment later on.

peano4_project.output.makefile.add_Fortran_modules( [
  "MainVariables.f90",    "expintegrator_type.f90",  "expintegrator_linalg.f90",
  "SpecificVarEqn99.f90", "GPRmaterials.f90",        "expintegrator_linalg_fast33.f90",
  "ComplexGeometry.f90",  "expintegrator_ode.f90",   "expintegrator.f90"
])
 
peano4_project.output.makefile.add_Fortran_files( 
  [ "ODE.f90", "C2P-GPRDR.f90", "PDE.f90", "InitialData.f90", "Tools.f90", "CGinterface.f90" ]
)
 
# Intel compiler
#peano4_project.output.makefile.add_Fortran_flag( "-r8 -cpp -auto -qopenmp-simd -O2 -xCORE-AVX2 -fma -DDim2 -DALIGNMENT=32 -DEQNTYPED99 -DODESOLVER -DOPT_KERNELS -I." )
 
# This is for GNU
peano4_project.output.makefile.add_Fortran_flag( "-lstdc++ -fdefault-real-8 -fdefault-double-8 -cpp -std=legacy -ffree-line-length-512 -fPIC -DDim2 -DALIGNMENT=32 -DEQNTYPED99 -DODESOLVER -DOPT_KERNELS" )
peano4_project.output.makefile.add_linker_flag( "-lstdc++ -fPIC -lgfortran" )

I don't want to program all the constants manually in my C++ code later on, so I simply export them from the Python script into the C++ stuff. They will be found in a generated file called Constants.h.

peano4_project.constants.export( "NumberOfUnknowns", number_of_unknowns )

peano4_project.constants.export_string( "Scenario", "TPV3" ) # called reference in the ExaHyPE 1 project, but I find scenario better

OK, time to generate the actual C++ stuff. After that, we can compile. We can either use the Python front-end to compile (there's a build() command) or directly call make. Once Peano 4 has written the makefile and the C++ code, the code is stand-alone, i.e. doesn't need Python anymore. So a plain make does the job.

peano4_project.generate()

generate all code ...
some artefacts have already been added to repository ... assume this is intentional (by higher abstraction layer, e.g.)
generated gprdr-main.cpp
write ./Makefile
write ././AbstractGPRDR.h (generated from template /home/tobias/git/Peano/python/exahype2/solvers/fv/GenericRiemannFixedTimeStepSizeAbstract.template.h)
write ././AbstractGPRDR.cpp (generated from template /home/tobias/git/Peano/python/exahype2/solvers/fv/GenericRiemannFixedTimeStepSizeAbstract.template.cpp)
write ./observers/SolverRepository.h (generated from template /home/tobias/git/Peano/python/exahype2/ProjectSolverRepository.template.h)
write ./observers/SolverRepository.cpp (generated from template /home/tobias/git/Peano/python/exahype2/ProjectSolverRepository.template.cpp)
write ./celldata/GPRDRQ.h (generated from template /home/tobias/git/Peano/python/peano4/datamodel/PatchToDoubleArray.template.h)
write ./celldata/GPRDRQ.cpp (generated from template /home/tobias/git/Peano/python/peano4/datamodel/PatchToDoubleArray.template.cpp)
write ./facedata/GPRDRQ.h (generated from template /home/tobias/git/Peano/python/peano4/datamodel/PatchToDoubleArray.template.h)
write ./facedata/GPRDRQ.cpp (generated from template /home/tobias/git/Peano/python/peano4/datamodel/PatchToDoubleArray.template.cpp)
write ./facedata/GPRDRQNew.h (generated from template /home/tobias/git/Peano/python/peano4/datamodel/PatchToDoubleArray.template.h)
write ./facedata/GPRDRQNew.cpp (generated from template /home/tobias/git/Peano/python/peano4/datamodel/PatchToDoubleArray.template.cpp)
write ./observers/DataRepository.h
write ./observers/DataRepository.cpp
write ././observers/CreateGrid2exahype2_solvers_fv_FV0.h
write ././observers/CreateGrid2exahype2_solvers_fv_FV0.cpp
write ././observers/CreateGrid2exahype2_solvers_fv_FV1.h
write ././observers/CreateGrid2exahype2_solvers_fv_FV1.cpp
write ././observers/CreateGrid2exahype2_grid_SetLabels2.h
write ././observers/CreateGrid2exahype2_grid_SetLabels2.cpp
write ././observers/CreateGrid.h
write ././observers/CreateGrid.cpp
write ././observers/InitGrid2exahype2_solvers_fv_FV0.h
write ././observers/InitGrid2exahype2_solvers_fv_FV0.cpp
write ././observers/InitGrid2peano4_toolbox_blockstructured_ProjectPatchOntoFaces1.h
write ././observers/InitGrid2peano4_toolbox_blockstructured_ProjectPatchOntoFaces1.cpp
write ././observers/InitGrid2peano4_toolbox_blockstructured_BackupPatchOverlap2.h
write ././observers/InitGrid2peano4_toolbox_blockstructured_BackupPatchOverlap2.cpp
write ././observers/InitGrid2exahype2_grid_SetLabels3.h
write ././observers/InitGrid2exahype2_grid_SetLabels3.cpp
write ././observers/InitGrid.h
write ././observers/InitGrid.cpp
write ././observers/CreateGridButPostponeRefinement2exahype2_solvers_fv_FV0.h
write ././observers/CreateGridButPostponeRefinement2exahype2_solvers_fv_FV0.cpp
write ././observers/CreateGridButPostponeRefinement2exahype2_grid_SetLabels1.h
write ././observers/CreateGridButPostponeRefinement2exahype2_grid_SetLabels1.cpp
write ././observers/CreateGridButPostponeRefinement.h
write ././observers/CreateGridButPostponeRefinement.cpp
write ././observers/PlotSolution2peano4_toolbox_blockstructured_PlotPatchesInPeanoBlockFormat0.h
write ././observers/PlotSolution2peano4_toolbox_blockstructured_PlotPatchesInPeanoBlockFormat0.cpp
write ././observers/PlotSolution2exahype2_grid_SetLabels1.h
write ././observers/PlotSolution2exahype2_grid_SetLabels1.cpp
write ././observers/PlotSolution.h
write ././observers/PlotSolution.cpp
write ././observers/TimeStep2exahype2_solvers_fv_FV0.h
write ././observers/TimeStep2exahype2_solvers_fv_FV0.cpp
write ././observers/TimeStep2exahype2_solvers_fv_FV1.h
write ././observers/TimeStep2exahype2_solvers_fv_FV1.cpp
write ././observers/TimeStep2peano4_toolbox_blockstructured_ReconstructPatchAndApplyFunctor2.h
write ././observers/TimeStep2peano4_toolbox_blockstructured_ReconstructPatchAndApplyFunctor2.cpp
write ././observers/TimeStep2peano4_toolbox_blockstructured_ProjectPatchOntoFaces3.h
write ././observers/TimeStep2peano4_toolbox_blockstructured_ProjectPatchOntoFaces3.cpp
write ././observers/TimeStep2exahype2_solvers_fv_FV4.h
write ././observers/TimeStep2exahype2_solvers_fv_FV4.cpp
write ././observers/TimeStep2peano4_toolbox_blockstructured_BackupPatchOverlap5.h
write ././observers/TimeStep2peano4_toolbox_blockstructured_BackupPatchOverlap5.cpp
write ././observers/TimeStep2exahype2_grid_SetLabels6.h
write ././observers/TimeStep2exahype2_grid_SetLabels6.cpp
write ././observers/TimeStep.h
write ././observers/TimeStep.cpp
write ./observers/StepRepository.h
write ./observers/StepRepository.cpp
write ././gprdr-main.h
write ././gprdr-main.cpp
write ./Constants.h
generation complete

# parallel_builds = 8
# peano4_project.build( make_clean_first = True, number_of_parallel_builds = parallel_builds )
!make -j4

g++ -std=c++0x -g -O2 -DPeanoDebug=2  -g -O2 -fopenmp -DDimensions=2 -I. -I../../../src -c -o AbstractGPRDR.o AbstractGPRDR.cpp
g++ -std=c++0x -g -O2 -DPeanoDebug=2  -g -O2 -fopenmp -DDimensions=2 -I. -I../../../src -c -o observers/SolverRepository.o observers/SolverRepository.cpp
^C
make: *** [Makefile:28: observers/SolverRepository.o] Interrupt

Run the code

As the code is a stand-alone executable (once I generated the stuff, there's no need for the Python front-end anywmore, you can take the code and run it on a supercomputer, e.g.), we can just type in run. There's also a Python wrap-around in Peano 4, but I usually prefer the command line version. The one below should work, but very

!./peano4

/bin/bash: ./peano4: No such file or directory

Visualisation

The new version of Peano/ExaHyPE writes out so-called patch files. This is a bespoke file format. There multiple ways how to manipulate/read it. I prefer to do everything within Paraview via Paraview. For this, I ensure that the PYTHONPATH points to Peano's Python directory, start up Paraview and then open the Python terminal. From here:

import peano4.api.visualisation
data = peano4.api.visualisation.Visualiser( "solution-GPRDR.peano-patch-file" )
data.append_filter(peano4.api.visualisation.ExtractFineGridFilter())
data.display()

From here, you can either use

data.select_dataset(any number)

to step through the time steps or you call

data.write_vtu_time_series()

which gives you a vtu file with all the snapshots which you can load in Paraview. The conversion itself can last for ages if you run it within Paraview. I do recommend that you use the Python environment

pvpython

to issue the conversion therein. With the pvpython environment, it is also possible to convert the data in the background while you open a separate Paraview instance to study the data already.

How to modify the actual code

This is ExaHyPE 2, but on the user side it is almost 1:1 the same interface as the original ExaHyPE. All the knowledge can be found in the files GPRDR.h and GPRDR.cpp. These files have been generated by the Python notebook, as you have named the solver GPRPR above.