 |
Peano
|
|
Peano
|
Functions | |
| add_dynamic_mesh_refinement_and_particle_resorting (steps, alter_position, species_sets, sort_with_lifts_and_drops, verbose) | |
| Add action sets to step (aka mesh traversal) to adopt mesh to particle configuration. | |
| exaparty.api.graphcompiler.AMRLogic.add_dynamic_mesh_refinement_and_particle_resorting | ( | steps, | |
| alter_position, | |||
| species_sets, | |||
| sort_with_lifts_and_drops, | |||
| verbose | |||
| ) |
Add action sets to step (aka mesh traversal) to adopt mesh to particle configuration.
The AMR logic is based upon the observation that Peano's toolbox always holds the particles in the right place and hence allows us to realise all particle-particle interactions. If we don't get the mesh right, we might not be optimal in terms of cost. But we are never wrong!
In principle, SPH's AMR logic is pretty simple in the context of a the dual tree with its lifts and drops. The illustration below shows this trivial workflow on the left side:
The image is the time axis and the bullets are the individual mesh traversals. On the left, we see the user input which eventually triggers a cascade of actions on the bookkeeping side (right). The user alters the particle position and hence might invalidate the particle-grid association. We have to re-sort. Some sorting can happen right in the same mesh traversal in which we alter the particles' properties in the touchCellLastTime or touchVertexLastTime actions. The green colour denotes stuff that's happening in the last data access - which might also introduce some MPI sends.
In the subsequent mesh traversal, we have to receive MPI data, move particles down in their tree and associate that to the correct vertex. This happens prior to any operation (denoted blue). Now that particles are in the right place, we can update their parallel state.
A straightforward implementation now might evaluate the AMR criterion, as all particles are in place and we can do so. Once we have collected all AMR info, we can distribute it globally and hand it out in the subsequent mesh traversal in which we consequently trigger refinements. Peano does not realise refinements immediately, but waits for the next mesh traversal to do so.
When we have done the refinement, we have to resort. Again, the resort has to spread over two mesh traversals, as the first half of the sorting might trigger MPI exchanges and multilevel traversals.
In this native data flow scheme, the user action affects the behaviour of five follow-up mesh traversals.
Alternative to add_dynamic_mesh_refinement_and_particle_resorting_aggressive_mesh_adoption() which is compatible with memory pool solutions, as long as you insert a dummy mesh sweep after each routine which changes the particle-mesh topology (or you ensure that the subsequent sweep does not evaluate particle-particle interactions).
The big difference to add_dynamic_mesh_refinement_and_particle_resorting_aggressive_mesh_adoption() is that we do analyse the AMR after we've resorted the particles. However, we do not realise the AMR up until we have to due to physics. So we make the AMR lag behind even further.
The big differences hence arise in the section where we do the association, and then we introduce the additional gather routines. As we do not update the association four times but only twice, we can also update the state only once instead of two times.
The optimised workflow accepts that we we are fine to deal with a suboptimal mesh layout, as long as we minimise the number of particle movements. This is a different cost objective than in the description above. We realise that we will always need a resort when we have updated a mesh.
The idea is that we know that a user interaction forces us to update the particle-mesh association, while AMR does not: We have to lift particles from areas of the tree which are coarsened, but otherwise AMR does not invalidate a valid mesh association. Particles might reside on too coarse levels, but still all the kernel evaluations remain correct. The lift is the only thing that causes issues, as it might mean that particles literally travel.
To avoid that we have to do two sorts per particle position update, we evaluate the AMR criterion two grid sweeps ahead of the actual particle position update. This way, the AMR realisation and the actual position update coincide in the same mesh sweep. No need for an additional sorting step.
The disadvantage of this scheme is that the mesh always lags behind. The AMR just is about the realise when the particles move again. So it might have been optimal before, but now it is not good anymore. In return, we however get a scheme which does, amortised, need only one resorting step per mesh update plus AMR update.
The aggressive AMR is not compatible with particles that gather data in memory pools.
This implementation puts emphasis that the mesh is always as tighly adopted to the particles as possible. If necessary, this might mean that we resort the particles multiple times per grid sweep: If they change their position, we resort. But once we have resorted, we might want to change the mesh, which in turn means we have to resort again, as we might have new cells or cells might have been deleted.
As so often, the order of the action sets is tremendously important, as the individual action sets interact with each other. When we discuss the order here, we may assume that all the function steps aka action sets are already fully implemented and in the right order.
The trigger the AMR can be integrated whereever we want within the pipeline. It simply deploys events, i.e. it is order-agnostic.
For the remaining action sets, we observe the following properties:
So we want to have the following lifecycle from a grid's point of view:
| steps | [peano4.api.solversteps.Step] |
| alter_position | [{True,False}] |
| species_sets | [Species] |
| verbose | {True, False} Controls how much debug information is dropped. |
Definition at line 15 of file AMRLogic.py.
1.9.8