 |
Peano
|
|
Peano
|
Hardcoded domain decomposition strategy. More...
#include <Hardcoded.h>
Data Structures | |
| struct | Split |
Public Member Functions | |
| Hardcoded (std::initializer_list< int > splittingTrees, std::initializer_list< int > numberOfCells, std::initializer_list< int > destinationRanks, std::initializer_list< peano4::SplitInstruction::Mode > modes) | |
| Construct hard-coded load balancing strategy. | |
| Hardcoded (std::initializer_list< int > splittingTrees, std::initializer_list< int > numberOfCells, std::initializer_list< int > destinationRanks, peano4::SplitInstruction::Mode mode=peano4::SplitInstruction::Mode::BottomUp) | |
| Shortcut around other constructor. | |
| virtual void | finishStep () override |
| Finish the current sweep. | |
| virtual void | finishSimulation () override |
 Public Member Functions inherited from toolbox::loadbalancing::AbstractLoadBalancing | |
| AbstractLoadBalancing (Configuration *configuration, CostMetrics *costMetrics) | |
| Constructor. | |
| virtual | ~AbstractLoadBalancing () |
| Destructor. | |
| virtual std::string | toString () const |
| Create a string representation of load balancing object. | |
| virtual void | enable (bool) |
| Switch on/off. | |
| virtual bool | hasSplitRecently () const |
| Has mesh split recently. | |
| bool | isEnabled (bool globally) const |
| Is load balancing enabled. | |
| virtual int | getGlobalNumberOfTrees () const |
| Delegate to stats. | |
| virtual bool | hasStagnated () const |
| A load balancing can either be stagnating or be switched off for this predicate to hold. | |
| void | setConfigurationAndMetricsNullWithoutDelete () |
| This is only used when you concatenate balancing rules and you want to disable any deletion. | |
Private Member Functions | |
| void | keepAndValidateLocalSplits () |
| Drop the splits of other ranks and validate the remaining ones. | |
| void | issueSplits () |
| Hand out the splits that are due. | |
Private Attributes | |
| std::list< Split > | _splits |
| bool | _hasKeptAndValidatedLocalSplits |
Static Private Attributes | |
| static tarch::logging::Log | _log |
Additional Inherited Members | |
| Static Public Attributes inherited from toolbox::loadbalancing::AbstractLoadBalancing | |
| static constexpr int | NoHeaviestTreeAvailable = -1 |
| Is used by tree identification and either indicates that there are no trees at all or means that the heaviest tree is on the blacklist. | |
| static constexpr int | NumberOfMeshSweepsAfterSplitToMakeMeshConsistentAgain = 3 |
| Number of mesh sweeps a split needs until the mesh is consistent again. | |
| Protected Member Functions inherited from toolbox::loadbalancing::AbstractLoadBalancing | |
| bool | fitsIntoMemory (State state) const |
| Ensure enough memory is left-over. | |
| bool | isInterRankBalancingBad () const |
| Is the balancing between the ranks ok. | |
| bool | isIntraRankBalancingBad () const |
| Is the balancing on the rank ok. | |
| bool | areRanksUnemployed () const |
| double | getWeightOfHeaviestLocalSpacetree () const |
| int | getIdOfHeaviestLocalSpacetree () const |
| Determines the maximum spacetree size a tree should have in the optimal case. | |
| int | getIdOfHeaviestLocalSpacetree (double tolerance) const |
| Similar to getIdOfHeaviestLocalSpacetree() but you might get one of the trees back that is close to the heaviest one up to tolerance. | |
| Protected Attributes inherited from toolbox::loadbalancing::AbstractLoadBalancing | |
| Blacklist | _blacklist |
| Statistics | _statistics |
| Configuration * | _configuration |
| CostMetrics * | _costMetrics |
| State | _state |
| Ensure that you invoke. | |
| Static Protected Attributes inherited from toolbox::loadbalancing::AbstractLoadBalancing | |
| static tarch::logging::Log | _log |
Hardcoded domain decomposition strategy.
Hardcoded load balancing is useful if you know exactly which domain decomposition you want to build up, or if you have to debug a particular (deterministic) setup. It also comes in handy whenever you can transform your dynamic load balancing challenge into an offline problem. The load balancing's Python API contains all the tools to derive good balancing strategies for offline problems (see Load balancing API).
The strategy holds an enumerated list of splits. The \( k \) th split reads
Tree splittingTrees[k] hands numberOfCells[k] fine grid cells over to a new tree on rank destinationRanks[k], and it uses modes[k] to select these cells.
The strategy does not say when a split happens. It hands out the splits in the order given, and it hands out the \( k \) th split as soon as
If one of these conditions does not hold, the \( k \) th split and all subsequent splits wait for the next mesh sweep. Several splits can go out in the same mesh sweep. Cells that the splitting tree has already promised to other trees in this sweep do not count towards the second condition.
The picture is generated from Hardcoded.dot via
Peano inverts its traversal in every other mesh sweep, i.e. a split which goes out in an inverted sweep picks its cells from the other end of the space-filling curve. We do not care about this inversion. The new tree still gets the requested number of cells.
As the splits wait until the splitting tree is large enough, they usually fire once the mesh construction has reached the mesh level on which the cells are to be found. For adaptive meshes, a split might hence fire before the mesh construction has terminated.
peano4::SplitInstruction::Mode::AggressiveTopDown hands over whole refined cells including all their children, while peano4::SplitInstruction::Mode::BottomUp selects cells on the finest mesh level. Peano applies the mode of the first split that a tree triggers in a mesh sweep to all splits of this tree in this sweep. The strategy therefore never lets one tree issue two splits with different modes in the same mesh sweep. A split with a different mode waits for the next mesh sweep.
Split a regular mesh of \( 27 \times 27 \) cells into four trees on one rank:
Tree 0 holds 9 and then 81 cells while the mesh is built up, which is not enough for the first split. Once tree 0 holds all 729 cells, all three splits go out in the same mesh sweep: Tree 0 has 729, 547 and 365 cells left over, respectively, before it hands out the next 182 cells.
If you change the mode of the last split into BottomUp
then the first two splits go out together, while the third split is not allowed to go out in the same sweep as the top-down splits. It goes out one mesh sweep later. As it uses BottomUp, it hands over exactly 182 cells, while the two top-down splits hand over whole refined cells and hence end up with slightly more or fewer cells.
Splits can also continue to split trees that earlier splits have created:
Here, the second split waits until tree 1 exists and holds more than 182 cells.
All ranks receive the same configuration. Each rank keeps only the splits whose splitting tree lives on this rank and hands them out in the order given. Splits of trees on other ranks hence never block a rank, and the ranks do not have to synchronise.
To tell which rank hosts a tree, we rely on Peano's tree numbering, which is round robin: peano4::parallel::Node::getId() gives the nth tree on rank r the number n*R+r for R ranks, and peano4::parallel::Node::getRank() maps tree number t onto rank tR. A split instruction cannot say which number the tree that it creates will get, as Peano picks this number. We therefore always assume that the trees are assigned to the ranks round robin, i.e. that the tree which the configuration calls t is created on rank tR. Only then does a later split of tree t in the configuration split the right tree. The Python API (create_constructor_initialisation_string() in toolbox.loadbalancing.api.output.Hardcoded) checks this assumption, as only the offline plan knows which tree each split is meant to create.
The idea behind the hardcoded strategy is that users pipe in a particular configuration tied to their specific choice of ranks. If a hardcoded setup does not use all ranks, there's nothing we can do here. Indeed, there might be cases where this makes sense, if the hardcoded strategy is used in combination with another load balancing scheme. If the strategy is used with fewer ranks than the configuration uses, i.e. if the hardcoded strategy wants to deploy to rank 20 and there are only 10 ranks, then we write an error and continue with a mock up. See Split::validateAndAmend() for details.
I originally thought it would be nice to have support for hard-coded load-balancing being encoded in a file. However, this contradicts Peano's philosophy that everything should be in the Python script. Therefore, this strategy does not support file I/O but the load balancing's Python interface does.
There's also a hardcoded orchestration strategy in tarch::multicore::orchestration::Hardcoded. This is a totally different notion of being hardcoded: There's the decision-making is hardcoded whereas here the actual decisions are. In this sense, the present name might be chosen inappropriately.
Definition at line 188 of file Hardcoded.h.
| toolbox::loadbalancing::strategies::Hardcoded::Hardcoded | ( | std::initializer_list< int > | splittingTrees, |
| std::initializer_list< int > | numberOfCells, | ||
| std::initializer_list< int > | destinationRanks, | ||
| std::initializer_list< peano4::SplitInstruction::Mode > | modes | ||
| ) |
Construct hard-coded load balancing strategy.
The splits are passed as four initialiser lists. Entry \( k \) of each list describes the \( k \) th split. So if you pass
then tree 2 will split three times. It will deploy 11, 12 and 13 cells to the ranks 22, 23 and 24, respectively. See the class documentation for when the splits happen.
We do not work with initialiser lists internally. Instead, the constructor takes the list and transforms it into a sequence of Split objects which are then held internally.
I originally wanted the constructor to dumb some info on what type of load balancing exactly had been constructed. However, some codes use the load balancing object as static variable, so it might come up before the logging infrastructure is properly configured. This means that the logInfo will crash. So better stay silent and avoid logInfo() calls here.
At this point, I may not yet call validateAndAmend() or filter out the splits of other ranks, as this requires MPI to be properly initialised. If the hardcoded strategy is used as kind of a global variable - a typical application case - then the initialisation consequently breaks down. Both steps therefore happen in the first finishStep().
| toolbox::loadbalancing::strategies::Hardcoded::Hardcoded | ( | std::initializer_list< int > | splittingTrees, |
| std::initializer_list< int > | numberOfCells, | ||
| std::initializer_list< int > | destinationRanks, | ||
| peano4::SplitInstruction::Mode | mode = peano4::SplitInstruction::Mode::BottomUp |
||
| ) |
Shortcut around other constructor.
In this variant, all split modes are of the same type and there's a default. See the other constructor for details.
|
overridevirtual |
Reimplemented from toolbox::loadbalancing::AbstractLoadBalancing.
|
overridevirtual |
Finish the current sweep.
This routine hands out the splits that are due according to the rules in the class documentation. Upon the first call, it also drops all splits whose splitting tree is hosted by another rank, and it validates the remaining ones.
If all the splits have been issued, the strategy disables itself. Usually, it is up to the user to disable a load balancing scheme if they want this to happen. For the hard-coded variant, this makes no sense: If there are no split instructions left over, we can as well just disable the strategy straightaway. This helps to speed up other things potentially, where a decision depends on the load balancing being deactivated already.
Standard step of any load balancing: update the global view of the statistics, and then also the blacklist. Eventually, we dump the statistics to the terminal.
Implements toolbox::loadbalancing::AbstractLoadBalancing.
|
private |
Hand out the splits that are due.
Runs through the splits in the order given and stops at the first split which is not due yet. See the class documentation for the rules.
|
private |
Drop the splits of other ranks and validate the remaining ones.
Is called by the first finishStep(). See the pitfall discussion in the constructor.
|
private |
Definition at line 334 of file Hardcoded.h.
|
staticprivate |
Definition at line 281 of file Hardcoded.h.
|
private |
Definition at line 332 of file Hardcoded.h.
1.9.8