 |
Peano
|
|
Peano
|
I recommend atm to use gpu10, gpu11 or gpu12, i.e. the A100 machines. By default, I usually work with NVC++ on this cluster, but the OneAPI software stack should work, too.
I use the setup
on the A100 nodes.
We had some issues (compiler crashes) with NVC++ 2022.5. Those are filed with NVIDIA. For the time being, please revert to another compiler generation. Version 2022.3 works for example.
Some nodes (e.g. gpu3) have Titan XP which doesn't support OpenMP offloading; only GPUs with compute capability >=70 can offload with OpenMP. You can check the GPU model with nvidia-smi -L.
Note that there is a bug in the NVIDIA software stack at the moment. Many runs will crash with a complaint about a lack of visible devices. Ensure hence that you set the following environment before you start your experiments:
I used this to configure Peano with Intel's TBB enabled
Codeplay's website has some useful information on compiler flags. Here's a short explaination of the above SYCL related flags:
-fsycl tells the compiler to link SYCL libraries and makes it aware of SYCL code. It seems that Clang behaves slightly different to icpx, where you need -lsycl in the linker flags.-fsycl-targets= allows the compiler to generate code for different devices, e.g., different vendors, host or device.-Xsycl-target-backend= is required if you want to select a specific device architecture. For example, I put --cuda-gpu-arch=sm_80 after it to compile for A100 with compute capability 8.0.It also needs at least 2023.0.0 version of oneAPI with Codeplay's nvidia GPU plugin which is not installed on NCC. Intel encourages the use of their own compiler, icpx. However, they use a lower precision by default in favour of performance, use -fp-model to change this behaviour.
I had linker errors about undefined referennces when building Peano with TBB. Just add the path to any relevant library if you experienced this.
When running the executable on the GPU, include SYCL_DEVICE_FILTER=gpu before the terminal command to explicitly tell SYCL to run the code on GPU. In the case of a runtime error, it is also useful to add SYCL_PI_TRACE=2 to check where the error was raised.
On SuperMUC-NG Phase 2, we can basically rely on the standard Intel toolchain:
Some remark on this setup:
This is a guideline for AMD Cloud machines that we use in hackathons, e.g. Login here works with standards ssh
where you have to pick an active port (7000 or 7010). We only need one module for the core GPU tests:
(and older version is module load amdclang/17.0-6.0.0) and then use the following configuration:
A good starting point for the GPU work is benchmarks/exahype2/ccz4/performance-testbed.
This configuration is meant to be a minimal build configuration when working with TBB locally. It is important to use an up-to-date c++ compiler such as icpx (e.g. g++-12 will not work with the c++-20 features used in Peano's tbb code).
First, the oneapi environment has to be set up:
or
Configuration:
With exahype2:
1.9.8