Reference configurations for Durham's hardware tesbed

Table of Contents

• Login nodes
• DINE
  • ScoreP
• MAD07 (Ice Lake)
• MAD08/MAD09 (Genoa)
• Grace Hopper

Durham University maintains a hardware testbed suite that can be used for testing code on new hardware.

Further documentation can be found at https://cosma.readthedocs.io/en/latest/cosma.html.

This page contains information on particular machines that we use heavily. The page is not supposed to provide general information about certain software stacks. For such information, please consult the toolchain discussions on the parent page Vendor Software Stacks and System- and Compiler-specific Settings.

Login nodes

The login nodes can be accessed at login8.cosma.dur.ac.uk. Further nodes can be accessed via these nodes. The following modules should be used for an Intel toolchain:

module purge
module load cosma
module load intel_comp/2025.2.1 compiler-rt tbb umf compiler mpi
module load intel_mpi
export FLAVOUR_NOCONFLICT=1
module load gnu_comp/13.1.0
module load gsl
export I_MPI_CXX=icpx
module load python

DINE

DINE (Durham Intelligent NIC Environment) is a small supercomputer attached to Cosma, which we use to prototype some of our algorithms. Access is through the Cosma loging nodes (login8.cosma.dur.ac.uk). As the login node and DINE feature the same generation of AMD chips, you can compile on the login node and ship the executable to DINE by submitting a slurm job to the bluefield1 partition. Should you choose to compile on DINE use the same modules as on the login node, which are:

module purge
module load cosma
module load intel_comp/2025.2.1 compiler-rt tbb umf compiler mpi
module load intel_mpi
export FLAVOUR_NOCONFLICT=1
module load gnu_comp/13.1.0
module load gsl
export I_MPI_CXX=icpx
module load python

There are a few things to consider whenever you use DINE:

• The default Python version usually is a little bit too old, so switching to a more recent Python package is reasonable.
• Don't use the -xHost flag in configure. The resulting code will crash on DINE. You can use

  -march=native

• Try not to dump data into the home directory, as this one is backed up and has a quota. DINE mounts the Cosma 5 data directory (/cosma5/data/do009/myaccount) which you should use to store data files.

ScoreP

ScoreP, Scalasca and Cube are installed for DINE. It supports gcc and this specific Intel toolchain. If you wish to use them load the modules:

module load scorep scalasca cube

ScoreP has to be installed anew for each toolchain, so it may not be installed for other toolchains.

MAD07 (Ice Lake)

mad07 is an IceLake server with Barlow Pass memory which is added to Cosma to facilitate benchmarking for the SKA project. It can be accessed with a slurm job to the cosma8-ska partition.

module purge
 
module unload python
module load python/3.10.12
 
module load intel_comp/2023.2.0 compiler mpi 
module load gnu_comp/13.1.0
 
./configure --enable-exahype --enable-blockstructured --enable-loadbalancing --enable-particles CC=icc CXX=icpx --with-multithreading=omp  CXXFLAGS="-std=c++20 -qopenmp -funroll-loops -Ofast -xHost" LIBS=-ltbb

Previous modules/installations provided a oneAPI module. As this seems to be deprecated by Intel, one now has to load intel_comp and then the corresponding compiler and mpi.

The chip is from the Ice Lake-SP family, i.e. the generation released summer 2021. Without any further settings, OpenMP delivers a concurrency of around 11,170. This is due to two processors each featuring 28 cores with hyperthreading. I don't use hyperthreading usually, i.e. set

export OMP_NUM_THREADS=56
export OMP_PROC_BIND=true

I thought they have 2x40 cores, but my benchmarks give me an OpenMP concurrency of 11,170, which suggests it has 112 cores in total. Does that mean we have 2x28 and hyperthreading is enabled?

MAD08/MAD09 (Genoa)

These two servers feature the Genoa chipset. They can be accessed by submitting a slurm job to the cosma8-shm3 partition.

module purge
 
module unload python
module load python/3.10.12
 
 
module load intel_comp/2023.2.0 compiler mpi 
module load gnu_comp/13.1.0
 
./configure --enable-exahype --enable-blockstructured --enable-loadbalancing --enable-particles CC=icc CXX=icpx --with-multithreading=omp  CXXFLAGS="-std=c++20 -qopenmp -funroll-loops -Ofast -xHost" LIBS=-ltbb

There are a few pitfalls with the AMD servers if you use the Intel toolchain. The Intel compiler does not recognise that the chip supports modern instruction sets. Notably, it is not aware of its AVX512 capabilities. Therefore, it produces machine code which does not use these features. You have to manually force it to produce the bytecode that you wanna have, by adding the compile flags

-O3 -fomit-frame-pointer -fstrict-aliasing -ffast-math -funroll-loops -axCOMMON-AVX512 -march=x86-64-v4 -mavx512vbmi 

ExaParty also requires you to pass

-qopt-zmm-usage=high

to vectorisesome particular compute-heavy tasks (gravity).

Grace Hopper

Grace Hopper nodes can be accessed directly from the login node using

ssh gn002

Code can be compiled here, or submitted as a slurm job to the gracehopper partition where it will run on a compute node.

The Grace Hopper build configuration is similar to any other build:

./configure CXX=nvc++ CC=nvc++ CXXFLAGS="-O4 --std=c++17 -mp=gpu -gpu=cc90,cuda12.3" LDFLAGS="-mp=gpu -gpu=cc90,cuda12.3" --with-multithreading=omp  --with-gpu=omp --enable-exahype --enable-blockstructured --enable-loadbalancing

It is important to use cc90 here plus a reasonably new CUDA version. Furthermore, most systems deliver the NC++ pipeline with a rather old GCC/STL version. It might be necessary to load a newer GCC. Depending on the SLURM configuration, people might have to request GPUs explicitly when they want a Grace Hopper node.