You will find examples how this toolchain is used on particular machines on the page Reference Configurations for some machines.

The NVIDIA toolchain requires us to make CXX and CC point to the C++ compiler, as both are passed the same arguments by configure, which are actually only understood by the C++ version.

NVIDIA analysis tools

Peano offers a dedicated NVIDIA logger. It pipes tracing information into a format that NSight et al can process, i.e. it provides you with domain-specific information on top of your profile. To enable NVIDIA's NVPTX logging, you have to configure with the

--with-toolchain=nvptx

toolchain.

OpenMP

NVIDIA's compiler does not support all of OpenMP. Therefore, its OpenMP implementation is rather picky when it comes to taskloops and other C++ features. Peano has internal workarounds for all of these items, i.e. the build should in principle succeed yet the code might be slightly slower than the Intel or native Clang counterpart.

To be compatible with NVIDIA, the respective tricky code parts all are embedded into proper ifdefs. The idea is that every file includes the compiler-specific settings:

// This file is part of the Peano project. For conditions of distribution and
// use, please see the copyright notice at www.peano-framework.org
#pragma once
 
#if (defined(__INTEL_COMPILER) || defined(__ICC)) && defined(__WIN32__)
  #include "tarch/compiler/WindowsIntelLegacy.h"
#elif defined(__GNUC__) && defined(__WIN32__)
  #include "tarch/compiler/WindowsGCC.h"
 
#elif (defined(__INTEL_COMPILER) || defined(__ICC)) && defined(__linux)
  #include "tarch/compiler/LinuxIntelLegacy.h"
#elif defined(SYCL_LANGUAGE_VERSION) && defined(__INTEL_LLVM_COMPILER) && defined(__linux)
  #include "tarch/compiler/LinuxIntelSYCL.h"
#elif defined(__INTEL_LLVM_COMPILER) && defined(__linux)
  #include "tarch/compiler/LinuxIntelICPX.h"
#elif defined(__AMDGPU__) && defined(__linux)
  #include "tarch/compiler/LinuxAMD.h"
#elif defined(__clang__) && defined(__linux)
  #include "tarch/compiler/LinuxClang.h"
// https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/#nvcc-identification-macro
#elif defined(__NVCOMPILER) /*|| defined(_NVHPC_CUDA) || defined(__NVCOMPILER_CUDA__) || defined(__CUDACC__) || defined(NVCC) || defined(__NVCC__)*/ && defined(__linux)
  #include "tarch/compiler/LinuxNVIDIA.h"
#elif defined(__GNUC__) && defined(__linux)
  #include "tarch/compiler/LinuxGCC.h"
 
#elif defined(__clang__) && (defined(__APPLE__) || defined(__apple_build_version__))
  #include "tarch/compiler/MacOSClang.h"
#elif defined(__GNUC__) && (defined(__APPLE__) || defined(__apple_build_version__))
  #include "tarch/compiler/MacOSGCC.h"
#else
  #error Unknown compiler target. Please adopt CompilerSpecificSettings.h
#endif

The header file reads out preprocessors and delegates to the right header file. Within the header file, we have a lot of defines. Below is a copy of the header file which should be used for NVIDIA on Linux systems.

// Default variant of CompilerSpecificSettings.h for NVIDIA on Linux.
// For a detailed description of the semantics of the settings, please
// consult LinuxIntel.h which is the most elaborate documentation.
 
//#define UseTestSpecificCompilerSettings
//#define CompilerCLANG
//#define CompilerICC
#define CompilerHasProcStat
#define CompilerHasUTSName
#define CompilerHasTimespec
#define CompilerHasSysinfo
//#define CompilerDefinesMPIMaxNameString
#if !defined(noMPISupportsSingleSidedCommunication) and !defined(MPISupportsSingleSidedCommunication)
#define MPISupportsSingleSidedCommunication
#endif
 
#if !defined(UseManualInlining) && !defined(noUseManualInlining)
#endif
 
#define LittleEndian
 
#if !defined(AlignmentOnHeap)
#define AlignmentOnHeap 16
#endif
 
#if PeanoDebug == 0
#undef assert
#define assert(...)
#endif
 
#define KeywordToAvoidDuplicateSymbolsForInlinedFunctions static //inline
 
#undef OpenMPTaskGroup

You see for example that we explicitly undefine OpenMPTaskGroup, as task groups is one of the things NVIDIA does not support at the moment. So the corresponding code parts employing task groups can read out this ifdef and then provide fitting alternatives.

OpenMP offloading to GPUs

The OpenMP offloading requires us to specify both the GPU OpenMP target and a corresponding compute capability:

CC=nvc++ CXX=nvc++ ./configure --with-multithreading=omp --with-gpu=omp CFLAGS="-O3 -W -Wall --diag_suppress=unrecognized_attribute -fopenmp -Munroll=c:100 -mp=gpu -gpu=ccXX" CXX="nvc++" CXXFLAGS="-std=c++20 -O3 -W -Wall --diag_suppress=unrecognized_attribute -fopenmp -Munroll=c:100 -mp=gpu -gpu=ccXX" LDFLAGS="-march=native -mtune=native -fopenmp -Munroll=c:100 -mp=gpu -gpu=ccXX"

where XX has to be replaced by the actual compute capability of the device.

C++

We have experimental support for std::par offloading. To enable it with NVIDIA, you have to specify

CC=nvc++ CXX=nvc++ CFLAGS="-O3 -W -Wall --diag_suppress=unrecognized_attribute -fopenmp -Munroll=c:100 -mp=gpu -gpu=ccXX" CC="nvc" CXXFLAGS="-std=c++20 -O3 -W -Wall --diag_suppress=unrecognized_attribute -fopenmp -Munroll=c:100 -mp=gpu -stdpar=gpu -gpu=ccXX" LDFLAGS="-march=native -mtune=native -fopenmp -Munroll=c:100 -mp=gpu -gpu=ccXX"

where XX has to be replaced by the actual compute capability of the device. Note that the above setup still uses OpenMP on the host for the multithreading.