Intel's OneAPI ecosystem

Table of Contents

• Intel-specific compiler flags
• Intel toolchain (tool traces)
• Threading Building Blocks (TBB)
• OpenMP and OpenMP offloading
• SYCL and SYCL offloading
  • Intel GPUs
  • NVIDIA GPUs
  • AMD GPUs
• Intel MPI
• Troubleshooting

You will find examples how this toolchain is used on particular machines on the page Reference Configurations for some machines.

Intel-specific compiler flags

There are two particular flags that we find useful in Peano if Intel's software stack is used.

• Adding

  CXXFLAGS="... -DTBB_USE_THREADING_TOOLS -parallel-source-info=2"

  to all compiles yields more detailed trace data.
• Adding

  CXXFLAGS="... -DTBB_USE_ASSERT"

  helps to debug TBB-based applications.

At the moment, we add the following flags to the compile automatically when the Intel toolchain is activated (see below). Obviously, nothing stops users from adding those flags manually even if they prefer not to rely on Peano's support for the Intel toolchain.

Intel toolchain (tool traces)

Peano provides support for Intel's (oneAPI) toolchain through two routes: The ITAC interface and the Instrumentation and Tracing Technology (ITT). To activate either of these toolchains, you have to reconfigure with one of the two options below:

./configure ... --with-toolchain=itac
./configure ... --with-toolchain=itt

Broadly speaking, the toolchains alters the code in two ways:

• It enables ITAC or ITT, respectively.
• It switches the line logger to the ITAC or ITT logger. That is, trace data are not pipes onto the terminal anymore, but are passed into the ITAC or ITT API.
• It adds some Intel-specific compile flags such as the ones discussed above.

You might have to set some additional environment variables or configure with a bespoke path to use these tools. More information is curated on the pages for the loggers itself.

ITAC is there for legacy reasons. As Intel deprecates ITAC, it is recommended to use ITT (for VTune) and otherwise employ a tool such as Scalasca/Score-P for MPI. For these tools, Peano offers dedicated loggers, too.

The enumeration shows that the name Intel toolchain is misleading. We are actually not tailoring the build to the Intel toolchain, but we tailor the setup to the Intel analysis tools. Most production runs will not use the Intel toolchain, but rather set all Intel-specific flags for compiler and linker manually.

Despite having switched to the Intel loggers, you will not get meaningful trace information if you build in release mode. You have to switch to the trace mode (or assertions or debug) - compare general remarks on Peano's build modi - to get traces or annotation info. Once you try to trace your code with the Intel tool, the size of the traces quickly becomes unmanageable or the performance might go down. However, you can combine log filters to narrow down the number of classes and namespaces that actually trace.

Threading Building Blocks (TBB)

Peano relies on advanced TBB features, and not all of them are found in normal release (as of January 2026): some of the features we need and have developed are so far only in TBB preview versions. Consequently, only loading a TBB module and adding

LIBS = "-ltbb"

might not work with Peano. Details on the TBB extensions are compiled in a separate document on TBB extensions.

OpenMP and OpenMP offloading

CC=icx CXX=icpx ./configure --with-multithreading=omp --with-gpu=omp CFLAGS="-O3 -W -Wall -Wno-attributes -fiopenmp -fopenmp-targets=spir64" CXXFLAGS="-std=c++20 -O3 -W -Wall -Wno-attributes -fiopenmp -fopenmp-targets=spir64" LDFLAGS="-march=native -mtune=native -fiopenmp -fopenmp-targets=spir64"

Intel suggests to use -fiopenmp for performance and features: https://www.intel.com/content/www/us/en/developer/articles/guide/porting-guide-for-icc-users-to-dpcpp-or-icx.html

SYCL and SYCL offloading

Peano can use SYCL as both multithreading backend and as GPU offloading pathway. If SYCL is only used for the accelerator, you can combine with TBB and C++ multithreading besides with SYCL itself.

We consider using SYCL as multithreading back-end to be experimental at best at the moment. For production runs, we rather recommend to combine SYCL for the GPUs with TBB on the host.

SYCL is directly supported via icpx. There are only a few things to do:

• Translate your code with --with-multithreading=sycl and/or --with-gpu=sycl, depending on for what you want to use SYCL.
• Add CXXFLAGS="... -fsycl" to your compiler flags. This way, all the headers of SYCL are known to icpx.
• Add LDFLAGS="... -fsycl" to your linker flags. This way, the linker will automatically add all SYCL libraries. Furthermore, it will know when it builds the final applications that it should embed both the device kernels and the CPU kernels into the executable. If you use LIBS="... -lsycl", the linking will succeed, but the very first time you invoke a SYCL kernel, you'll get a message similar to

  terminate called after throwing an instance of 'sycl::_V1::runtime_error'
    what():  No kernel named _ZTSZZN7toolbox15blockstructured64interpolateCellDataAssociatedToVolumesIntoOverlappingCell_linearEiiiiiPKdPdN6peano45utils15LoopParallelismEENKUlRT_E_clIN4sycl3_V17handlerEEEDaS8_EUlNSC_2idILi3EEEE_ was found -46 (PI_ERROR_INVALID_KERNEL_NAME)
  Aborted (core dumped)

• Ensure CXX=icpx points to the icpx compiler.
• Ensure CC=icx points to the icpx compiler if you work with the autotools. This is not intuitive and actually quite a hack, but you need it to work: configure will test if you C compiler is compatible with your linker flags. However, the -fsycl command is unknown to the default C compiler (usually gcc) and your configure will fail miserably. By letting the C compiler variable point to the Intel oneAPI C compiler, you ensure that the sanity check within the configuration phase passes.
• Reset the MPI compiler (see below; if you want to use MPI).

Here are some example scripts:

Intel GPUs

CC=icx CXX=icpx ./configure --with-multithreading=sycl --with-gpu=sycl CFLAGS="-O3 -W -Wall -Wno-attributes -fsycl -fsycl-targets=spir64 -fsycl-device-code-split -fp-model=consistent" CXXFLAGS="-std=c++20 -O3 -W -Wall -Wno-attributes -fsycl -fsycl-targets=spir64 -fsycl-device-code-split -fp-model=consistent" LDFLAGS="-march=native -mtune=native -fsycl -fsycl-targets=spir64 -fsycl-device-code-split -fp-model=consistent"

NVIDIA GPUs

Her we use the Intel LLVM patch as provided through Codeplay:

CC=icx CXX=icpx ./configure --with-multithreading=sycl --with-gpu=sycl CFLAGS="-O3 -W -Wall -Wno-attributes -fsycl -fsycl-targets=nvptx64-nvidia-cuda -fsycl-device-code-split -fp-model=consistent -Xsycl-target-backend=nvptx64-nvidia-cuda --cuda-gpu-arch=sm_XX" CXXFLAGS="-std=c++20 -O3 -W -Wall -Wno-attributes -fsycl -fsycl-targets=nvptx64-nvidia-cuda -fsycl-device-code-split -fp-model=consistent -Xsycl-target-backend=nvptx64-nvidia-cuda --cuda-gpu-arch=sm_XX" LDFLAGS="-march=native -mtune=native -fsycl -fsycl-targets=nvptx64-nvidia-cuda -fsycl-device-code-split -fp-model=consistent -Xsycl-target-backend=nvptx64-nvidia-cuda --cuda-gpu-arch=sm_XX"

where XX has to be replaced by the actual compute capability of the device.

It is our understanding at the moment (January 2026) that this toolchain is not maintained anymore. You might want to switch to OpenSYCL instead.

AMD GPUs

Again with Intel toolchain as patched by Codeplay:

CC=icx CXX=icpx ./configure --with-multithreading=sycl --with-gpu=sycl CFLAGS="-O3 -W -Wall -Wno-attributes -fsycl -fsycl-targets=amdgcn-amd-amdsha -fsycl-device-code-split -fp-model=consistent -Xsycl-target-backend=amdgcn-amd-amdsha --offload-arch=gfxXXX" CXXFLAGS="-std=c++20 -O3 -W -Wall -Wno-attributes -fsycl -fsycl-targets=amdgcn-amd-amdsha -fsycl-device-code-split -fp-model=consistent -Xsycl-target-backend=amdgcn-amd-amdsha --offload-arch=gfxXXX" LDFLAGS="-march=native -mtune=native -fsycl -fsycl-targets=amdgcn-amd-amdsha -fsycl-device-code-split -fp-model=consistent -Xsycl-target-backend=amdgcn-amd-amdsha --offload-arch=gfxXXX"

where XXX has to be replaced by the actual compute capability of the device.

It is our understanding at the moment (January 2026) that this toolchain is not maintained anymore. You might want to switch to OpenSYCL instead.

Intel MPI

Intel's MPI wrapper is mpiicpc even though they now want you to use icpx instead of icc/icpc. To tell the MPI wrapper that you want to use icpx, you have to set some environment variables:

export I_MPI_CXX=icpx
./configure --with-mpi=mpiicpc

Troubleshooting

All the troubleshooting is covered by FAQs and Troubleshooting.