Skip to content

The HPE Cray Programming Environment

[back: The Lmod module system]

On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated as Cray PE). The environment provides several tools, including compilers, communication libraries, optimised math libraries and various other libraries, analyzers and debuggers.

The Cray PE is made available through environment modules that allow to select particular versions of tools and to configure the environment in a flexible way.

HPE Cray PE components

Cray compiler environments

Cray PE supports multiple compilers, Cray and third party compilers as well: AOCC, Cray, Intel, AMD ROCm, NVIDIA, GNU. The Cray and GNU compilers have a special status and can be expected on any Cray system while the others are offered depending on the architecture of the system and wishes of the customer.

Users can access the compilers loading a programming environment module (see below).

Compilers in the Cray Programming Environment are almost always used through the Cray compiler wrappers. In fact, until recently, some compilers could not be used without the wrappers as they failed to even detect their own include files and some components. The behaviour of the wrapper will depend on the other modules that are loaded in the environment. E.g., there is no separate wrapper for MPI as the MPI header files and libraries are automatically activated through the regular wrappers as soon as a Cray MPI module is loaded.

The commands to invoke compiler wrappers are ftn (Fortran), cc (C), CC (C++). They wrap automatically to the right compilers based on the programming environment module that is loaded and the compiler module.

The online help can be accessed with the -help option. E.g.: cc -help, CC -help. One of the most frequently used options of the compiler wrappers is -craype-verbose: 

 ftn -help | grep verbose -A 1 
   -craype-verbose              Print the command which is forwarded 
                                to compiler invocation
More information is available with the info or man commands. E.g.: both info cc or man cc will display the man page of the C compiler wrapper. HPE now also has a documentation web site for the Programming Environment.

The compiler wrappers call the correct compiler in the currently loaded programming environment, with appropriate options to build and link applications with relevant libraries, as required by the modules loaded. Besides a number of generic options, they also will pass all other options to the underlying compile, so you can still use the regular compiler options also.

The compiler wrappers should replace direct calls to compiler drivers in Makefiles and build scripts to ensure that the proper libraries are used.

Note

For system libraries, only dynamic linking is supported by compiler wrappers on the Cray EX system

Cray Compiling Environment (CCE)

The Cray Compiling Environment is set by the module PrgEnv-cray, which is loaded by default at login. The compiler itself is contained in the cce module.

CCE consists of Cray compilers performing code analysis during compilation to generate highly optimized code. Supported languages include Fortran, C and C++, UPC (Unified Parallel C) and HIP. The Cray C/C++ compiler is basically the Clang/LLVM compiler with a back-end configured by HPE Cray that also adds some Cray-specific options while the Cray Fortran compiler uses an HPE-Cray specific front-end with a back-end based on LLVM. The Fortran compiler supports most of Fortran 2018 and tends to be considerably stricter than the GNU or Intel Fortran compilers.

The CCE compilers also support OpenMP offload to NVIDIA and AMD GPUs. The Fortran compiler (but not the C/C++ compiler) also supports OpenACC for offloading to GPU and is being updated to newer versions of this standard (but still focusing on 2.6 and 2.7 rather than 3.0 and later).

The classic pre-LLVM Cray C/C++ compiler is not available on Cray EX systems.

Compiler-specific manpages can be accessed on the system with man crayftn, man craycc or man crayCC.

More details are given in the documentation web site for the Programming Environment. For older versions, there are the Cray Fortran Reference Manual and the Cray Compiling Environment Release that unfortunately are hidden deep in the new support pages of HPE. The Clang Compiler User’s Manual is another source of information for the Cray C and C++ Clang compilers. The major version of the cce module and compiler corresponds to the Clang major versions.

For more information about compiler pragmas and directives, see man intro_directives on the system or search in the documentation web site for the Programming Environment.

Third-Party Compilers

GNU

The GNU C/C++ and Fortran compilers are probably the best supported third party compiler in the Cray PE.

In older versions of the PE, HPE Cray delivered their own builds of the GNU compiler collection, and executables had the regular names gcc, g++ and gfortran. Compiler-specific manpages were accessed on the system with man gcc, man g++ or man gfortran.

Nowadays HPE Cray uses the development packages from SUSE and these compilers have the major version in their name, e.g., for 25.03 this includes gcc-14, g++-14 and gfortran-14, and these commands come with their respective documentation.

OpenMP offload for AMD GPUs is not enabled in any of those GNU compilers.

More details are provided by the GCC online documentation.

AOCC

There AOCC compilers, the AMD Optimizing C/C++ Compiler and matching fortran compilers, AMD's compiler offering for CPU-only systems, have a matching programming environment module and are a full citizen of the Cray PE.

Compiler-specific documentation is available in the AOCC User Guide.

Cray provides a bundled package of support libraries to install into the PE environment to enable AOCC, and Cray PE utilities such as debuggers and performance tools work with AOCC.

AMD ROCm compilers

The AMD ROCm compilers are supported on systems with AMD CPUs. AMD contributes a lot of code back to the Clang/LLVM community, so these compilers are close to the standard distributions. The major version of Clang/LLVM used in ROCm chang change even if the major version of ROCm does not. E.g., ROCm 6.0 uses Clang 17 while ROCm 6.2 already uses Clang 18.

Intel (not on LUMI)

The Cray PE also provides a programming environment module to enable the Intel® oneAPI compiler and tools.

The documentation is available in the Intel® oneAPI Programming Guide

Cray provides a bundled package of support libraries to install into the Cray PE to enable the Intel compiler, allowing utilities such as debuggers and performance tools to work with it.

NVIDIA HPC toolkit

The NVIDIA HPC Toolkit compilers (formerly PGI) are supported on systems with NVIDIA GPUs.

Cray Scientific and Math Library

  • Manpages: intro_libsci, intro_fftw3

The Cray Scientific and Math Libraries (CSML, also known as LibSci) are a collection of numerical routines optimized for best performance on Cray systems.

These libraries satisfy dependencies for many commonly used applications on Cray systems for a wide variety of domains.

When the module for a CSML package (such as cray-libsci or cray-fftw) is loaded, all relevant headers and libraries for these packages are added to the compile and link lines of the cc, ftn, and CC compiler wrappers, so linking with them is completely transparent (to the extent that users wonder where the libraries are).

The CSML collection contains the following Scientific Libraries:

  • As part of the cray-libsci modules:
    • BLAS (Basic Linear Algebra Subroutines)
    • BLACS (Basic Linear Algebra Communication Subprograms)
    • CBLAS (Collection of wrappers providing a C interface to the Fortran BLAS library)
    • IRT (Iterative Refinement Toolkit)
    • LAPACK (Linear Algebra Routines)
    • LAPACKE (C interfaces to LAPACK Routines)
    • ScaLAPACK (Scalable LAPACK)
  • libsci_acc (a subset of routines from LibSci with GPU acceleration)
  • HDF5
  • NetCDF (Network Common Data Format)
  • FFTW3 (the Fastest Fourier Transforms in the West, release 3)

Cray Message Passing Toolkit

MPI is a widely used parallel programming model that establishes a practical, portable, efficient, and flexible standard for passing messages between ranks in parallel processes.

Cray MPI is derived from Argonne National Laboratory MPICH and implements the MPI-3.1 standard as documented by the MPI Forum in MPI: A Message Passing Interface Standard, Version 3.1. Version 9 of the cray-mpich modules will support the MPI 4 standard, with an implementation derived from MPICH 4.1.

Support for MPI varies depending on system hardware. To see which functions and environment variables the system supports, please have a look at the corresponding man pages with man intro_mpi on the system.

Note that though on LUMI only the OpenFabric Interface (OFI) based version of the library is supported.

DSMML

Distributed Symmetric Memory Management Library (DSMML) is a HPE Cray proprietary memory management library.

DSMML is a standalone memory management library for maintaining distributed shared symmetric memory heaps for top-level PGAS languages and libraries like Coarray Fortran, UPC, and OpenSHMEM.

DSMML allows user libraries to create multiple symmetric heaps and share information with other libraries.

Through DSMML, interoperability can be extracted between PGAS programming models.

Further details are available in the man page on the system with man intro_dsmml.

Other components

  • Cray Performance and Analysis Tools:

    Tools to analyze the performance and behavior of programs that are run on Cray systems, and a Performance API (PAPI).

  • Cray Debugging Support Tools:

    Debugging tools, including gdb4hpc and Valgrind4hpc.

Configuring the Cray PE through modules

Multiple releases of the HPE Cray PE can be installed simultaneously on the system and users can mix-and-match components. However, it is also possible to load only components for a specific release of the PE. Cray PE versions have version numbers of the form yy.mm, e.g., 24.03 for the version released in March 2024. However, each of the components have their own version number and it is not easy to see which version of a component came with which version(s) of the Cray PE. There used to be 10 releases each year, but this has been reduced to 3 more substantial releases, around March, July and November.

Below we only discuss those modules that are important when building software with EasyBuild. Debuggers, profilers, etc., are not included in the list.

The programming environment modules

The usual way to initialise the Cray PE is by loading one of the PrgEnv-* modules. This module will then load the appropriate compiler, compiler wrappers and other libraries. Some of the components that will be loaded are configured through the /etc/cray-pe.d/cray-pe-configuration.sh file, so the list of modules may depend on the actual system that you are using.

The Cray PE supports the following PrgEnv-* modules. On LUMI, only the first three are currently available:

Module Compiler module What?
PrgEnv-cray cce The Cray Compiling Environment compilers
PrgEnv-gnu gcc The GNU compilers
PrgEnv-aocc aocc AMD compilers for CPU-only systems
PrgEnv-amd rocm AMD ROCm compilers for GPU systems
PrgEnv-intel intel The Intel compilers
PrgEnv-nvidia nvidia NVIDIA HPC toolkit compilers (formerly PGI) (deprecated)
PrgEnv-nvhpc nvhpc New NVIDIA HPC toolkit compilers

All PrgEnv-* modules belong to the same module family. Hence Lmod will automatically unload any already loaded PrgEnv-* module when you load a different one.

Selecting the version through the cpe meta-module

The Cray PE on the EX system provides the meta-module cpe: the purpose of the meta-module is similar to the scope of the cdt and cdt-cuda meta-modules available on the XC systems.

Loading one of the cpe/yy.mm modules (e.g., cpe/24.03) has the following effects:

  • It sets the default versions of each of the Cray PE modules to the version that comes with that particular release of the Cray PE. E.g.,

    module load cpe/24.03
module load cce

    would load that version of the cce compiler that comes with the 24.03 release of the Cray PE.

  • It will try to reload all already loaded Cray PE modules and switch them over to the version corresponding to that particular release of the Cray PE.

Limitation when combining multiple loads in one module load statement

The Cray PE sets the default version of each module by adding a file to the list of files in the LMOD_MODULERCFILE environment variable. This is because Lmod does not re-evaluate the visibility of modules and the internal list of default version immediately when the value of LMOD_MODULERCFILE is changed but only the next time the module command is executed. Hence 

module load cpe/23.12 ; module load cce
and 
module load cpe/23.12 cce
do not have the same effect. In the first version, the version of cce loaded is the version that corresponds to the 23.12 release of the Cray PE. In the second case however the default version of the cce module is determined by whatever list of default modules was used when calling the module command so may or may not be the one of the 22.02 release. If you did this at the time of the last update of this part the text directly after login, with 24.03 as the default release of the programming environment, the version of cce that you got was that of the 24.03 release.

The compiler wrapper module craype

The craype module loads the compiler wrappers. There is only one set of compiler wrappers for all compilers. Which compiler will be called, which libraries will be included, but also processor and GPU target options will be used, is all determined by other modules. Hence it is in principle possible to use a single Makefile for a project and still reconfigure the build by loading certain modules.

Target modules

The targets for CPU and GPU optimization, the network library for MPI bt also some other compiler options, can be set through target modules:

  • craype-x86-* (and similar options can be expected on ARM-based systems) set the target for CPU optimisations. For LUMI, the crape-x86-rome, craype-x86-milan and craype-x86-trento modules are relevant.

    This can also be used to cross-compile to a different target architecture unless the compiler target gets overwritten by a compiler flag added to the command line through the Makefile or other means, something that unfortunately happens more and more often in faulty software installation procedures.

  • craype-accel-* sets the target for OpenMP offload (and likely other technologies in the future). E.g., loading craype-accel-amd-gfx90a tells the compilers to target AMD MI200 family GPUs, while loading craype-accel-host tells the compiler to use the CPU instead (according to the documentation, the latter is for PrgEnv-cray only).

    Unfortunately there is no craype-accel-none that would have the same effect as unloading any craype-accel-* module that may be loaded, as that would be a useful feature in EasyBuild recipes to ensure that no such module is loaded without having to search for the name of loaded modules.

  • craype-network-* selects the communication library to be used by Cray MPICH. On Slingshot 11 EX systems, only craype-network-ofi is supported, but older Slingshot 10 EX systems also offer support for UCX through the craype-network-ucx module (and LUMI did so before the upgrade of the network interfaces that came with the installation of LUMI-G).

  • The craype-hugepages* modules enable Cray Huge Pages support. To fully enable this support they have to be used at link-time and at run-time. At link time, support is compiled into the binary while at run-time they are used to set the actual size of the huge pages.

    The craype-hugepages* modules are not supported by all compilers. E.g., the AOCC compiler does not support huge pages at the moment, and loading the module at link time will cause an error message from the linker.

The compiler modules

The compiler modules have already been discussed with the PrgEnv-* modules above. The different regular compiler modules also all belong to the same family so no two different compilers can be loaded simultaneously and Lmod will automatically unload the other compiler when a new one is loaded.

The MPI modules

To load the Cray MPI libraries, both one of the craype-network-* modules and a compiler module has to be loaded as the MPI libraries are both network- and compiler specific.

Loading an MPI module will also automatically configure the regular compiler wrappers to compile with support for that MPI module. However, the libfabric and UCX versions of the MPI library have compatible interfaces, so it is always possible to swap between those versions at runtime.

The Cray Scientific libraries

The Cray Scientific Libraries are loaded through the cray-libsci module (or cray-libsci_acc for the GPU versions). Loading this module makes the BLAS, LAPACK, and ScaLAPACK libraries available, and also the Cray IRT (Iterative Refinement Toolkit), a Cray-specific library. It will also configure the compiler wrappers to link with these libraries, so no additional include or link options are needed.

The fftw module

As FFTW is a third party library, it gets its own module (cray-fftw) and is not included in the cray-libsci module.

The cray-fftw module can only be loaded if one of the processor target modules (the craype-x86-* modules) is loaded first.

Some unexpected behaviour of the Cray PE

On Cray EX systems, dynamic linking is the preferred way of linking applications. With that comes some unexpected behaviour of the Cray modules. EasyBuild users expect that at run time the versions of the libraries that are used, are the ones from the modules that are loaded. This is not always the case for the runtime libraries of the Cray PE. By default the Cray PE will use a default set of libraries that is determined by the system default version of the Cray PE (which is set by the sysadmins, not determined by the cpe module). This can only be avoided by either using rpath-linking (which is also special in the Cray PE as the wrappers activate rpath linking if the environment variable CRAY_ADD_RPATH is defined and set to yes) or by manually adjusting the library path after loading the modules: 

export LD_LIBRARY_PATH=${CRAY_LD_LIBRARY_PATH}:$LD_LIBRARY_PATH
The latter is automated through the lumi-CrayPath module that will also correctly reset LD_LIBRARYPATH when unloaded or adjust when reloaded.

The net result of this feature is that some reproducibility of the results is lost. Programs will react differently if the system default version of the Cray PE is changed as that will change the set of default libraries that will be used at runtime unless rpath-linking is used or users redefine LD_LIBRARY_PATH. But the advantage is that most compiled binaries will remain working after a system update that broke older MPI versions and that has happened in the past.

Further reading

[next: LUMI software stacks]