Skip to content

[package list]

ccpe

License information

User documentation

These containers are beta software

They are made available by HPE without guarantee of suitability for your purpose, as a way for users to test programming environments that are not (yet) on the system.

LUST together with HPE have made modules and implemented changes to the containers to adapt them to LUMI and integrate somewhat in the regular environment.

However, working with these containers is different from working with a programming environment that is installed natively on the system and requires a good insight in how containers work. So they are not for every user, and LUST can only offer very limited support. These containers are only for users who are very experienced with the Cray Programming Environment and also understand how singularity containers work.

The container only offers PrgEnv-cray and PrgEnv-gnu. With some imports from the system, we also offer PrgEnv-amd, but it may not be entirely as intended by the version of the PE as we may be using a different version of ROCm. The container does contain some elements of PrgEnv-nvidia but that is obviously not functional on LUMI. PrgEnv-aocc is not available.

HPE has a community Slack channel for feedback and questions at slack.hpdev.io, channel #hpe-cray-programming-environment, but bear in mind that this is mostly a community channel, monitored by some developers, but those developers don't have time to answer each and every question themselves. It is a low volume channel and in no means a support channel for inexperienced users.

LUST cannot really offer much support, though we are interested in learning about issues as this is useful feedback for HPE. These containers are really meant for experienced users who want to experiment with a newer version before it becomes available on LUMI.

Where to get the containers?

The CPE containers are made available in /appl/local/containers/easybuild-sif-images.

Note the licensing conditions though. These containers should only be used on LUMI.

How to enable the containers?

We recommend using our EasyBuild modules to run the HPE CPE containers as these modules do create a lot of bind mounts to provide all necessary parts from the system to the container.

All modules provide a number of environment variables to make life easier:

  • Outside (and brought into) the container, SIF and SIFCCPE point to the container file, which is very handy to use with the singularity command.

  • Inside the container, INITCCPE contains the commands to fully initialise the CPE in the container. Use as eval $INITCCPE.

    This is not needed when using singularity run or the corresponding wrapper script.

  • Outside (and brought into) the container, EXPORTCCPE is a list of environment variables set by the ccpe modules that we want to bring in the container or in a job script, even if we otherwise want to start the job script with a clean environment.

  • Outside (and brought into) the container, SWITCHTOCCPE is an environment variable containing a large block of code that is used at the start of the job script to switch to executing the job script in the container.

The module also provides access to four wrapper scripts to start the container. Note though that those wrapper scripts only function properly when the module is loaded. They do not take care of the bindings themselves and in that sense are certainly different from the wrapper scripts provided by Tykky/lumi-container-wrapper. All these scripts do however purge all modules before going into the container, as modules from the system PE are not valid in the container, and fully clear Lmod. Currently, the following scripts are provided:

  • ccpe-shell to start a shell in the container. The arguments of ccpe-shell are simply added to the singularity shell $SIF.

  • ccpe-exec to run a command in the container. The arguments of ccpe-exec are simply added to the singularity exec $SIF command.

  • ccpe-run to run the container. The arguments of ccpe-run are simply added to the singularity run $SIF command.

  • ccpe-singularity will clean up the environment for the singularity, then call singularity passing all arguments to singularity. So with this command, you still need to specify the container also (e.g., using the SIF or SIFCCPE environment variable), but can specify options for the singularity subcommand also.

Installing the EasyBuild recipes

To install the container module, chose the appropriate EasyConfig from this page, and make sure you have a properly set up environment as explained in the LUMI documentation in the "Installing software"section, "EasyBuild". In particular, it is important to set a proper location using EBU_USER_PREFIX, as your home directory will quickly fill up if you install in the default location. To install the container, use

module load LUMI/24.03 partition/container EasyBuild-user
eb <name_of_easyconfig>

e.g., 

module load LUMI/24.03 partition/container
eb ccpe-24.11-B-rocm-6.2-LUMI.eb --sourcepath <directory with the cpe_2411.sif file>

Any more recent version of the LUMI stack on the system will also work for the installation.

After that, the module installed by the EasyConfig (in the example, ccpe/24.11-rocm-6.2-LUMI) will be available in all versions of the LUMI stack on the system and in CrayEnv. So, e.g.,

module load CrayEnv ccpe/24.11-B-rocm-6.2-LUMI

is enough to gain access to the container and all its tools explained on this page.

How to get a proper environment in the container?

Unfortunately, there seems to be no way to properly (re-)initialise the shell or environment in the container directly through singularity shell or singularity exec.

The following strategies can be used:

  • In the container, the environment variable INITCCPE contains the necessary commands to get a working Lmod environment again, but then with the relevant CPE modules for the container. Run as

    eval $INITCCPE

  • Alternatively, sourcing /etc/bash.bashrc will also properly set up Lmod.

Cases that do give you a properly initiated shell, are singularity exec bash -i and singularity run. These commands do source /etc/bash.bashrc but do not read /etc/profile. But the latter shouldn't matter too much as that is usually used to set environment variables, and those that are typically set in that file and the files it calls, are typically fine for the container, or overwritten anyway by the files sourced by /etc/bash.bashrc.

Launching jobs: A tale of two environments

The problem with running jobs, is that they have to deal with two incompatible environments:

  1. The environment outside the container that does not know about the HPE Cray PE modules of the PE version in the container, and may not know about some other modules depending on how /appl/lumi is set up (as this may point to a totally separate software stack specific for the container but mounted on /appl/lumi so that it behaves just as the regular stacks on the system).

  2. The environment inside the container that does not know about the HPE Cray PE modules installed in the system, and may not know about some other modules depending on how /appl/lumi is set up.

This is important, because unloading a module in Lmod requires access to the correct module file, as unloading is done by "executing the module file in reverse": The module file is executed, but each action that changes the environment, is reversed. Even a module purge will not work correctly without the proper modules available. Environment variables set by the modules may remain set. This is also why the module provides the ccpe-* wrapper scripts for singularity: These scripts are meant to be executed in an environment that is valid outside the container, and clean up that environment before starting commands in the container so that the container initialisation can start from a clean inherited environment.

See how broken the job environment can be...

This example is developed running a container for the 24.11 programming environment on LUMI in March 2025 with the 24.03 programming environment as the default.

The 24.03 environment comes with cce/17.0.1 while the 24.11 environment comes with cce/18.0.1. When loading the module, it sets the environment variable 'CRAY_CC_VERSION' to the version of the CCE compiler.

Start up the container:

ccpe-run

Check the version of the module tool:

module --version

which returns version 8.7.37:

Modules based on Lua: Version 8.7.37  [branch: release/cpe-24.11] 2024-09-24 16:53 +00:00
    by Robert McLay mclay@tacc.utexas.edu

and list the modules:

module list

returns

Currently Loaded Modules:
1) craype-x86-rome                                 6) cce/18.0.1           11) PrgEnv-cray/8.6.0
2) libfabric/1.15.2.0                              7) craype/2.7.33        12) ModuleLabel/label (S)
3) craype-network-ofi                              8) cray-dsmml/0.3.0     13) lumi-tools/24.05  (S)
4) perftools-base/24.11.0                          9) cray-mpich/8.1.31    14) init-lumi/0.2     (S)
5) xpmem/2.9.6-1.1_20240510205610__g087dc11fc19d  10) cray-libsci/24.11.0

Where:
S:  Module is Sticky, requires --force to unload or purge

so we start with the Cray programming environment loaded.

Now use an interactive srun session to start a session on the compute node.

srun -n1 -c1 -t10:00 -psmall -A<my_account> --pty bash

Let's check the version of the module tool again:

module --version

now returns version 8.7.32: 

Modules based on Lua: Version 8.7.32  2023-08-28 12:42 -05:00
    by Robert McLay mclay@tacc.utexas.edu

as we are no longer in the container but in a regular LUMI environment.

Trying

module list

returns

Currently Loaded Modules:
6) craype-x86-rome                                 6) cce/18.0.1           11) PrgEnv-cray/8.6.0
7) libfabric/1.15.2.0                              7) craype/2.7.33        12) ModuleLabel/label (S)
8) craype-network-ofi                              8) cray-dsmml/0.3.0     13) lumi-tools/24.05  (S)
9) perftools-base/24.11.0                          9) cray-mpich/8.1.31    14) init-lumi/0.2     (S)
10) xpmem/2.9.6-1.1_20240510205610__g087dc11fc19d  10) cray-libsci/24.11.0

Where:
S:  Module is Sticky, requires --force to unload or purge

so the modules we were using in the container.

The environment variable CRAY_CC_VERSION is also set:

echo $CRAY_CC_VERSION

returns 18.0.1.

Now do a

module purge

which shows the perfectly normal output

The following modules were not unloaded:
(Use "module --force purge" to unload all):

1) ModuleLabel/label   2) lumi-tools/24.05   3) init-lumi/0.2

The following sticky modules could not be reloaded:

1) lumi-tools

and 

module list

now shows

Currently Loaded Modules:
1) ModuleLabel/label (S)   2) lumi-tools/24.05 (S)   3) init-lumi/0.2 (S)

Where:
S:  Module is Sticky, requires --force to unload or purge

but 

echo $CRAY_CC_VERSION

still return 18.0.1, so even though it appears that the cce/18.0.1 module has been unloaded, not all (if any) environment variables set by the module, have been correctly unset.

We can now load the cce module again:

module load cce

and now

module list cce

returns

Currently Loaded Modules Matching: cce
1) cce/17.0.1

so it appears we have the cce module from the system. This went well in this case. And in fact,

module list

which returns

Currently Loaded Modules:
1) ModuleLabel/label (S)   4) craype/2.7.31.11     7) craype-network-ofi   10) PrgEnv-cray/8.5.0
2) lumi-tools/24.05  (S)   5) cray-dsmml/0.3.0     8) cray-mpich/8.1.29    11) cce/17.0.1
3) init-lumi/0.2     (S)   6) libfabric/1.15.2.0   9) cray-libsci/24.03.0

Where:
S:  Module is Sticky, requires --force to unload or purge

suggests that some other modules, like cray-mpich and cray-libsci have also been reloaded.

echo $CRAY_CC_VERSION

returns 17.0.1 as expected, and after

module purge

we now note that

echo $CRAY_CC_VERSION

returns nothing and is reset.

However, it is clear that we are now in an environment where we cannot use what we prepared in the container.

Job script template to run the batch script in the container

To make writing job scripts easier, some common code has been put in an environment variable that can be executed via the eval function of bash.

This job script will start with as clean an environment as possible, except when called from a correctly initialised container with passing of the full environment:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
#!/bin/bash
#
# This test script should be submitted with sbatch from within a CPE 24.11 container.
# It shows very strange behaviour as the `module load` of some modules fails to show
# those in `module list` and also fails to change variables that should be changed.
#
#SBATCH -J example-jobscript
#SBATCH -p small
#SBATCH -n 1
#SBATCH -c 1
#SBATCH -t 5:00
#SBATCH -o %x-%j.md
# And add line for account

#
# Ensure that the environment variable SWITCHTOCCPE and with it 
#
if [ -z "${SWITCHTOCCPE}" ]
then
    module load CrayEnv ccpe/24.11-B-rocm-6.2-LUMI || exit
fi

#
# Now switch to the container and clean up environments when needed and possible.
#
eval $SWITCHTOCCPE

#
# Here you have the container environment and can simply work as you would normally do:
# Build your environment and start commands. But you'll still have to be careful with
# srun as whatever you start with srun will not automatically run in the container.
#

module list

srun <srun options> singularity exec $SIFCCPE <command>

What this job script does:

  • The body of the job script (lines after eval $SWITCHTOCCPE) will always run in the container.

    This is where you would insert your code that you want to run in the container.

  • When launching this batch script from within the container:

    • When launched with --export flag, the body will run in the environment of the calling container.

      It does require that the job is started from a properly initialised container with active Lmod though, as that currently sets the environment variable to detect if the container is properly initialised.

      If you started the calling container with ccpe-run, there is no issue though. In other cases, you may have to execute eval $INITCCPE. But in general, if you were able to load Cray PE modules before calling sbatch, you should be OK.

    • When launched using sbatch --export=$EXPORTCCPE, the body will run in a clean container environment, but will not need to re-load the container module.

    • Behaviour with --export=none: As the container cannot be located, 

      if [ -z "${SWITCHTOCCPE}" ]
then
    module load CrayEnv ccpe/24.11-B-rocm-6.2-LUMI || exit
fi

      will first try to load the container module, and if successful, proceed creating a clean environment.

      Note that you need to adapt that line to the module you are actually using!

  • When launching this batch script from a regular system shell:

    • When launched using sbatch --export=$EXPORTCCPE, the body will run in a clean container environment.

    • When launched with --export flag, eval $SWITCHTOCCPE will first try to clean the system environment (and may fail during that phase if it cannot find the modules that you had loaded when calling sbatch.)

      If the ccpe module was not loaded when calling the job script, the block 

      if [ -z "${SWITCHTOCCPE}" ]
then
    module load CrayEnv ccpe/24.11-B-rocm-6.2-LUMI || exit
fi

      will try to take care of that. If the module can be loaded, the script will proceed with building a clean container environment.

      Note that you need to adapt that line to the module you are actually using!

    • Behaviour with --export=none: As the container cannot be located, 

      if [ -z "${SWITCHTOCCPE}" ]
then
    module load CrayEnv ccpe/24.11-B-rocm-6.2-LUMI || exit
fi

      will first try to load the container module, and if successful, proceed creating a clean environment.

      Note that you need to adapt that line to the module you are actually using!

  • So in all cases you get a clean environment (which is the only logical thing to get) except if sbatch was already called from within the container without --export flag.

For technical information about how all this works under the hood (and it may be important to understand this to always use the template correctly), check the subsection "Starting jobs" in the "Technical documentation" section of this page.

Known restrictions

  • PrgEnv-aocc is not provided by the container. The ROCm version is taken from the system and may not be the optimal one for the version of the PE.

  • salloc does not work in the container.

    Workaround: USe salloc outside the container, then go into the container with ccpe-run.

Singularity containers with modules for binding and extras

Install with the EasyBuild-user module in partition/container: 

module load LUMI partition/container EasyBuild-user
eb <easyconfig>
The module will be available in all versions of the LUMI stack and in the CrayEnv stack.

To access module help after installation use module spider ccpe/<version>.

EasyConfig:

  • EasyConfig ccpe-24.11-B-rocm-6.2-LUMI.eb, will provide ccpe/24.11-B-rocm-6.2-LUMI

    This module provides a modified version of the CPE container as it comes from HPE. The goal is to integrate it better with the current LUMI environment. Changes to the container are also made through this EasyConfig, so you may use it as a template to adapt the CPE containers to your needs.

    This B-rocm version bind mounts the actual ROCm installation from a SquashFS file provided on LUMI which should give better compile performance than when simply mounting a directory from one of the LUMI parallel file systems with the ROCm installation. Some libraries on the OS side may not be the best version, so in case of trouble, switch to the C-rocm version.

    This container can be installed in partition/container via LUMI/24.03 or more recent LUMI stacks, after which the module will be available in CrayEnv and all versions of the LUMI stack.

  • EasyConfig ccpe-24.11-C-rocm-6.2-LUMI.eb, will provide ccpe/24.11-C-rocm-6.2-LUMI

    This module provides a modified version of the CPE container as it comes from HPE. The goal is to integrate it better with the current LUMI environment. Changes to the container are also made through this EasyConfig, so you may use it as a template to adapt the CPE containers to your needs.

    This C-rocm version also integrates ROCm in the container and installs it using the SUSE zypper install tool, guaranteeing that all dependencies are also present in the right version. As such, it is the most robust variant of the containers with ROCm. However, installing ROCm with the unprivileged build process is extremely slow, so expect long build times (over one and a half hour on a compute node), rendering this approach rather inefficient. The assembly of the container may also run out-of-memory on the login nodes as the memory available to a single user is restricted to 96GB.

    This container can be installed in partition/container via LUMI/24.03 or more recent LUMI stacks, after which the module will be available in CrayEnv and all versions of the LUMI stack.

  • EasyConfig ccpe-25.03-B-rocm-6.3-SP5-LUMI.eb, will provide ccpe/25.03-B-rocm-6.3-SP5-LUMI

    This module provides a modified version of the CPE container as it comes from HPE. The goal is to integrate it better with the current LUMI environment. Changes to the container are also made through this EasyConfig, so you may use it as a template to adapt the CPE containers to your needs.

    This B-rocm version bind mounts the actual ROCm installation from a SquashFS file provided on LUMI which should give better compile performance than when simply mounting a directory from one of the LUMI parallel file systems with the ROCm installation. Some libraries on the OS side may not be the best version, so in case of trouble, switch to the C-rocm version.

    This container can be installed in partition/container via LUMI/24.03 or more recent LUMI stacks, after which the module will be available in CrayEnv and all versions of the LUMI stack.

  • EasyConfig ccpe-25.03-C-rocm-6.3-SP5-LUMI.eb, will provide ccpe/25.03-C-rocm-6.3-SP5-LUMI

    This module provides a modified version of the CPE container as it comes from HPE. The goal is to integrate it better with the current LUMI environment. Changes to the container are also made through this EasyConfig, so you may use it as a template to adapt the CPE containers to your needs.

    This C-rocm version also integrates ROCm in the container and installs it using the SUSE zypper install tool, guaranteeing that all dependencies are also present in the right version. As such, it is the most robust variant of the containers with ROCm. However, installing ROCm with the unprivileged build process is extremely slow, so expect long build times (over one and a half hour on a compute node), rendering this approach rather inefficient. The assembly of the container may also run out-of-memory on the login nodes as the memory available to a single user is restricted to 96GB.

    This container can be installed in partition/container via LUMI/24.03 or more recent LUMI stacks, after which the module will be available in CrayEnv and all versions of the LUMI stack.

Technical documentation

These containers should not be spread outside LUMI, and some even contain unofficial versions and should not be spread to more users than needed. So do not spread without explicit agreement with HPE.

Issues

Which approach?

  1. One can do as much as possible outside the container, injecting any change via bind mounts. This makes for very easy debugging, as one can simply change those injected files without rebuilding the container.

    As we shall see below, some minimal components have to be injected though to get Slurm to work.

  2. Create a custom container, implementing modifications and additions as much as possible in the container.

    This also enables us to install a lot of extra software in the container, and give users an easy way to build a fully customised version.

    It also greatly simplifies the list of bind mounts.

    Debugging is a bit easier, but if we store copies from the files installed in the container also outside the container, it is still easy to temporarily inject them and experiment with changes.

The first approach also enables us to make the standard container available on a place accessible to all users, without the need for them to store a copy in their project. On the other hand, having a local copy protects against changes that admins may make, and the size of the container is small compared to the size of a dataset one can expect for users who use LUMI to solve bigger problems.

Managing bind mounts

It is not hard to build a module that sets the SINGULARITY_BIND environment variable with all necessary bind mounts. Moreover, that environment variable is also made available in the container so can be further propagated easily to a job script.

A tale of two environments

On the system and inside the container, the module view is different. On the system, one sees all modules form the programming environments installed on the system, and modules in the LUST-installed stacks and possibly other local stacks.

In the container, one sees a different set of programming environment modules, and depending on which software stacks are mounted, a different set of software stack modules.

This matters because Lmod can only fully unload a module if it has access to the module file. Unloading is done by executing the module file while reversing the effect of commands that make changes in the environment. If you execute a module purge or module --force purge on a list of modules for which not all module files are available, they will still appear as unloaded, but environment variables set by these modules will not be unset, and worse, directories will not be correctly removed from PATH-style environment variables. Lmod will not warn for that: It is a feature that any unload operation always succeeds, even if it could not be done correctly.

This may be troublesome for the programming environment. As so much data is communicated to the compiler wrappers via environment variables, they may be mislead and do unintended operations like trying to link in libraries that should not be linked in.

This has implication on how we go into containers - one should clean up the system environment before entering in the container - and on Lmod: One should not use the same cache for both environments. It also complicates starting jobs.

All these elements will be discussed below.

Wrapper scripts

The EasyBuild-installed modules do provide some wrapper scripts that make some tasks easier. They try to deal with the two environments problem by first purging the system environment before calling the singularity command.

singularity shell wrapper script ccpe-shell

This is a convenience wrapper script and is not useful if you want to pass arguments to singularity (rather than to the shell it starts).

The script does a clean-up of the modules in your environment and then cleans up Lmod completely, as the environment outside the container is not compatible with the environment in the container and as the loaded modules are needed to correctly unload them. It does save the environment variables set by the ccpe module to restore them after the module --force purge operation as otherwise the container wouldn't function properly anymore.

To make sure that /etc/profile would execute properly if it were called, the script also unsets PROFILEREAD, as the environment generated by the system /etc/profile may not be the ideal one for the container.

singularity exec wrapper script ccpe-exec

This is a convenience wrapper script and is not useful if you want to pass arguments to singularity rather than to the command you start.

It performs the same functions as ccpe-shell, but passes its arguments to the singularity exec $SIFCCPE command.

singularity run wrapper script ccpe-run

This is a convenience wrapper script and is not useful if you want to pass arguments to singularity (rather than to the command it tries to run, if given).

It performs the same functions as ccpe-shell, but passes its arguments to the singularity run $SIFCCPE command.

singularity wrapper script ccpe-singularity

This wrapper only cleans up the environment and then calls singularity passing all arguments of the wrapper unmodified to the singularity command. So you also need to pass the name of the container image, but can now call any singularity command with all singularity command-specific options in a clean environment.

Lmod caches

As the environment in the container is not compatible with the environment outside, we cannot use the regular user Lmod cache, or it may get corrupted, certainly if a user is working both in and out of the container at the same time.

Possible solutions/workarounds:

  1. Work with LMOD_IGNORE_CACHE=1 in the container. As the whole of /appl/lumi is mounted in the containers by our EasyConfigs, this will slow down module searches in Lmod considerably.

  2. Modify /opt/cray/pe/lmod/lmod/libexec/myGlobals.lua: Look for the line with usrCacheDir and define a unique directory for it, e.g., .cache/lmod/ccpe-{version}{versionsuffix} or any other container-specific version string.

    This procedure is very easy in the approach where we do as much as possible work inside the container. All one need is a sed command in the %post section of the build process.

    When trying to do everything as much as possible outside the container, the solution is to use a singularity exec to copy the file from the container to the system, edit that file (both can be done in a postinstallcmds in EasyBuild), and then bind mount that file to the container.

Initialisation

We need two types of initialisation of the CPE container:

  • When first going into the container, an environment fully compatible with the container needs to be set up.

  • When starting Slurm jobs from within the container that should also run in the container, then we really would like an option to propagate that environment.

    This requires some care when writing the job script, but the module defines an environment variable that can be eval'ed to properly initialise the environment in the job script and run the job script itself in a container.

Also, a full initialisation cannot be done entirely in the container:

  • Singularity will pass the environment from the calling process. This includes also the Lmod data structures and all variables set by the currently loaded modules.

    While it is easy to reset the Lmod data structures, it is not possible to properly reset all other environment variables that are set by those modules. This can only be done by unloading the modules (which executes the module script while reverting the effect of all commands that set something in the environment). As the regular CPE modules from the system are not available in the container, the unloading cannot be done in the container but has to be done before calling singularity.

  • When running an interactive shell in the container, you then want to construct a proper environment in the container. Singularity may source /etc/bash.bashrc which in turn may or may not source other initialisation scripts such as /etc/bash.bashrc.local.

    It looks like if you call singularity exec or singularity shell, there is no automatic initialisation taking place though. So we create an environment variable in the container, INITCCPE, that will at least take care of initialising Lmod properly.

Initialisation is influenced by the following scripts:

  • Scripts in /singularity.d/env: Used at the start of singularity shell, singularity exec, singularity run.

    What one can do in these scripts, is limited though. It is a good place to set environment variables that should be available in the container.

  • What happens with profile, bash.bashrc, profile.local and bash.bashrc.local, depends also on which Linux variant, et., is being used.

    For the CPE containers:

    • In SUSE, one is advised to only use profile.local and bash.bashrc.local for site-specific changes and to not change profile and bash.bashrc.

    • /etc/profile will source the scripts in /etc/profile.d and then source /etc/profile/local if that script exists. The script does not exist in the CPE containers though.

    However, neither of those is called when a shell is started with singularity shell or singlarity exec. As can be seen from files in /.singularity.d/actions, singularity exec simply execs the command in a restricted shell (/bin/sh) while singularity shell starts bash with the --norc option.

    singularity run as defined for the CPE container however does source /etc/bash.bashrc and hence /etc/bash.bashrc.local and the ~/.bashrc file from the user. However, after reading ~/.bashrc, there is still some code somewhere that resets the PS1 environment variable to either the value of SINGULARITYENV_PS1 or Singlarity>. Somehow, before calling ~/.bashrc, PROMPT_COMMAND is set to something like PS1=<prompt from singularity> ; unset PROMPT_COMMAND. Now if PROMPT_COMMAND is set, it is executed before showing the prompt defined by PS1 and this hence resets the prompt that is set in., e.g., ~/.bashrc.

As currently we have no proper solution to fully initialise the container from the regular Linux scripts when using singularity shell or singularity exec, the modules define the INITCCPE environment variable which contains the commands to execute to initialise Lmod in the container. Use eval $INITCCPE for that purpose.

Our EasyBuild modules do provide a /etc/bash.bashrc.local file that does the same initialisations as eval $INITCCPE. So calling source /etc/bash.bashrc is also an option to initialise Lmod in the container.

Knowing we are in the container

There are different solutions for that purpose in job scripts:

  • Check if the directory /.singularity.d exists

  • Singularity sets a number of environment variables, e.g., SINGULARITY_CONTAINER, and one can check for those.

  • Specifically for the CPE containers, one could also check for one of the environment variables set by the ccpe modules.

During interactive use:

  • Singularity will set the prompt to Singularity>

    It does so in a dirty way by putting the command to set the prompt in the environment variable PROMPT_COMMAND and then unset that environment variable as part of the command. As a consequence, in those cases where ~/.bashrc is read, any prompt defined in that script may be reset with the singularity one if you do not explicitly unset PROMPT_COMMAND.

  • It is possible to overwrite the default singularity prompt by setting SINGULARITYENV_PS1.

  • One can define a suitable prompt in ~/.bashrc, at least for singularity run, and use any of the tricks given above to detect if one ~/.bashrc is executing in the container.

How to recognise if an environment is compatible with the container?

There is no easy way to see this from the PE modules that are loaded as these modules do not set environment variables that point at the release of the PE, except that in recent version, the PE release is part of the version number for LibSci and perftools.

Current solution: Set and environment variable: CCPE_VERSION={version}{versionsuffix}} (or some other unique version for the container) after a proper initialisation of the environment in the container.

This is important as we do not want to clear an environment that is compatible with the container when we start a job or jog step, and only want to do so if it is not.

When starting Slurm jobs from within the container, this is important as one can then set the necessary environment variables in the calling container already, mimicking the behaviour that users are used to from running jobs outside containers. Moreover, we need to be able to set up an environment in the job script that is then properly inherited when using srun to create job steps as otherwise, each MPI rank would also have to first create a proper environment.

Getting Slurm to work

The container images that LUST provides have been prepared for Slurm support.

The container images that LUST provides as base images, have been modified in two crucial places to enable Slurm support by only bind mounting other files and directories. The text below is relevant though if you want to download your own image from the HPE support site and derive from our EasyConfigs to use (on, e.g., a different system for which you happen to be licensed to use the containers).

Bind mounting the Slurm commands and libraries and some other libraries and work directories that they use, is not enough to get Slurm working properly in the container. The container needs to know the slurm user with the correct user- and groupid. The slurm user has to be known in /etc/passwd and /etc/group in the container.

We know only one way to accomplish this: Rebuilding the container and using the %files section in the definition file to copy those two files from LUMI:

Bootstrap: localimage

From: cpe-24.11.sif

%files

    /etc/group
    /etc/passwd

Approaches that try to modify these files in the %post phase, don't work. At that moment you're running a script in singularity, and you don't see the real files, but virtual ones with information about your userid and groups added to those files. Any edit will fail or be discarded, depending on how you do it.

Bind-mounting those files from the system also does not work, as singularity then assumes that those files contain all groups and userid, and will not add the lines for userid and groups of the user that is running the container to the virtual copies.

We have adapted the base images that we provide so that the -raw modules below that only rely on bind mounts and environment variables set through the module, can still support running Slurm commands inside the container. However, if a user wants to adapt those scripts for another container downloaded from the HPE web site, or even the same container if they are licensed to use it elsewhere and want to build on our work, they will have to rebuild that image with the above definition file.

And of course, if they would like to use it on a different system, things can be different, as, e.g., the numeric user and group id for the Slurm user may be different. Forget about portability of containers if you need to use these tricks...

Starting jobs

The problem with running jobs, is that they have to deal with two incompatible environments, as discussed before.

We'd like to run the job script in the container to be able to construct an environment compatible with the container, but Slurm will of course start a job batch script or regular job step in a regular system shell. So we will need to call singularity at the right point.

In total there are six scenarios that we take into account for jobs launched with sbatch:

  1. Jobs can be launched from within the container:

    1. Using sbatch without --export: LUMI uses the default behaviour of Slurm, which is inheriting the full environment. As we like to run the batch script also in a container, we'd also like to inherit that environment in the container that the job will start up.

      This is also the behaviour that we want when starting job steps with srun from within the container.

    2. Using sbatch with --export=$EXPORTCCPE: The environment variable EXPORTCCPE contains the names of all environment variables set by the ccpe module and that are essential to be able to run the container. So with this way of starting, the batch script will start with a clean environment with no system modules loaded, yet the essential environment variables needed to start the container in the batch script will still be there.

    3. Using sbatch with --export=NONE: The batch script will start in a clean system environment, unaware of the CPE container from which it was started.

  2. Jobs can be launched from the system:

    1. Using sbatch without --export: Now the batch script will run in the system environment and that environment has to be cleaned up before starting the singularity container.

      That clean-up is no different from the clean-up the wrapper scripts do.

      Note that one cannot be sure that the environment variables from the ccpe moudle will be set as the batch script can be launched without them.

    2. Using sbatch with --export=$EXPORTCCPE: Not really different from when launching from within the container. If the ccpe module is not loaded and EXPORTCCPE is undefined, this may behave a little strange...

    3. Using sbatch with --export=NONE: The batch script will start in a clean system environment.

The steps that we take to start a job and execute the job script in the container:

  1. One should first ensure that necessary environment variables to find back the container, do the proper bindings, etc., are defined. So one my have to load the container if those variables are not present. This code should not be executed in the container as it would fail to find the module anyway.

  2. Except in one case, we now have a system environment with just some default module loaded, or other modules also loaded.

    Clear that environment in a similar way as we do for the wrappers.

  3. Restart the job script in the singularity container, but skip the code for the previous two steps.

  4. Ensure that the container is properly initialised if we did not inherit a valid container environment.

  5. Execute the remainder of the job script in the container.

Possible code to accomplish this is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
#!/bin/bash
#SBATCH -J jobscript-template
…

#
# Step 1: Make sure the container environment variables are set.
# 
if [ -z "${SWITCHTCCPE}" ]
then
    module load CrayEnv ccpe/24.11-LUMI || exit
fi

if [ ! -d "/.singularity.d" ]
then

    #
    # Clear the system environment
    #
    if [ "$CCPE_VERSION" != "{local_ccpe_version}" ]
    then

        for var in ${{EXPORTCCPE//,/ }}
        do
            if [ -v $var ] 
            then
                eval $(declare -p $var | sed -e "s/$var/save_$var/")
            fi
        done

        module --force purge
        eval $($LMOD_DIR/clearLMOD_cmd --shell bash --full --quiet)
        unset LUMI_INIT_FIRST_LOAD
        unset PROFILEREAD

        for var in ${{save_EXPORTCCPE//,/ }}
        do
            varname="save_$var"
            if [ -v $varname ]
            then
                eval $(declare -p $varname | sed -e "s/save_$var/$var/")
                unset $varname
            fi
        done

    fi

    # 
    # Restart the job script in the container
    #
    exec singularity exec "$SIFCCPE" "$0" "$@"

fi

#
# Ensure that the container is properly initialised, if this is not yet the case
#
if [ "$CCPE_VERSION" != "{local_ccpe_version}" ]
then

    lmod_dir="/opt/cray/pe/lmod/lmod"

    function clear-lmod() {{ [ -d $HOME/.cache/lmod ] && /bin/rm -rf $HOME/.cache/lmod ; }}

    source /etc/cray-pe.d/cray-pe-configuration.sh

    source $lmod_dir/init/profile

    mod_paths="/opt/cray/pe/lmod/modulefiles/core /opt/cray/pe/lmod/modulefiles/craype-targets/default $mpaths /opt/cray/modulefiles /opt/modulefiles"
    MODULEPATH=""
    for p in $(echo $mod_paths)
    do 
        if [ -d $p ] 
        then
            MODULEPATH=$MODULEPATH:$p
        fi
    done
    export MODULEPATH=${{MODULEPATH/:/}}

    LMOD_SYSTEM_DEFAULT_MODULES=$(echo ${{init_module_list:-PrgEnv-$default_prgenv}} | sed -E "s_[[:space:]]+_:_g") ;
    export LMOD_SYSTEM_DEFAULT_MODULES
    eval "source $BASH_ENV && module --initial_load --no_redirect restore"
    unset lmod_dir

    export CCPE_VERSION="{local_ccpe_version}"

fi

unset SLURM_EXPORT_ENV ;

# 
# From here on, the user can insert the code that runs in the container.
#

module list

srun … singularity exec $SIFCCPE <command>

This is of course way to complicated to expose to users, but line 57-86 is put in the environment variable INITCCPE so that code can be replaced with eval $INITCCPE, and then the whole block from line 13 till (and including) line 88 is put in the environment variable SWITCHTOCCPE so that block can be replaced with eval $SWITCHTOCCPE

So basically, all that a user needs is

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
#!/bin/bash
#SBATCH -J jobscript-template
…

#
# Step 1: Make sure the container environment variables are set.
# 
if [ -z "${SWITCHTCCPE}" ]
then
    module load CrayEnv ccpe/24.11-LUMI || exit
fi

#
# Steps 2-4: Clean up, switch to executing in the container and ensure a proper 
# container environment.
#
eval $SWITCHTOCCPE

# 
# From here on, the user can insert the code that runs in the container.
#

module list

srun … singularity exec $SIFCCPE <command>

Let us analyse the code a bit more:

The block from line 13 till 53 is only executed if not in the context of the container, so the first time the batch script runs. If it does not detect an environment from the container (the test on line 19, with {local_ccpe_version} replaced with the actual version string for the container as determined by the EasyConfig) then:

  • It first saves some environment variables set by the CCPE modules that should not be erased.

    This needed some clever bash trickery to avoid that environment variables get expanded.

  • Next, it purges all currently loaded modules which hopefully are from the system environment as otherwise variables may not be unset,

  • Next it clears Lmod to that all Lmod data structures are removed. Lmod does come with its own command to do that which is what we call here in the special way required by Lmod (as the command basically generates a sequence of bash commands that do the work).

  • Next it restores the environment variables from the ccpe module as they have been erased by the module purge.

Finally, on line 50m it restarts the batch script with all its arguments in the container. This causes the batch script to execute again from the start, but as SWITCHTOCCPE should be defined when we get here, and since we will now be in the container, all code discussed so far will be skipped.

The code between line 54 and 86 is already executed in the container. So if the code detects that there is already a valid environment for the container (where we again simply test for the value of CCPE_VERSION), nothing more is done, but if there is no proper environment, the remaining part of this routine basically runs the code used on LUMI to initialise Lmod with the proper modules from the HPE Cray Programming Environment, but now in the container. As it is done in the container, you will get the programming environment from the container.

On line 88, the script unsets SLURM_EXPORT_ENV. This environment variable would be set by Slurm if --export was used to submit the batch job, but we do not want this to propagate to job steps started with srun in the container, as there we want the full environment that the user builds in the job script to be propagated.

The salloc command does not yet work in a container

Currently, we haven't found a way yet to get the salloc command to work properly when started in the container. The workaround is to use salloc outside the container, then go in the container.

EasyBuild

Versions 24.11-*-rocm-6.2-LUMI

The -LUMI versions (currently the only ones offered) are built specifically to support a LUMI software stack based on the version of the CPE in the container.

Two different ways are currently offered

  • 24.11-B-rocm-6.2-LUMI bind mounts a SquashFS file with ROCm 6.2.4 to the container. This keeps the size of the container small and makes it easier to adapt the container to the needs of a specific project.

  • 24.11-C-rocm-6.2-LUMI installs ROCm from the AMD site using the SUSE zypper tool. It is a slow approach with build times easily exceeding an hour and a half. However, using zypper enables users to change the ROCm version themselves more easily, and also ensures that all OS dependencies are available in the proper version.

An alternative is to install the ROCm installation from a compressed tar file that LUST could provide, but though that may be a faster build process, it does not guarantee that all OS dependencies are available in the proper version.

Our advise is to start with the -B-rocm version and if there are issues that may come from library compatibility versions, switch to the -C-rocm versions.

For the 24.11-B-rocm-6.2-LUMI version, the ROCm SquashFS file and corresponding bzip2-compressed tar files were obtained from a ROCm installation in another container. To repeat the trick as a user with a different version of ROCm, you will have to modify either the bind mount source (-B-rocm) or the location of the compressed tar file (-C-rocm variant) as these are in a location managed by LUST.

  • The rocm, amd and amd-mixed module files are copied from the system (in %files) and and then edited through sed in %post to change the version to 6.2.4.

  • The rocm*.pc files in /usr/lib64/pkgconfig are copied from the system ((in %files) and and then edited through sed in %post to change the version to 6.2.4.

  • Also in %post, a symbolic link for /opt/rocm is created pointing to /opt/rocm-6.2.4 through /etc/alternatives.

We made several modifications to the container so that we can install a LUMI software stack almost the way we would do so without a container. Several of the files that we inject in the container, also have copies in the installation directory of the container, subdirectory config, which may be useful to experiment with changes and overwrite the versions in the container.

Some elements of the EasyBuild recipes and modifications to the container:

  • Variables defined at the top of the EasyConfig file:

    • local_ccpe_version: This will be used as the value for CCPE_VERSION and the directory used for the Lmod cache. Set to 24.11-raw for this container.

    • local_appl_lumi: System subdirectory that will be used for /appl/lumi in the container.

  • The container that has been provided by LUST as a starting point, does have some protection built in to prevent it being taken to other systems. One element of that protection, is some checks of the /etc/slurm/slurm.conf file.

    To be able to use the %post section during the "unpriveleged proot build" process, that file has to be present in the container. Therefore we copy that file in the %files phase, but remove it again in the %post phase as whe running the container, the whole Slurm configuration directory is bind mounted in the container.

  • We inject the file /.singularity.d/env/99-z-ccpe-init which we use to define additional environment variables in the container that can then be used to execute commands.

    Currently used so that eval $INITCCPE does a full (re)initialization of Lmod so that it functions in the same way as on LUMI.

  • As the container is already set up to support a runscript, we simply inject a new one via %files which makes it easier to have a nice layout in the runscript and in the container definition file.

  • Lmod cache strategy: User cache stored in a separate directory, ~/.cache/lmod/ccpe-%(version)s-%(versionsuffix)s, by editing /opt/cray/pe/lmod/lmod/libexec/myGlobals.lua.

  • libfabric and CXI provider: Bind mount from the system.

    To find the correct directories and files to bind, execute the following commands:

    module --redirect show libfabric | grep '"PATH"' | awk -F'"' '{ print $4 }' | sed -e 's|/bin||'
module --loc --redirect show libfabric | sed -e 's|\(.*libfabric.*\)/.*|\1|'
ldd $(module --redirect show libfabric | grep '"LD_LIBRARY_PATH"' | awk -F'"' '{ print $4 }')/libfabric.so | grep libcxi | awk '{print $3}'

  • ROCm: ROCm 6.2.4, either bind mounted from a SquashFS file or installed in the container via zypper.

  • Slurm support is still provided as much as possible by binding files from the system to ensure that the same version is used in the container as LUMI uses, as otherwise we may expect conflicts.

    One thing is done during the build process though: We need to copy the /etc/group and /etc/passwd files from the system into the container during the %files phase (editing those files in the %post phase does not work).

  • We made a deliberate choice to not hard-code the bindings in the ccpe-* scripts in case a user would want to add to the environment SINGULARITY_BIND variable, and also deliberately did not hard-code the path to the container file in those scripts as in this module, a user can safely delete the container from the installation directory and use the copy in /appl/local/containers/easybuild-sif-images instead if they built the container starting from our images and in partition/container.

    The ccpe-* wrapper scripts are defined in the EasyConfig itself (multiline strings) and brought on the system in postinstallcmds via a trick with bash HERE documents.

  • The sanity check is specific to the 24.11 containers and will need to be updated for different versions of the programming environment.

Versions 25.03-*-rocm-6.3-SP5-LUMI

These containers build upon the Cray SUSE 15 SP5 version of the container. They add both ROCm 6.3.4 and 6.2.4 to the container. The rationale is that the CPE in the container is really meant to be used with ROCm 6.3, but running with ROCm 6.3 may fail as the driver on LUMI is too old, so one can experiment with both versions or see if it is possible to compile with ROCm 6.3 (which may happen even if another ROCm version is loaded anyway as the CCE compiler now already includes some code from ROCm 6.3) while run with the older version of the libraries.

Two different versions are currently provided:

  • 25.03-B-rocm-6.3-SP5-LUMI bind mounts SquashFS file with ROCm 6.2.4 and 6.3.4 to the container. This keeps the size of the container small and makes it easier to adapt the container to the needs of a specific project.

    The container build recipe only creates the /opt/rocm-6.2.4 and /opt/rocm-6.3.4 directories as mount points and as it is needed to successfully complete some other steps discussed below.

  • 25.03-C-rocm-6.3-SP5-LUMI installs ROCm 6.2.4 and 6.3.4 from the AMD site using the SUSE zypper tool. It is a slow approach with build times easily exceeding an hour and a half. However, using zypper enables users to change the ROCm version themselves more easily, and also ensures that all OS dependencies are available in the proper version.

An alternative is to install the ROCm installation from a compressed tar file that LUST could provide, but though that may be a faster build process, it does not guarantee that all OS dependencies are available in the proper version.

Our advise is to start with the -B-rocm version and if there are issues that may come from library compatibility versions, switch to the -C-rocm versions. T

Furthermore,

  • The rocm, amd and amd-mixed module files are copied from the system (in %files) and and then edited through sed in %post to change the version to 6.2.4 and 6.3.4.

  • The rocm*.pc files in /usr/lib64/pkgconfig are copied from the system ((in %files) and and then edited through sed in %post to change the version to 6.2.4 and 6.3.4.

  • Also in %post, a symbolic link for /opt/rocm is created pointing to /opt/rocm-6.3.4 through /etc/alternatives.

Other elements in the build:

  • Variables defined at the top of the EasyConfig file:

    • local_ccpe_version: This will be used as the value for CCPE_VERSION and the directory used for the Lmod cache. Set to 24.11-raw for this container.

    • local_appl_lumi: System subdirectory that will be used for /appl/lumi in the container.

    • ROCm-related variables:

      • local_rocm_version: System ROCm version, currently 6.0.3

      • local_c1_rocm_version: Default ROCm version for the container, 6.3.4

      • local_c2_rocm_version: Backup ROCm version for the container, 6.2.4.

  • The file /.singularity.d/env/99-z-ccpe-init is used to define additional environment variables in the container that can then be used to execute commands. It is generated in the EasyConfig and then injected in the container through the %files section of the definition file.

    Currently used so that eval $INITCCPE does a full (re)initialization of Lmod so that it functions in the same way as on LUMI.

  • The /etc/bash/bashrc.local file is replaced with one that just calls eval $INITCCPE, and there is an empty placeholder for /etc/profile.local.

  • As the container is already set up to support a runscript, we simply inject a new one via %files which makes it easier to have a nice layout in the runscript and in the container definition file.

  • Compared to the 24.11 container, several packages were missing so a lot more needs to be added with zypper. This includes an editor, but more importantly, only the C and POSIX locales were known which was not even enough to use archspec or install some other packages that give additional effects in EasyBuild. The solution was to install glibc-locale.

  • Lmod cache strategy: User cache stored in a separate directory, ~/.cache/lmod/ccpe-%(version)s-%(versionsuffix)s, by editing /opt/cray/pe/lmod/lmod/libexec/myGlobals.lua.

  • libfabric and CXI provider: Bind mount from the system.

    To find the correct directories and files to bind, execute the following commands:

    module --redirect show libfabric | grep '"PATH"' | awk -F'"' '{ print $4 }' | sed -e 's|/bin||'
module --loc --redirect show libfabric | sed -e 's|\(.*libfabric.*\)/.*|\1|'
ldd $(module --redirect show libfabric | grep '"LD_LIBRARY_PATH"' | awk -F'"' '{ print $4 }')/libfabric.so | grep libcxi | awk '{print $3}'

  • The 25.11 containers need the xpmem libraries and module from the system which is done through bind mounts (could in principle replace with copying from the system).

  • ROCm: Either bound from a SquashFS file, installed from tar files or installed via zyppr, depending on the container.

  • Slurm support is still provided as much as possible by binding files from the system to ensure that the same version is used in the container as LUMI uses, as otherwise we may expect conflicts.

    One thing is done during the build process though: We need to copy the /etc/group and /etc/passwd files from the system into the container during the %files phase (editing those files in the %post phase does not work).

  • We made a deliberate choice to not hard-code the bindings in the ccpe-* scripts in case a user would want to add to the environment SINGULARITY_BIND variable, and also deliberately did not hard-code the path to the container file in those scripts as in this module, a user can safely delete the container from the installation directory and use the copy in /appl/local/containers/easybuild-sif-images instead if they built the container starting from our images and in partition/container.

    The ccpe-* wrapper scripts are defined in the EasyConfig itself (multiline strings) and brought on the system in postinstallcmds via a trick with bash HERE documents.

  • The sanity check is specific to the 25.11 containers and will need to be updated for different versions of the programming environment. We've tried to catch everything which depends on the version of the PE in variables in the EasyConfig, defined just above the sanity check commands (currently only 1).