Skip to content

Using external modules from the Cray PE

[back: Creating easyconfig files]

What are external modules?

EasyBuild supports the use of modules that were not installed via EasyBuild. We refer to such modules as external modules. External modules do not define the EBROOT* and EBVERSION* environment variables that EasyBuild would define and uses internally in several easyblocks and some easyconfig files.

External modules are used extensively on Cray systems to interface with the Cray PE (which comes with its own modules and cannot be installed via EasyBuild): external modules can be used as dependencies, by including the module name in the dependencies list, along with the EXTERNAL_MODULE constant marker.

For example, to specify the module cray-fftw as a dependency, you should write the following in your easyconfig file: 

dependencies = [('cray-fftw', EXTERNAL_MODULE)]

For such dependencies, EasyBuild will:

  • load the module before initiating the software build and install procedure,

  • include a module load statement in the generated module file (for runtime dependencies),

  • create EBROOT* and EBVERSION* environment variables if proper metadata is defined for the external module (see later in this section). The latter can be picked up by easyblocks just as the corresponding variables set by regular EasyBuild-installed modules.

Note

The default version of the external module will be loaded unless a specific version is given as dependency, and here that version needs to be given as part of the name of the module and not as the second element in the tuple.

dependencies = [('cray-fftw/3.3.8.12', EXTERNAL_MODULE)]

If the specified module is not available, EasyBuild will exit with an error message stating that the dependency can not be resolved because the module could not be found, without searching for a matching easyconfig file from which it could generate the module.

EasyBuild Metadata for external modules

Metadata can be supplied to EasyBuild for external modules: using the --external-modules-metadata configuration option, the location of one or more metadata files can be specified.

The files are expected to be in INI format, with a section per module name and key-value assignments specific to that module.

The external modules metadata file can be also defined with the corresponding environment variable: 

echo $EASYBUILD_EXTERNAL_MODULES_METADATA 
/apps/common/UES/jenkins/production/easybuild/cpe_external_modules_metadata-21.04.cfg

The following keys are supported by EasyBuild:

  • name: software name(s) provided by the module
  • version: software version(s) provided by the module
  • prefix: installation prefix of the software provided by the module

For instance, the external module version loaded by the dependency cray-fftw can be specified as follows: 

[cray-fftw]
name = FFTW
prefix = FFTW_DIR/..
version = 3.3.8.10

The environment variable $EBROOTFFTW will also be defined according to the prefix specified in the metadata file, which is always pointing to an environment variable and starting a path relative from there.

As a result of this definition, to a correctly written easyblock, the cray-fftw/3.3.8.10 module will appear as if it is the EasyBuild-installed FFTW/3.3.8.10 module and during the installation process, the environment variables EBROOTFFTW and EBVERSIONFFTW will be defined as expected.

On LUMI, users don't need to be too concerned about the metadata file as the EasyBuild-user (and other hidden EasyBuild configuration modules) take care of pointing to the right metadata file, which is specific for each version of the Cray PE and hence each version of the LUMI software stack.

[next: Implementing easyblocks]