Skip to content

Frequently Asked Questions

This document is based on questions asked during user coffee breaks after the system update.

  1. How is the (driver) update schedule for LUMI looking in the future? Will there be more frequent updates than so far; or should we plan to stick with then ROCm 6.0-6.1 for a long time? Having at least ROCm 6.1 or newer would be nice to have (even if it is just AMD CLR 6.1).

    • We explain why this is not trivial in our update page.

      • The driver needs to be compatible with the OS.

      • The ROCm version needs to be compatible with the other compilers and MPI library.

        And we need full upstream support from HPE and AMD for that combination.

      • A driver update can cause a cascade of other updates that need to be done first. Anybody wants another 3 weeks downtime anytime soon? Updating a system the size of LUMI is not the same as updating a workstation or server. Try updating firmware on 1000 switches, 12000 GPUs, 7000 CPUs, 15000 NICs and be sure that the update went fine on all... Some software updates actually require firmware updates also.

    • So we will probably stay on the ROCm 6.0 driver for some time but we are working on making newer ROCm version available through modules. This is on a "use at your own risk" basis as we cannot guarantee full compatibility with other libraries on the system. Also note that the driver supports 2 minor version up and down (5.6, 5.7, 6.0, 6.1, 6.2). We will also provide containers that provide ROCm 6.1 and 6.2.

  2. I've noticed that LUMI/24.03 now includes PrgEnv-nvhpc/8.5.0 and PrgEnv-nvidia/8.5.0, but does not include PrgEnv-amd/8.5.0. For the NVHPC / NVIDIA variants, I expect that this is just a small issue and that they're not intended to be there. My question is if PrgEnv-amd/8.5.0 is a compiler environment which is (or will be) supported. This may influence our testing for our software and which installations we provide via EasyBuild for example. It does exist when just logging into LUMI (using CrayEnv).

    • We need to check while it is hidden. Note though that if you are using EasyBuild, you should not use any PrgEnv-* module directly and instead use the matching cpe* module (e.g., cpeAMD instead of PrgEnv-amd)

      Note that which cpe* modules are available, also depends on the partition module that is loaded. cpeAMD is irrelevant in partition/L and partition/C and hence is not installed in those partitions, and cpeAOCC is irrelevant in partition/G and not installed there.

    • Not sure though if the PrgEnv-nvhpc/8.5.0 and PrgEnv-nvidia/8.5.0 would even function correctly on the NVIDIA visualisation nodes.

  3. How can I compile an application with HIP support? Are there any changes to the previous method, it doesn't seem to be working for my application

    • You should be able to continue use the same tools in the same way.

      What has changed though is the location of the header files. There has been a deprecation warning since ROCm 5.3 and on ROCm 6 the deprecated headers were removed. But if you neglected the warnings and did not make the necessary changes, you may indeed run into problems.

  4. Is there a way to know the amount of GPU and CPU hours used by every project member?

    • No, and there are no plans to implement this.

      LUMI projects are meant to be a close collaboration between the people in the project, where each user knows well what the others are doing, and then this is just not really needed. It is also not on the other CSC systems as far as I know, and the scripts used on LUMI are derived from those.

      If what users do is so different that you'd like to manage a split of resources between users, these should have been different projects in the first place.

    • But you can get some unprocessed information via the Slurm sreport and sacct commands. This is rough data though and not in billing units.

      The mapping between Slurm resource use and billing units actually happens offline on a different system. Slurm cannot really handle the formulas that are used, and take into account the different formulas for GPU and CPU nodes.

  5. It seems natural that LUMI would provide a utility into which we feed the intended resources (as specified for SLURM) and what falls out is the number of billing units that will be used up. Is there such a tool yet?

    • It is trickier than it appears. The are more intricacies as the billing units depend also on the node type used and also on the exact parameters being used in the script. Especially with regards to core to memory ratio, see also https://docs.lumi-supercomputer.eu/runjobs/lumi_env/billing/.

      It is not realistic to develop a system that would correctly interpret all options that one can give to Slurm and that may even be interpreted differently depending on the Slurm partition and command used.

      The tool would also need to interpret the Slurm configuration files to understand all the defaults that Slurm would use.

  6. Is there a plan or a possibility in a near future to mount the XXX (name your organisation) servers on LUMI? This way it will spare a lot of duplication of data for the case we can afford some delay for reading the files.

    • No, and this is technically not feasible for many reasons:

      • On Cray OS, Lustre is the only networked file system. Other remote file systems would have to be offered via DVS (Data Virtualisation Service), also requiring capacity on the management nodes that would actually connect to the remote server. Cray does this because this is essential to control OS jitter, which is important for scalable applications.

        It is clear that this is not a solution that scales. And it would also require considerable extra hardware resources (and hence budget) for each such server.

      • NFS or most other network file systems you would use, work very badly over such long connections. The bandwidth that can be obtained, would be severely limited by the latency of the connection.

        We've tried this in Belgium between clusters with direct optical links and even that was unworkably slow for many transfers and certainly not an option for accessing large amounts of data. It's not "just a little delay".

        It would also lead to huge inefficiencies (lost time) when done on the compute nodes.

      • We could do this in Belgium because it was between clusters with a joint user domain. But this is not the case between servers of other organisations and LUMI, not even between puhti or mahti (CSC clusters) and puhti, so there would be some mapping between users in both domains needed. Not something that is quickly done or easy to maintain.

      • Any problem on the side of the other organisation, would also cause problems on LUMI that would (a) create a lot of work for our sysadmins and (b) be very hard to diagnose as they have no access to the remote site.

        Also performance problems would be impossible to diagnose.

      • And I'm pretty sure that your local sysadmin would also come with a lot of security issues if you would propose that to them. It is simply stupid to sent that kind of traffic over the public internet. You'd have to implement secure virtual networks over a long distance.

  7. Is there a way to get job notification emails on Lumi? I have tried with #SBATCH --mail-type=ALL and my email address such #SBATCH --mail-user=user@email.com but I do not get any notification emails.

    • There are also no plans to enable this function of Slurm email notifications. It is not that easy to do with architecture of LUMI based on isolated services, and also there has been abuse of this functionality on other systems. Moreover, the pattern of emails sent from LUMI would make it look like a spam bot to many systems, so there is not even a guarantee that the mails would ever arrive at your site, or worse, other CSC systems may also get blocked.