Allow for CUDA compute capability fallbacks when initialising EESSI #5

ocaisa · 2025-06-12T11:40:24Z

ocaisa · 2025-06-12T11:49:14Z

bot: build repo:eessi.io-2023.06-software instance:eessi-bot-mc-aws arch:x86_64/amd/zen2

eessi-bot-aws · 2025-06-12T11:49:20Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2025.06/pr_5/69119

date	job status	comment
Jun 12 11:49:19 UTC 2025	submitted	job id `69119` awaits release by job manager
Jun 12 11:49:51 UTC 2025	released	job awaits launch by Slurm scheduler
Jun 12 11:55:53 UTC 2025	running	job `69119` is running
Jun 12 12:04:00 UTC 2025	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-69119.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-17497293750.tar.gz` size: 0 MiB (5891 bytes) entries: 3 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `2023.06/init/bash` `2023.06/init/eessi_environment_variables` `2023.06/init/modules/EESSI/2023.06.lua`
Jun 12 12:04:00 UTC 2025	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos %scale=1_node /aeb2d9df @BotBuildTests:x86_64_amd_zen2+default P: perf: 442.103 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos %scale=1_node /04ff9ece @BotBuildTests:x86_64_amd_zen2+default P: perf: 427.154 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /775175bf @BotBuildTests:x86_64_amd_zen2+default P: latency: 1.99 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /52707c40 @BotBuildTests:x86_64_amd_zen2+default P: latency: 1.99 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /b1aacda9 @BotBuildTests:x86_64_amd_zen2+default P: latency: 3.89 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /c6bad193 @BotBuildTests:x86_64_amd_zen2+default P: latency: 4.13 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /15cad6c4 @BotBuildTests:x86_64_amd_zen2+default P: latency: 0.58 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /6672deda @BotBuildTests:x86_64_amd_zen2+default P: latency: 0.57 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /2a9a47b1 @BotBuildTests:x86_64_amd_zen2+default P: bandwidth: 7397.69 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /1b24ab8e @BotBuildTests:x86_64_amd_zen2+default P: bandwidth: 7365.31 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-69119.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

ocaisa · 2025-06-16T09:07:19Z

Saw the test step failing in #11 (comment) and want to check that it is not as a result of merging that PR.

bot: build repo:eessi.io-2023.06-software instance:eessi-bot-mc-aws arch:x86_64/amd/zen2

eessi-bot-aws · 2025-06-16T09:07:25Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2025.06/pr_5/70017

date	job status	comment
Jun 16 09:07:24 UTC 2025	submitted	job id `70017` awaits release by job manager
Jun 16 09:07:49 UTC 2025	released	job awaits launch by Slurm scheduler
Jun 16 09:13:51 UTC 2025	running	job `70017` is running
Jun 16 09:15:53 UTC 2025	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-70017.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-17500652490.tar.gz` size: 0 MiB (23621 bytes) entries: 10 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `2023.06/init/bash` `2023.06/init/easybuild/eb_hooks.py` `2023.06/init/eessi_defaults` `2023.06/init/eessi_environment_variables` `2023.06/init/lmod/bash` `2023.06/init/lmod/csh` `2023.06/init/lmod/fish` `2023.06/init/lmod/ksh` `2023.06/init/lmod/zsh` `2023.06/init/modules/EESSI/2023.06.lua`
Jun 16 09:15:53 UTC 2025	test result	😢 FAILURE (click triangle for details) Reason EESSI test suite was not run, test step itself failed to execute. Details ✅ job output file `slurm-70017.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

ocaisa · 2025-06-16T09:23:25Z

Something is going wrong with the new setup, untouched files are changing

ocaisa · 2025-06-17T13:45:11Z

bot: build repo:eessi.io-2023.06-software instance:eessi-bot-mc-aws arch:x86_64/amd/zen2

eessi-bot-aws · 2025-06-17T13:45:17Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2025.06/pr_5/70236

date	job status	comment
Jun 17 13:45:16 UTC 2025	submitted	job id `70236` awaits release by job manager
Jun 17 13:45:55 UTC 2025	released	job awaits launch by Slurm scheduler
Jun 17 13:51:57 UTC 2025	running	job `70236` is running
Jun 17 14:00:05 UTC 2025	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-70236.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-17501683640.tar.gz` size: 0 MiB (5902 bytes) entries: 3 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `2023.06/init/bash` `2023.06/init/eessi_environment_variables` `2023.06/init/modules/EESSI/2023.06.lua`
Jun 17 14:00:05 UTC 2025	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos %scale=1_node /aeb2d9df @BotBuildTests:x86_64_amd_zen2+default P: perf: 439.204 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos %scale=1_node /04ff9ece @BotBuildTests:x86_64_amd_zen2+default P: perf: 435.363 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /775175bf @BotBuildTests:x86_64_amd_zen2+default P: latency: 1.85 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /52707c40 @BotBuildTests:x86_64_amd_zen2+default P: latency: 1.9 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /b1aacda9 @BotBuildTests:x86_64_amd_zen2+default P: latency: 6.87 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /c6bad193 @BotBuildTests:x86_64_amd_zen2+default P: latency: 4.22 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /15cad6c4 @BotBuildTests:x86_64_amd_zen2+default P: latency: 0.61 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /6672deda @BotBuildTests:x86_64_amd_zen2+default P: latency: 0.54 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /2a9a47b1 @BotBuildTests:x86_64_amd_zen2+default P: bandwidth: 7300.05 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /1b24ab8e @BotBuildTests:x86_64_amd_zen2+default P: bandwidth: 7266.98 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-70236.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

trz42 · 2025-06-25T08:53:34Z

init/eessi_environment_variables

-                  show_msg "archdetect found supported accelerator for CPU target ${EESSI_ACCEL_SOFTWARE_SUBDIR}: ${EESSI_ACCEL_SUBDIR}"
+              if [ ! -d $EESSI_ACCEL_SOFTWARE_PATH/${EESSI_ACCEL_SUBDIR} ]; then
+                  # We should try to use the fallback compute capability
+                  EESSI_ACCELERATOR_TARGET="${EESSI_ACCEL_SUBDIR::-1}0"


This makes the assumption that one can always replace the last digit with a zero. While that may be true, it might be better to define a function for this ... and add a comment about it (maybe even a link to some documentation).

In case the naming does not imply compatibility in the same way as now, one could then also later change the function instead of changing the variable expansion/setting.

It is not enough to make a function just here, the same also needs to apply to the EESSI module. That feels overly complicated for now and could be done in a follow-up PR?

I can add a CI test that verifies that the fallback is as expected, that should make a follow-up PR easier.

With 39272de it should at least be more obvious that this is being checked, so should help sanity checking a follow-up PR

ocaisa · 2025-06-25T10:15:11Z

Rebuild is not necessary, but why not...

bot: build repo:eessi.io-2023.06-software instance:eessi-bot-mc-aws arch:x86_64/amd/zen2

eessi-bot-aws · 2025-06-25T10:15:17Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2025.06/pr_5/71934

date	job status	comment
Jun 25 10:15:16 UTC 2025	submitted	job id `71934` awaits release by job manager
Jun 25 10:16:02 UTC 2025	released	job awaits launch by Slurm scheduler
Jun 25 10:22:04 UTC 2025	running	job `71934` is running
Jun 25 10:30:13 UTC 2025	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-71934.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-17508469860.tar.gz` size: 0 MiB (5902 bytes) entries: 3 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `2023.06/init/bash` `2023.06/init/eessi_environment_variables` `2023.06/init/modules/EESSI/2023.06.lua`
Jun 25 10:30:13 UTC 2025	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos %scale=1_node /aeb2d9df @BotBuildTests:x86_64_amd_zen2+default P: perf: 437.569 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos %scale=1_node /04ff9ece @BotBuildTests:x86_64_amd_zen2+default P: perf: 443.987 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /775175bf @BotBuildTests:x86_64_amd_zen2+default P: latency: 6.85 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /52707c40 @BotBuildTests:x86_64_amd_zen2+default P: latency: 1.97 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /b1aacda9 @BotBuildTests:x86_64_amd_zen2+default P: latency: 4.31 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /c6bad193 @BotBuildTests:x86_64_amd_zen2+default P: latency: 4.11 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /15cad6c4 @BotBuildTests:x86_64_amd_zen2+default P: latency: 0.6 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /6672deda @BotBuildTests:x86_64_amd_zen2+default P: latency: 0.57 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /2a9a47b1 @BotBuildTests:x86_64_amd_zen2+default P: bandwidth: 7956.38 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /1b24ab8e @BotBuildTests:x86_64_amd_zen2+default P: bandwidth: 7523.1 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-71934.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
Jun 27 14:13:15 UTC 2025	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-amd-zen2-17508469860.tar.gz` to S3 bucket succeeded

lorisercole

I'm not an expert of EESSI, but this looks reasonable. Looking forward to test it!

bedroge

Looks good to me

bedroge · 2025-06-27T14:54:41Z

Staging PR merged, tarball ingested.

Allow for CUDA compute capability fallbacks when initialising EESSI

408dabf

ocaisa mentioned this pull request Jun 12, 2025

Allow for CUDA compute capability fallbacks when initialising EESSI EESSI/software-layer#1115

Closed

ocaisa added the ready-to-deploy label Jun 12, 2025

Merge remote-tracking branch 'upstream/main' into cuda_fallbacks

467b2ca

ocaisa removed the ready-to-deploy label Jun 16, 2025

Merge branch 'main' into cuda_fallbacks

e8bbaa4

ocaisa added the ready-to-deploy label Jun 17, 2025

ocaisa mentioned this pull request Jun 17, 2025

Add CI for EESSI-extend #15

Merged

Update verify_eessi_environment.py

fc60373

trz42 reviewed Jun 25, 2025

View reviewed changes

ocaisa added 2 commits June 25, 2025 12:04

Make CI clearer about how it checks overrides

39272de

Make sure the environment variable is actually set

04a8053

ocaisa added ready-to-deploy and removed ready-to-deploy labels Jun 25, 2025

ocaisa requested a review from trz42 June 25, 2025 10:30

lorisercole approved these changes Jun 27, 2025

View reviewed changes

bedroge approved these changes Jun 27, 2025

View reviewed changes

bedroge added bot:deploy and removed ready-to-deploy labels Jun 27, 2025

bedroge merged commit aa362c4 into EESSI:main Jun 27, 2025
50 checks passed

Allow for CUDA compute capability fallbacks when initialising EESSI #5

Allow for CUDA compute capability fallbacks when initialising EESSI #5

Uh oh!

Conversation

ocaisa commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ocaisa commented Jun 12, 2025

Uh oh!

eessi-bot-aws bot commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ocaisa commented Jun 16, 2025

Uh oh!

eessi-bot-aws bot commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ocaisa commented Jun 16, 2025

Uh oh!

ocaisa commented Jun 17, 2025

Uh oh!

eessi-bot-aws bot commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trz42 Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

ocaisa Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

ocaisa Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

ocaisa commented Jun 25, 2025

Uh oh!

eessi-bot-aws bot commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lorisercole left a comment

Choose a reason for hiding this comment

Uh oh!

bedroge left a comment

Choose a reason for hiding this comment

Uh oh!

bedroge commented Jun 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ocaisa commented Jun 12, 2025 •

edited

Loading

eessi-bot-aws bot commented Jun 12, 2025 •

edited

Loading

eessi-bot-aws bot commented Jun 16, 2025 •

edited

Loading

eessi-bot-aws bot commented Jun 17, 2025 •

edited

Loading

eessi-bot-aws bot commented Jun 25, 2025 •

edited

Loading