Description:
I tried to run oellm-eval on LUMI across multiple checkpoints for the all[eus_Latn] and all[nno_Latn|nob_Latn] evaluations using the following script:
#!/bin/bash
export HF_HOME="/scratch/project_465002530/cache/huggingface"
oellm-eval schedule --models ".../iter_0002000" --task_groups "all[lang_codes]" > submit_output.txt 2>&1
The Issue:
The lighteval jobs consistently fail due to hitting the time limit. Looking at the logs, it appears the model is falling back to the CPU instead of utilizing the GPUs:
...
[2026-06-24 21:00:22,415] [ INFO]: Using Data Parallelism, putting model on device cpu (transformers_model.py:222)...
...
Potential Cause & Proposed Fix:
Upon checking the submit_evals.sbatch script, I noticed it only sets the CUDA_VISIBLE_DEVICES environment variable. While this seems to work fine for lm-evaluation-harness, it is problematic for Accelerate-based lighteval on AMD hardware.
Because LUMI uses AMD GPUs, the script likely needs to set the AMD equivalent, HIP_VISIBLE_DEVICES, to prevent the CPU fallback.
Description:
I tried to run oellm-eval on LUMI across multiple checkpoints for the all[eus_Latn] and all[nno_Latn|nob_Latn] evaluations using the following script:
The Issue:
The lighteval jobs consistently fail due to hitting the time limit. Looking at the logs, it appears the model is falling back to the CPU instead of utilizing the GPUs:
Potential Cause & Proposed Fix:
Upon checking the submit_evals.sbatch script, I noticed it only sets the CUDA_VISIBLE_DEVICES environment variable. While this seems to work fine for lm-evaluation-harness, it is problematic for Accelerate-based lighteval on AMD hardware.
Because LUMI uses AMD GPUs, the script likely needs to set the AMD equivalent, HIP_VISIBLE_DEVICES, to prevent the CPU fallback.