Dear Rika,
welcome to the VASP Forum! Thank you for the detailed description of the issue. However, may I ask you to also upload a minimal reproducible example (see also the forum posting guidelines) with the input and output files for your problematic run? With this information it will be much easier to dissect from where this hangup originates. With 15 hours of initialization of the ML force field I am already pretty sure that there is some hangup involved.
Am I correct to assume that you are working on the MUSICA HPC cluster? My colleague Alexander Hampel already did some testing on this system and gave me some hints. The issue may be that the line
Code: Select all
mpirun --bind-to none -np ${SLURM_NTASKS} "$VASP_BIN"
basically disables processor pinning (i.e. MPI ranks are not assigned to fixed cores but are free to be placed and even moved by the OS). This is generally not recommended on HPC systems because performance may suffer. In this specific case it may even lead to the hangups, Alexander noted that he always had to do explicit pinning to get these multi-GPU jobs to work on MUSICA. From his notes I tried to put together a recipe you can try out. First, I suggest to change the job script as follows:
Code: Select all
#!/bin/bash
#SBATCH --job-name=train_PbS
#SBATCH -N 1
#SBATCH --gres=gpu:4
#SBATCH --ntasks=4
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=16
#SBATCH --threads-per-core=1
#SBATCH -p zen4_0768_h100x4
#SBATCH --qos zen4_0768_h100x4
#SBATCH --time=71:59:00
set -euo pipefail
module --force purge
module load EESSI/2025.06
module load ASC/2025.06
module load NVHPC/25.9
module load FFTW/3.3.10-NVHPC-25.9
module load HDF5/1.14.5-NVHPC-25.9
module load CUDA/13.0
ulimit -s unlimited
export OMP_STACKSIZE=2048m # increased from 512m
export VASP_BIN=/home/rw74596/vasp.6.6.1/bin/vasp_std
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
export MKL_NUM_THREADS=${SLURM_CPUS_PER_TASK} # added
export OMP_PROC_BIND=close # "close" instead of "spread"
export OMP_PLACES=cores
export OMP_WAIT_POLICY=PASSIVE # added
export NVSHMEM_DISABLE=1 # added
export UCX_COLL_ENABLE=n # added
export UCX_WARN_UNUSED_ENV_VARS=n # added
echo "Job started: $(date)"
mpirun -n ${SLURM_NTASKS} ./gpu_wrapper.sh "$VASP_BIN" # call gpu_wrapper.sh for explicit pinning
echo "Job finished: $(date)"
I appended a short comment in the lines I edited. The mpirun command now calls a separate GPU wrapper script gpu_wrapper.sh. Here are the contents of this script (thanks to Alexander for sharing):
Code: Select all
#!/usr/bin/env bash
#
# GPU + CPU pinning wrapper using taskset
#
set -euo pipefail
# --- identify local rank ---
if [[ -n "${OMPI_COMM_WORLD_LOCAL_RANK:-}" ]]; then
LOCAL_RANK=${OMPI_COMM_WORLD_LOCAL_RANK}
elif [[ -n "${SLURM_LOCALID:-}" ]]; then
LOCAL_RANK=${SLURM_LOCALID}
else
echo "Error: cannot determine local rank" >&2
exit 1
fi
if (( LOCAL_RANK < 0 || LOCAL_RANK > 3 )); then
echo "Error: LOCAL_RANK=${LOCAL_RANK} not supported" >&2
exit 1
fi
# --- affinity tables ---
CPU_RANGES=(
"72-95,264-287"
"48-71,240-263"
"144-167,336-359"
"96-119,288-311"
)
GPUS=(0 1 2 3)
CPU_RANGE=${CPU_RANGES[$LOCAL_RANK]}
GPU_ID=${GPUS[$LOCAL_RANK]}
# --- threads ---
OMP_NUM_THREADS=${OMP_NUM_THREADS:-1}
# Optional sanity check
CORES_AVAILABLE=$(( ${CPU_RANGE#*-} - ${CPU_RANGE%-*} + 1 ))
if (( OMP_NUM_THREADS > CORES_AVAILABLE )); then
echo "Error: OMP_NUM_THREADS=${OMP_NUM_THREADS} > available cores (${CORES_AVAILABLE})" >&2
exit 1
fi
# --- environment ---
export CUDA_VISIBLE_DEVICES=${GPU_ID}
export OMP_PLACES=cores
export OMP_PROC_BIND=close
echo "[Rank ${LOCAL_RANK}] GPU=${GPU_ID} CPUs=${CPU_RANGE} OMP_NUM_THREADS=${OMP_NUM_THREADS}" >&2
# --- exec with pinning ---
exec taskset -c ${CPU_RANGE} "$@"
Just put these commands in a file called gpu_wrapper.sh, put it into your run directory and make it executable with
Now you can try to resubmit your job and see if the hangups persist. Hope this helps, please report back if you find more issues!
All the best,
Andreas Singraber