ML_FF stuck in initialization

[akretschmer]

Hi,

I am training a MLFF, but get stuck during initialization at the current step. Looking at the ML_LOGFILE, it seems that memory could be an issue, but I run the calculation on a node with 1024 GB memory and 64 physical cores. I also reduced ML_MB = from 5000 to 2000, which changed nothing. The already stored data in the ML_AB file is also very small, less than a MB, so memory really should be no problem. The funny thing is that this training step worked before, but I had to redo it because I found a mistake in my INCAR file. Now I just get stuck. One change I recently did was change the vasp version from 6.5.1 to 6.6.0, I don't know if this could be related. I am attaching the input and output files and this is my submission script:

Code: Select all

#!/bin/sh
#SBATCH -J ZnO-bulk
#SBATCH -N 1
#SBATCH --partition=zen3_1024
#SBATCH --ntasks-per-node=128
#SBATCH --qos=zen3_1024
#SBATCH --time=3-00:00:00

export OMP_NUM_THREADS=1
export I_MPI_PIN_RESPECT_CPUSET=0 

module purge
module load --auto vasp6/6.6.0-intel-2024.0.0_vsc5
export UCX_TLS=self,sm,dc_mlx5

ulimit -s unlimited

mpirun -np 128 --map-by ppr:64:socket --bind-to core vasp_std

[christopher_sheldon1]

Dear akretschmer,

Thank you for your question. I can run your MLFF training to at least the first ionic step using 4 cores and 128 GB memory, so I doubt that it is a lack of memory. I have a few minor suggestions:

• You are using quite a dense k-mesh. Is there any particular reason for this? A 2x2x2 mesh might make more sense.

• NCORE=24 isn't divisible by 64 or 128 so is internally reset by VASP to NCORE = 1, which will make your calculation much less efficient.

It looks like something in your environment. Could you upload a stdout, OUTCAR, and submission scripts for 6.5.1 with the same setup where it worked? I'd also recommend trying srun (e.g. srun --cpu-bind=cores $PATH_TO_VASP_EXECUTABLE), rather than mpirun.

Best wishes,

Chris