Hello,
I am having an issue with a VASP calculation on NERSC Perlmutter where the calculation stops making progress after running for some time. There is no obvious error message, but the calculation eventually reaches the wall-time limit without progressing further.
The particular calculation where I am encountering this uses r2SCAN+LAK.
I initially encountered the problem using the NERSC VASP 6.6.0 CPU build. Since I wasn't sure whether the issue was related to that particular VASP build, I recently compiled VASP 6.6.1 myself on Perlmutter using the NVIDIA compiler environment. Unfortunately, I see the same behavior with VASP 6.6.1.
I contacted NERSC support, and they were also able to reproduce the problem with my example calculation. They tested different VASP builds/compiler environments. According to their tests:
1. the NVIDIA build stops making progress,
2. the GCC build also stops making progress after roughly an hour, and
3. the Intel build fails with a segmentation fault.
For the GCC calculation, they noticed that the last line written to standard output before it stops making progress is
'final diagonalization occupied'
They suggested that the issue may be occurring during a matrix diagonalization step and recommended that I ask the VASP developers for advice.
One other observation that may be useful is that I can run the same calculation successfully on the LONI cluster using VASP 6.5.0. So far, I have only encountered this problem on NERSC Perlmutter. I have also shared this information with NERSC support.
I am wondering what exactly VASP is doing at the final diagonalization occupied stage and whether there is a known issue that could cause a calculation to stop making progress there.
Is there anything you would recommend changing or testing to diagnose this further? For example, could this be related to the diagonalization method, ScaLAPACK/LAPACK, or the parallelization settings?
I can provide the complete reproducible example, including the INCAR, POSCAR, KPOINTS, OUTCAR/standard output, and job script.
POTCAR INFO:
PAW_PBE Mn_pv 02Aug2007 (VRHFIN = Mn: 3p4s3d)
PAW_PBE O 08Apr2002 (VRHFIN = O: s2p4)
PAW_PBE C 08Apr2002 (VRHFIN = C: s2p2)
PAW_PBE H 15Jun2001 (VRHFIN = H: ultrasoft test)
Thanks for your help!
Best,
Niraj