VASP Forum

Hello,

I am having an issue with a VASP calculation on NERSC Perlmutter where the calculation stops making progress after running for some time. There is no obvious error message, but the calculation eventually reaches the wall-time limit without progressing further.

The particular calculation where I am encountering this uses r2SCAN+LAK.

I initially encountered the problem using the NERSC VASP 6.6.0 CPU build. Since I wasn't sure whether the issue was related to that particular VASP build, I recently compiled VASP 6.6.1 myself on Perlmutter using the NVIDIA compiler environment. Unfortunately, I see the same behavior with VASP 6.6.1.

I contacted NERSC support, and they were also able to reproduce the problem with my example calculation. They tested different VASP builds/compiler environments. According to their tests:

1. the NVIDIA build stops making progress,
2. the GCC build also stops making progress after roughly an hour, and
3. the Intel build fails with a segmentation fault.

For the GCC calculation, they noticed that the last line written to standard output before it stops making progress is

'final diagonalization occupied'

They suggested that the issue may be occurring during a matrix diagonalization step and recommended that I ask the VASP developers for advice.

One other observation that may be useful is that I can run the same calculation successfully on the LONI cluster using VASP 6.5.0. So far, I have only encountered this problem on NERSC Perlmutter. I have also shared this information with NERSC support.

I am wondering what exactly VASP is doing at the final diagonalization occupied stage and whether there is a known issue that could cause a calculation to stop making progress there.

Is there anything you would recommend changing or testing to diagnose this further? For example, could this be related to the diagonalization method, ScaLAPACK/LAPACK, or the parallelization settings?

I can provide the complete reproducible example, including the INCAR, POSCAR, KPOINTS, OUTCAR/standard output, and job script.

POTCAR INFO:
PAW_PBE Mn_pv 02Aug2007 (VRHFIN = Mn: 3p4s3d)
PAW_PBE O 08Apr2002 (VRHFIN = O: s2p4)
PAW_PBE C 08Apr2002 (VRHFIN = C: s2p2)
PAW_PBE H 15Jun2001 (VRHFIN = H: ultrasoft test)

Thanks for your help!

Best,
Niraj

Hello Niraj,

welcome to the VASP forum! Thank you for your very detailed description of the problem, this helps a lot. However, before we can suggest anything specific it is important that you make the minimal reproducible example available to us. Actually we have some similar machines here where we can try to reproduce the hangup you are observing. Please upload all relevant input and output files (see also the forum posting guidelines). If available, please also upload the makefile.include you were using and describe the toolchain, i.e. which compiler and library versions you used.

Thank you!

All the best,
Andreas Singraber

Thanks Andreas Singraber for your reply.

I thought I had attached the relevant input/output files in my previous post, but it turns out I missed them.

I have attached them here, along with the makefile.include used to compile VASP.

Please check if you can see the attached

with this reply

Just to clarify, the attached output is from one of the calculations that ran normally for the first 4–5 hours and then stopped making any further progress. The job remained running without producing additional output until it eventually reached the 24-hour wall time.

This is the main issue I have been experiencing with these calculations.

Best,
Niraj

Dear Niraj,

thanks a lot for uploading the files, it actually allowed me to track down the issue. I used an NVIDIA toolchain and was able to reproduce the hangup on two independent runs (after 7 and 10 hours, each on two 48-core AMD EPYC 7643 nodes). The output would stall and the last output lines would contain "final diagonalization occupied".

The problem is a bug in VASP which affects calculations that use ALGO = All together with k-point parallelization (KPAR > 1). At the end of each electronic minimization, every MPI rank checks whether the SCF cycle has converged. In the ALGO = All code path this decision is made independently on each rank and is not synchronized between the k-point groups. Because of tiny floating-point rounding differences, one k-point group can occasionally conclude "converged" while another concludes "not yet". The groups then take different code paths and wait for each other in communication calls that never match. Hence, the job freezes without any error message.

A similar bug has surfaced before (see here) and a fix is actually already coming in the next VASP release. However, for now you can also try to apply this patch directly to your VASP 6.6.1 source: after 693 line src/electron_all.F add these two lines:

      CALLMPI(M_and(WDES%COMM, INFO%LABORT, 1))
      CALLMPI(M_or(WDES%COMM, LABORT_WITHOUT_CONV, 1))

Alternatively, download the patch file attached to this message and put it into your VASP 6.6.1 root directory. There, unpack and execute via

tar -xvzf hangup-fix.tar.gz
patch -p0 < hangup-fix.patch

to apply the fix automatically at the correct position. Once you have patched the source file, clean up (make veryclean) and rebuild VASP from scratch. Since triggering the issue takes quite some time I could not yet verify myself that the freezes do not occur any more (jobs are still running) but I am very confident that this solves the root cause of the problem.

I hope this helps, please report back whether the problem is gone with the fix applied. Thank you very much for testing and again for reporting!!

All the best,
Andreas Singraber