GW Out Of Memory Issues

[peter_gill]

Hello,

I am trying to do a GW calculation using VASP version 6.5.1 but keep running into Out Of Memory problems. During each run, VASP outputs a warning saying insufficient memory and quoting the available memory per rank along with the required memory. However, each time I try increasing the memory available, the required memory it quotes also goes up so that I never have enough memory and it always crashes. I have attempted increasing the total memory available and reducing the number of ranks.

I have tried running on 8 nodes x 14 tasks per node x 8 cpus per task and I am allocated a total of 3.50TB of memory. The WAVECAR that the calculation reads from is 2GB. The unit cell contains 152 atoms and there are a total of 320 electrons. I use 8 k-points which are all equivalent by symmetry. I am using the standard PAW_PBE potentials. All my parameters can be seen below. I have attached a zip file of the calculation inputs and corresponding OUTCAR files. The workflow is to run VASP using INCAR.DFT then INCAR.DIAG then INCAR.GW. The Out Of Memory error occurs on the GW step of the calculation which is the third step.

Does it make sense for the memory requirement to be this large for a WAVECAR of that size? And what could I try to reduce the memory requirements without losing convergence?

INCAR:

Code: Select all

ISMEAR =  0
NBANDS = 600
ENCUT = 400
EDIFF = 1E-8
ALGO = EVGW0
LSPECTRAL = .TRUE.
NOMEGA = 25
NELM = 1
PRECFOCK = Fast

KPOINTS:

Code: Select all

Automatically generated mesh
       0
Gamma
 2 2 2
 0 0 0

Submit script:

Code: Select all

#!/bin/bash -l
#SBATCH --time 10-00:00:00
#SBATCH --nodes=8
#SBATCH --ntasks-per-node=14
#SBATCH --cpus-per-task=8
#SBATCH --exclusive
module purge
module load bluebear
module load bear-apps/2024a/live
module load VASP/6.5.1-intel-2024a-Wannier90

export OMP_NUM_THREADS=1
export TERM=xterm

mpirun -np $SLURM_NTASKS vasp_std

exit

[leszek_nowakowski]

Hello!

Not sure about GW calculations, but I see some inconsistency in your setup.

You are allocating 8 cpus per each MPI rank, and then set OMP_NUM_THREADS to 1. It means that each rank uses only 1 cpu, not 8. Try always to set number of threads equal to cpus per task, eq. by setting OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK.
It will not change the memory allocation (threads share the same memory space), but it may speed up your calculations.

To investigate memory problems you can try allocate exclusively 1 MPI rank , cpus-per-task equal to number of cores on node (remember about OMP_NUM_THREADS!), and full memory. If your calculation will start, cancel it and double the ranks and lower cpus-per-task by two as well. You can shuffle them as much as you want - I see that You should have 112 cores per node ( 14 tasks and 8 cpus-per-task), so You can try 1 task-112 threads, 2 tasks-56 threads, 4 tasks-28 threads, 8 tasks-14 thread and so on) until You ran out of memory.

Keep in mind that OMP_STACKSIZE may be by default too small. Try to increase to 512m:

Code: Select all

export OMP_STACKSIZE=512m

Also, set this variables:

Code: Select all

export OMP_PLACES=cores
export OMP_PROC_BIND=close

It will pin threads to to cores closest to mother MPI rank.

Best regards,
Leszek

[merzuk.kaltak]

Dear Peter,

The majority of the problems you encounter stem from the use of the EVGW0 algorithm. I strongly advise switching to the low‑scaling GW variant EVGW0R (note the trailing R). Further details are available in our Practical Guide to GW.

Please be aware that GW calculations are most reliable for insulators or semiconductors—i.e., systems possessing a non‑zero band gap. The fractional occupancies present in your SCF step are likely to cause convergence difficulties if Gaussian smearing (ISMEAR=0) is retained. You have two possible strategies:

• If you wish to retain the highly symmetric structure (which inherently includes fractional occupancies), enable finite‑temperature many-body perturbation theory by adding LFINITE_TEMPERATURE = T to every INCAR file.

• If you can work with a symmetry‑broken version of the cell that exhibits a finite band gap, you may keep Gaussian smearing (ISMEAR=0) and avoid the finite‑temperature GW formalism altogether.

Regardless of the chosen approach, the current setting NBANDS = 720 is insufficient to describe the unoccupied manifold adequately for GW. I recommend increasing NBANDS to a value much closer to the maximum number of plane‑waves used in the SCF step. For example, the SCF output contains:

Code: Select all

$ grep "maximum number of plane-waves" OUTCAR.DFT
maximum number of plane-waves: 44336

Thus, a sensible choice would be NBANDS ≈ 44 400 for both the DIAG and GW steps. A sufficiently large NBANDS ensures convergence of the quasiparticle energies with respect to the basis set.

Additionally, you may experiment with MSDGW_F in the DIAG step. This tag implements the mixed stochastic‑deterministic GW approach of Altman et al. (Phys. Rev. Lett. 132, 086401), effectively reducing the number of virtual orbitals. When using MSDGW_F, adjust NBANDS in the GW step according to the number of “compressed bands” reported after the DIAG run. For instance, if the DIAG output contains:

Code: Select all

=> avg. energy ratio F (%): 1.00
bands after compression: 22400

you may set NBANDS = 22 400 for the subsequent GW calculation.

In my experience, the MSDGW feature becomes advantageous only for very high plane‑wave cutoffs (≥ 800 eV), which exceed the default cutoffs of most POTCAR files.