VASP Forum

Dear VASP Expert Team,

I am running VASP 6.6.0 with Intel GPUs, using vasp_gam with the HSE06 functional. Please find attached the INCAR and POSCAR files.

Unfortunately, the calculation works only when using #SBATCH --ntasks-per-node=1 in my SLURM script. Whenever I increase this value to 2, 3, 4, or 8 (8 is the maximum for my 1 GPU node), or even when using #SBATCH --nodes=2 in the SLURM script, I encounter the following error:

scaLAPACK will be used
LDA part: xc-table for (Slater+PW92), standard interpolation
POSCAR, INCAR and KPOINTS ok, starting setup
FFT: planning ... GRIDC
FFT: planning ... GRID_SOFT
FFT: planning ... GRID
WAVECAR not read
entering main loop
N E dE d eps ncg rms rms(c)
-----------------------------------------------------------------------------
| |
| EDDAV: Call to ZHEGV failed. Returncode = 100 1 8 |
| |
| ----> I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <---- |
| |
-----------------------------------------------------------------------------

I would like to ask why increasing the number of MPI tasks per node leads to this error. Could you advise on the recommended way to run this type of HSE06 calculation with Intel GPUs, please?

I would like to note that this calculation works well for VASP 6.4.2 - gamma only on CPUs.

Thank you very much in advance.

Best regards,
Chanaprom

Dear Chanaprom,

Thanks for reaching out.

I was able to run your system on our 2 Intel Max1100 GPUs using 2 MPI ranks without any issues on one node. (I used a Gamma-only KPOINTS file and the standard recommended POTCARs). I tested this both with the vasp_gam and the vasp_std versions.

To analyze what is going on for you, please provide a bit more information:

• Do I understand correctly that you have only 1 GPU per node?

• What types of GPUs do you have? On which cluster do you run?

• What toolchain do you use? (Which version of IFX, MKL, MPI, etc.)

• Please provide your makefile.include, your jobscipt, your complete stdout, and your OUTCAR file, as well as the output of

  Code: Select all

  ldd vasp_gam

Generally, VASP GPU calculations should not be run on more than 1 rank per GPU. So if your node has indeed only 1 GPU, 1 MPI rank is sufficient. To leverage more CPU performance for parts of the code that are not GPU ported, use OpenMP threads on the CPU via e.g.

export OMP_NUM_THREADS=8

.

Cheers, Michael

Dear Michael,

Thank you very much for testing the calculation. Here is some additional information:
1. I have 4 GPUs per node.
2. It's Intel Sapphire Rapids Intel Xeon Platinum 8480+. I use this cluster: https://doku.lrz.de/hardware-of-supermu ... 91050.html
3 and 4: I have collected all the relevant details in the attached file toolchain_details.txt, except for makefile.include, since I did not install VASP myself. However, if this information is necessary, I will contact the cluster admin. Please let me know.

Regarding OpenMP threads, I have also attached my Slurm script to the file. For some unknown reason, it still does not work.

I look forward to hearing from you.

Cheers,
Chanaprom

Dear Chanaprom,

thanks for the prompt reply. I see that you are using the 2025.2.0 toolkit, but the build for Intel GPUs requires 2025.3.0 compilers or later!

On Supermuc at LRZ you should be able to load the 2025.3 toolchain like this:

module unload stack/24.1.0
module load stack/24.5.0
ml load intel/2025.3.0 mpi_settings/2.0 intel-toolkit/2025.3.0 hdf5/1.14.5-intel25

which should give you:

ml list
Currently Loaded Modulefiles:
 1) mpi_settings/2.0{mode=default}   5) intel-mkl/2025.3.0         9) intel-tbb/2022.3.0    13) intel-dpl/2022.10.0     

 2) stack/24.5.0{arch=auto}          6) intel-inspector/2024.1.0  10) intel-ipp/2022.3.0    14) intel-dpct/2025.3.0     

 3) intel/2025.3.0                   7) intel-dnn/2025.3.0        11) intel-dal/2025.6.0    15) intel-toolkit/2025.3.0  

 4) intel-mpi/2021.17.0              8) intel-itac/2022.4.0       12) intel-ippcp/2025.3.0  16) hdf5/1.14.5-intel25

Of course you (or someone) will need to re-compile VASP with these modules loaded.

Let me know if this fixes your issue, so I can either lock the topic, or try myself on Supermuc.

Cheers, Michael

Dear Michael,

Thank you very much. It works well now.

Cheers,
Chanaprom

Glad to hear it. Can I lock the topic?

yes, please feel free to lock it.