Speeding up Γ-only HSE in vasp_gam 6.6.1 on Intel GPUs

Queries about input and output files, running specific calculations, etc.


Moderators: Moderator, Global Moderator

Post Reply
Message
Author
chanaprom_cholsuk1
Newbie
Newbie
Posts: 5
Joined: Fri May 10, 2024 9:29 am

Speeding up Γ-only HSE in vasp_gam 6.6.1 on Intel GPUs

#1 Post by chanaprom_cholsuk1 » Mon Nov 16, 2026 3:53 pm

Dear VASP team,

I'm running HSE06 relaxations in bilayers (756 atoms, 15 A vacuum, gamma-only version 6.6.1, 1873 occupied bands) with the OpenMP-offload GPU port on 4× Intel Max 1550 per node, 1 rank per tile, 8 OpenMP threads.

Time per electronic step is fastest at 8 nodes (64 ranks) and gets slower at 12, 16, and more nodes. A 6.4.3 CPU run with > 1000 cores is faster than the best GPU setup.

I therefore have 2 quick questions:
1. Is this saturation in parallel scaling expected for HSE calculations, given that MPI parallelization is limited to bands (NCORE=1 and a single k-point)? In my tests, GPU scaling appears to saturate much earlier than CPU scaling.
2. Is there a recommended setting or algorithm for HSE on many GPUs? I found that changing NSIM or NBLOCK_FOCK made no difference.

Key tags are ENCUT = 500, LREAL = Auto, AEXX = 0.25, HFSCREEN = 0.2, ALGO = All, PRECFOCK = Fast.

Thank you in advance for the advice.

Best regards,
Chanaprom


manuel_engel1
Global Moderator
Global Moderator
Posts: 273
Joined: Mon May 08, 2023 4:08 pm

Re: Speeding up Γ-only HSE in vasp_gam 6.6.1 on Intel GPUs

#2 Post by manuel_engel1 » Tue Nov 17, 2026 9:18 am

Dear Chanaprom,

What you encounter is to some degree expected.

With a single k-point, the work can only be split over bands (on GPUs NCORE is always 1 as you correctly identified). In the exact-exchange part, every band (also the empty ones) is sent to all ranks, and the results are summed back on the rank that owns the band. The computation per rank shrinks as you add ranks, but each rank still moves roughly two copies of all bands per electronic step. It is typical for these calculations that communication becomes the bottleneck beyond about 8 nodes. NSIM and NBLOCK_FOCK only change how the work is batched, which is why they had no effect; for ALGO=All, NSIM is rounded up to a multiple of the number of ranks anyway. Your CPU run most likely scales further because it can use NCORE > 1, which cuts this traffic.

What I cannot say is whether your calculation is using your hardware efficiently. With multiple GPUs/CPUs per node, it's important that the processes are pinned correctly so that memory access is as fast as it can be. Often times, VASP is memory-bandwidth limited so improving that area is extremely important, though hybrids might be an exception since the computational demand is quite high.

Unfortunately, our expert on GPU offloading is not in the office this week, but once he comes back he might be able to provide additional insights. What might help in the meantime:

  • Set NBANDS yourself. The default might give you a large number of empty states. While some empty states are important for convergence, you can try to reduce that number. This will give better communication times and reduced compute across your whole setup.

  • Pre-relax with PBE and start HSE from that geometry and WAVECAR. Fewer ionic steps is the biggest saving.

If you'd like us to look further, please attach the OUTCAR files of an 8-node and a 16-node run (2-3 ionic steps are enough).

Manuel Engel
VASP developer


ahampel
Global Moderator
Global Moderator
Posts: 210
Joined: Tue Feb 16, 2016 11:41 am

Re: Speeding up Γ-only HSE in vasp_gam 6.6.1 on Intel GPUs

#3 Post by ahampel » Tue Nov 17, 2026 10:22 am

Dear Chanaprom,

just to add to Manuel's answer:

Please check also this wiki page https://vasp.at/wiki/GPU_ports_of_VASP . Importantly for intel GPU runs make sure to unset MKL_NUM_THREADS (then it is taken from OMP_NUM_THREADS) or set MKL_NUM_THREADS. It is also very important to set NSIM and NBLOCK_FOCK to larger values, although the default for NBLOCK_FOCK=32 is already quite okay. But 64 might improve performance further depending on the number of orbitals per rank / GPU. I think if you see scaling up to 8 nodes that is already very good. The 756 atoms unit cell should be relatively fast already then? However, the problem is that internode communication will kill you eventually as Manuel pointed out. To reduce MPI load you can try to switch to ALGO = Fast once. The RMM algo maybe scales better. If it converges for you.

Further upgrades to the Intel OMP offloading will come in newer VASP versions. If you want to help us to identify bottlenecks for future fixes you can compile your code once with -DVASP_PROFILING and share with us an OUTCAR of a 8 node run. Thanks! I would be curious to check the profile timings reported in the end.

Best,
Alex


Post Reply