Speeding up Γ-only HSE in vasp_gam 6.6.1 on Intel GPUs

Queries about input and output files, running specific calculations, etc.


Moderators: Moderator, Global Moderator

Post Reply
Message
Author
chanaprom_cholsuk1
Newbie
Newbie
Posts: 7
Joined: Fri May 10, 2024 9:29 am

Speeding up Γ-only HSE in vasp_gam 6.6.1 on Intel GPUs

#1 Post by chanaprom_cholsuk1 » Mon Nov 16, 2026 3:53 pm

Dear VASP team,

I'm running HSE06 relaxations in bilayers (756 atoms, 15 A vacuum, gamma-only version 6.6.1, 1873 occupied bands) with the OpenMP-offload GPU port on 4× Intel Max 1550 per node, 1 rank per tile, 8 OpenMP threads.

Time per electronic step is fastest at 8 nodes (64 ranks) and gets slower at 12, 16, and more nodes. A 6.4.3 CPU run with > 1000 cores is faster than the best GPU setup.

I therefore have 2 quick questions:
1. Is this saturation in parallel scaling expected for HSE calculations, given that MPI parallelization is limited to bands (NCORE=1 and a single k-point)? In my tests, GPU scaling appears to saturate much earlier than CPU scaling.
2. Is there a recommended setting or algorithm for HSE on many GPUs? I found that changing NSIM or NBLOCK_FOCK made no difference.

Key tags are ENCUT = 500, LREAL = Auto, AEXX = 0.25, HFSCREEN = 0.2, ALGO = All, PRECFOCK = Fast.

Thank you in advance for the advice.

Best regards,
Chanaprom


manuel_engel1
Global Moderator
Global Moderator
Posts: 275
Joined: Mon May 08, 2023 4:08 pm

Re: Speeding up Γ-only HSE in vasp_gam 6.6.1 on Intel GPUs

#2 Post by manuel_engel1 » Tue Nov 17, 2026 9:18 am

Dear Chanaprom,

What you encounter is to some degree expected.

With a single k-point, the work can only be split over bands (on GPUs NCORE is always 1 as you correctly identified). In the exact-exchange part, every band (also the empty ones) is sent to all ranks, and the results are summed back on the rank that owns the band. The computation per rank shrinks as you add ranks, but each rank still moves roughly two copies of all bands per electronic step. It is typical for these calculations that communication becomes the bottleneck beyond about 8 nodes. NSIM and NBLOCK_FOCK only change how the work is batched, which is why they had no effect; for ALGO=All, NSIM is rounded up to a multiple of the number of ranks anyway. Your CPU run most likely scales further because it can use NCORE > 1, which cuts this traffic.

What I cannot say is whether your calculation is using your hardware efficiently. With multiple GPUs/CPUs per node, it's important that the processes are pinned correctly so that memory access is as fast as it can be. Often times, VASP is memory-bandwidth limited so improving that area is extremely important, though hybrids might be an exception since the computational demand is quite high.

Unfortunately, our expert on GPU offloading is not in the office this week, but once he comes back he might be able to provide additional insights. What might help in the meantime:

  • Set NBANDS yourself. The default might give you a large number of empty states. While some empty states are important for convergence, you can try to reduce that number. This will give better communication times and reduced compute across your whole setup.

  • Pre-relax with PBE and start HSE from that geometry and WAVECAR. Fewer ionic steps is the biggest saving.

If you'd like us to look further, please attach the OUTCAR files of an 8-node and a 16-node run (2-3 ionic steps are enough).

Manuel Engel
VASP developer


ahampel
Global Moderator
Global Moderator
Posts: 211
Joined: Tue Feb 16, 2016 11:41 am

Re: Speeding up Γ-only HSE in vasp_gam 6.6.1 on Intel GPUs

#3 Post by ahampel » Tue Nov 17, 2026 10:22 am

Dear Chanaprom,

just to add to Manuel's answer:

Please check also this wiki page https://vasp.at/wiki/GPU_ports_of_VASP . Importantly for intel GPU runs make sure to unset MKL_NUM_THREADS (then it is taken from OMP_NUM_THREADS) or set MKL_NUM_THREADS. It is also very important to set NSIM and NBLOCK_FOCK to larger values, although the default for NBLOCK_FOCK=32 is already quite okay. But 64 might improve performance further depending on the number of orbitals per rank / GPU. I think if you see scaling up to 8 nodes that is already very good. The 756 atoms unit cell should be relatively fast already then? However, the problem is that internode communication will kill you eventually as Manuel pointed out. To reduce MPI load you can try to switch to ALGO = Fast once. The RMM algo maybe scales better. If it converges for you.

Further upgrades to the Intel OMP offloading will come in newer VASP versions. If you want to help us to identify bottlenecks for future fixes you can compile your code once with -DPROFILING and share with us an OUTCAR of a 8 node run. Thanks! I would be curious to check the profile timings reported in the end.

Best,
Alex


chanaprom_cholsuk1
Newbie
Newbie
Posts: 7
Joined: Fri May 10, 2024 9:29 am

Re: Speeding up Γ-only HSE in vasp_gam 6.6.1 on Intel GPUs

#4 Post by chanaprom_cholsuk1 » Tue Nov 17, 2026 8:34 pm

Dear Manuel and Alex,

Thank you for the explanations and suggestions.

Please find attached the calculation files for the 8-node and 16-node runs.

I had initially expected the calculation to run relatively quickly on 8 GPU nodes. However, the CPU run using 7,200 cores was substantially faster. I know that this is not a fair comparison, given the different resources used. Nevertheless, I had hoped that increasing the number of GPU nodes would provide a substantial speedup. Instead, GPU scaling appears to saturate much earlier than CPU scaling for this system. As the attached results show, increasing from 8 to 16 nodes did not significantly improve the performance.

I have already unset MKL_NUM_THREADS, as recommended. I have not yet tested the suggested ALGO setting, but I will try it.

Regarding the upcoming improvements to Intel OpenMP offloading, do you have an approximate timeline for a VASP release that would include them? Even a tentative estimate would help us plan our calculations.

Thank you again for your help.

Best regards,
Chanaprom

You do not have the required permissions to view the files attached to this post.

ahampel
Global Moderator
Global Moderator
Posts: 211
Joined: Tue Feb 16, 2016 11:41 am

Re: Speeding up Γ-only HSE in vasp_gam 6.6.1 on Intel GPUs

#5 Post by ahampel » Wed Nov 18, 2026 9:14 am

Dear Chanaprom,

thank you for sharing these. I did a mistake in my reply. Please do compile in the classical makefile.include with adding the CPP_OPTION: -DPROFILING . The cmake option would be -DVASP_PROFILING=ON . I mixed these up. Please try to rebuild and rerun. The end of the OUTCAR should have new sections for detailed function calls and timings. For these runs you can also cap the number of iterations to 10-15 . But let it run a few more steps than 5 to get statistics. Thanks!

From what I can see in these OUTCARs rank and thread pinning looks very good. I think your nodes have 112 cores no? Then you could use also more threads per rank. Something like 12 if possible. From your runs I can see that scaling beyond 8 nodes is not really beneficial. But I need the profiling sections to confirm where the time goes (MPI, or a specific function). That would help me a lot.

One problem could be that with 64 ranks each rank only gets 36 bands here (2304 bands in total reported in the OUTCAR). Less orbitals per rank will at some point slow down the GPU. It is wise to set then NSIM to a value that can be divided by that number so let's say NSIM=18 is good. Same for NBLOCK_FOCK. Try 36. But most likely this will not give you much. From the OUTCAR I can also see already that 95% of the time is in the fock routines.

I will wait for the profiles for more details.

Best,
Alex


chanaprom_cholsuk1
Newbie
Newbie
Posts: 7
Joined: Fri May 10, 2024 9:29 am

Re: Speeding up Γ-only HSE in vasp_gam 6.6.1 on Intel GPUs

#6 Post by chanaprom_cholsuk1 » Thu Nov 19, 2026 8:40 am

Dear Alex,

yes, one node has 112 cores. Please find attached the test run with the detailed profiles. I tested 3 cases with NELM = 15:
1. 8 nodes, 8 threads, NSIM = 18, N_BLOCK_FOCK = 36: it took 3 hr 55 mins.
2. 8 nodes, 12 threads, NSIM = 18, N_BLOCK_FOCK = 36: it took 3 hr 58 mins.
3. 16 nodes, 8 threads, NSIM = 9, N_BLOCK_FOCK = 18: it took 3 hr 58 mins.

Scaling is apparently not working: doubling the nodes gives no speed-up, and neither NSIM / NBLOCK_FOCK nor more threads per rank helped.

Thank you in advance for your suggestions.

Best,
Chanaprom

You do not have the required permissions to view the files attached to this post.

Post Reply