Speeding up Γ-only HSE in vasp_gam 6.6.1 on Intel GPUs
Dear VASP team,
I'm running HSE06 relaxations in bilayers (756 atoms, 15 A vacuum, gamma-only version 6.6.1, 1873 occupied bands) with the OpenMP-offload GPU port on 4× Intel Max 1550 per node, 1 rank per tile, 8 OpenMP threads.
Time per electronic step is fastest at 8 nodes (64 ranks) and gets slower at 12, 16, and more nodes. A 6.4.3 CPU run with > 1000 cores is faster than the best GPU setup.
I therefore have 2 quick questions:
1. Is this saturation in parallel scaling expected for HSE calculations, given that MPI parallelization is limited to bands (NCORE=1 and a single k-point)? In my tests, GPU scaling appears to saturate much earlier than CPU scaling.
2. Is there a recommended setting or algorithm for HSE on many GPUs? I found that changing NSIM or NBLOCK_FOCK made no difference.
Key tags are ENCUT = 500, LREAL = Auto, AEXX = 0.25, HFSCREEN = 0.2, ALGO = All, PRECFOCK = Fast.
Thank you in advance for the advice.
Best regards,
Chanaprom