Vasp on H100

[tienanh2828]

Hi all,

I am facing an issue when building vasp for GPU.
Our system administrator has attempted to build the GPU version of VASP 6.4.2 using CUDA 11.8 (minimum driver required for H100)
VASP compiles successfully, but the GPU is not detected at runtime.
With CUDA 12.2, The VASP build fails during compilation, preventing GPU usage.

Is VASP 6.4.2 officially compatible with the NVIDIA H100? Or is there any VASP version currently available that is confirmed to work correctly on H100 GPUs?
Which CUDA toolkit versions and compiler configurations do you recommend for building VASP with GPU support on H100 systems?
Are there any known patches, flags, or makefile/include settings necessary for proper GPU deêtction?

Thank you so much.

[ahampel]

Hi tienanh,

thank you for reaching out to us on the official VASP forum.

There is no limitation that I am aware of using cuda 11.8 oder cuda 12.2 with H100,H200 cards on VASP 6.4.2 . The H100 card requires compute capability 9.0 (https://developer.nvidia.com/cuda-gpus) , which is supported by cuda 11.8 and cuda 12.0 and newer. If anyone else knows of an issue he/she should write here as well.

To assess what is going wrong here I would need your full makefile.include you used to build and also the exact compiler versions you used, i.e. maybe the output of module list. Important are the choices you made here:

Code: Select all

CC          = mpicc  -acc -gpu=cc89,cuda11.8 -mp 
FC          = mpif90 -acc -gpu=cc89,cuda11.8 -mp 
FCL         = mpif90 -acc -gpu=cc89,cuda11.8 -mp -c++libs 

If you are able to compile on the node that has the GPU you can always set -gpu=native to avoid choosing anything wrong here. And make sure to add CPP_OPTIONS: -DACC_OFFLOAD -DNVCUDA at least. Also please share your build log where the compiler error occurs for me to assess.

Best,
Alex

[tienanh2828]

Hi Alex,

I can’t access the Makefile directly, but based on your recommendation, our system administrator has successfully built VASP 6.4.2 for the H100 GPUs. However, the performance is currently worse than when running on the V100 GPUs.
He suspects that the issue might be related to the older CUDA 11.8 driver and plans to rebuild using CUDA 12.2 or a newer version to see if that improves performance.
Do you have any suggestions or experiences with optimizing VASP performance on H100 GPUs?

Thanks,
Tien-Anh

[ahampel]

Hi Tien-Anh,

without specifics about the job your user is running this is hard to say. But in general I found that the H100/H200 are extremely fast, and they should easily outperform a V100. To test this I encourage to first run a job as a benchmark on a single GPU with 1 mpi rank to eliminate any problem due to MPI communications. Just to assess the speed of the card. The user should take a sizeable job that can run 10 scf iterations of a normal dft calculation in a few minutes and compare performance (LOOP and LOOP+ timers). There are some tips here: https://www.vasp.at/wiki/Optimizing_the_parallelization . Then go to 2 mpi ranks etc. I also tried to force on newer cards an older compute capability generated executable and did find drastic performance degradation. That said I never tried on the H100 specifically and the user should try a newer cuda version for sure to rule this out. Other than that we do not have any specific tuning for certain nvidia cards.

Best,
Alex

[tienanh2828]

Hi Alex,

Finally, I can now run VASP on H100.
I also tested the performance using a single GPU with 1 MPI rank as you recommended. But the speed is still very similar to the V100.

Thanks,
Tien Anh

[ahampel]

Would you mind turning on the option -DPROFILING for both VASP executables and post the OUTCAR file of each run (V100 and H100) here? I can have a look at it. Maybe I can then understand better. From my point of view the H100 should be much faster. I also attached a benchmark job we prepared (input files and one OUTCAR file for an H100 for reference).

Best,
Alex