Linking to math libraries on AMD CPU and NVIDIA GPU

[Zhiyuan Yin]

Hi,

I am building VASP 6.5.1 on an AMD EPYC 7b12 CPU with NVIDIA V100 SXM2 accelerators, and I am currently evaluating which CPU-side math library has better performance.

Is it supported to link NVHPC-compiled VASP GPU builds against AOCL (BLIS/libFLAME/ScaLAPACK) on EPYC? (i.e. makefile.include.nvhpc_ompi_aocl_omp_acc)

Are there any known issues with OpenMP runtime conflicts between AOCL and NVHPC?

Or should I link against external AOCL compiled by nvhpc?

In terms of stability and performance, is MKL still the recommended option even on AMD EPYC CPUs with NVIDIA GPUs? Is there any measurable difference in GPU-bound workloads between MKL vs AOCL vs open libraries + internal FFTLIB?

Regards,
Zhiyuan

[ahampel]

Hi Zhiyuan,

Let me start by answering your last question. I would still say that the Intel MKL lapack, and especially the fft libraries are very fast for the CPU target compiled VASP version. However, the aocl build chain including blis, libflame, amdfftw etc. is also very performant by now. This might depend a bit on the workload at hand. Across the board I found Intel MKL builds to perform a few percent faster, especially when MPI comm is irrelevant with smallish jobs. But also gnu builds with properly compiled openblas can be okay on amd cpus.

I don't see a reason why you could not build against blis, libflame etc as lapack/scalapack backend for the CPUs when building with nvhpc. But I do not think we have a makefile for this. At least I do not see the makefile.include you are mentioning. In principle for standard DFT calculations all relevant lapack + fft calls are done on the GPUs. However, there might be code features that are not ported. For that it is best to switch to a full CPU version anyway. So there should be no difference in performance for the OpenACC GPU build regarding the CPU libs used. For all intents you can just use whatever is available and compatible to your nvhpc modules. We typically link against intel mkl, but aocl libs should work as well. Let me know if there are problems for specific versions and I can have a look.

Best,
Alex