GPU ports of VASP - Revision history

Hampel: /* AMD */

2026-04-13T07:44:59Z

AMD

← Older revision		Revision as of 07:44, 13 April 2026
Line 148:		Line 148:
	* Currently VASP 6.6.0 cannot use shared memory as available on MI300A cards. Make sure that the feature is disabled at runtime via <code>export HSA_XNACK=0</code>		* Currently VASP 6.6.0 cannot use shared memory as available on MI300A cards. Make sure that the feature is disabled at runtime via <code>export HSA_XNACK=0</code>
	* To run on multiple AMD GPUs each MPI rank should only see its own GPU via <code>ROCR_VISIBLE_DEVICES</code> (will be fixed in future releases). To do so write a small wrapper script that is put in front of the VASP executable in your SLURM / PBS command:		* To run on multiple AMD GPUs each MPI rank should only see its own GPU via <code>ROCR_VISIBLE_DEVICES</code> (will be fixed in future releases). To do so write a small wrapper script that is put in front of the VASP executable in your SLURM / PBS command:
	{{CB\| #!/bin/bash		{{CB\|#!/bin/bash
			myGPU{{=}}${SLURM_LOCALID}
	myGPU{{=}}${SLURM_LOCALID}		# for PBS use this:
	# for PBS use this:		#myGPU{{=}}${PMI_RANK}
	#myGPU{{=}}${PMI_RANK}		export ROCR_VISIBLE_DEVICES{{=}}${myGPU}
	export ROCR_VISIBLE_DEVICES{{=}}${myGPU}		exec $*\|
	exec $*\|:}}		:}}

	===== Intel =====		===== Intel =====

Wolloch: /* AMD */

2026-04-13T07:39:27Z

AMD

← Older revision		Revision as of 07:39, 13 April 2026
Line 148:		Line 148:
	* Currently VASP 6.6.0 cannot use shared memory as available on MI300A cards. Make sure that the feature is disabled at runtime via <code>export HSA_XNACK=0</code>		* Currently VASP 6.6.0 cannot use shared memory as available on MI300A cards. Make sure that the feature is disabled at runtime via <code>export HSA_XNACK=0</code>
	* To run on multiple AMD GPUs each MPI rank should only see its own GPU via <code>ROCR_VISIBLE_DEVICES</code> (will be fixed in future releases). To do so write a small wrapper script that is put in front of the VASP executable in your SLURM / PBS command:		* To run on multiple AMD GPUs each MPI rank should only see its own GPU via <code>ROCR_VISIBLE_DEVICES</code> (will be fixed in future releases). To do so write a small wrapper script that is put in front of the VASP executable in your SLURM / PBS command:
			{{CB\| #!/bin/bash
	#!/bin/bash

	myGPU=${SLURM_LOCALID}		myGPU{{=}}${SLURM_LOCALID}
	# for PBS use this:		# for PBS use this:
	#myGPU=${PMI_RANK}		#myGPU{{=}}${PMI_RANK}
	export ROCR_VISIBLE_DEVICES=${myGPU}		export ROCR_VISIBLE_DEVICES{{=}}${myGPU}
	exec $*		exec $*\|:}}

	===== Intel =====		===== Intel =====

Hampel: /* AMD */

2026-04-13T07:30:29Z

AMD

← Older revision		Revision as of 07:30, 13 April 2026
Line 156:		Line 156:
	export ROCR_VISIBLE_DEVICES=${myGPU}		export ROCR_VISIBLE_DEVICES=${myGPU}
	exec $*		exec $*


	===== Intel =====		===== Intel =====

Hampel: /* OpenMP Offloading */

2026-04-13T07:30:21Z

OpenMP Offloading

← Older revision		Revision as of 07:30, 13 April 2026
Line 143:		Line 143:
	* Both MKL and ROCm can leverage CPU threads in library calls that run on the GPU. Make sure you have enough threads (4-8) available.		* Both MKL and ROCm can leverage CPU threads in library calls that run on the GPU. Make sure you have enough threads (4-8) available.

	AMD		===== AMD =====
	* The cray compilers uses normal OpenMP threads that are set via: <code>OMP_NUM_THREADS=8</code>		* The cray compilers uses normal OpenMP threads that are set via: <code>OMP_NUM_THREADS=8</code>
	* By default threads spin at 100% load in the cray omp runtime. You can safely disable this to give more headroom to ROCm helper threads by setting <code>export OMP_WAIT_POLICY=PASSIVE</code>		* By default threads spin at 100% load in the cray omp runtime. You can safely disable this to give more headroom to ROCm helper threads by setting <code>export OMP_WAIT_POLICY=PASSIVE</code>
Line 158:		Line 158:


	Intel		===== Intel =====
	* For Intel MKL, <code>MKL_NUM_THREADS</code> might be set. If it is too small, it can hurt performance, The savest way is to <code>unset MKL_NUM_THREADS</code> so that it defaults to <code>OMP_NUM_THREADS</code>, and set that.		* For Intel MKL, <code>MKL_NUM_THREADS</code> might be set. If it is too small, it can hurt performance, The savest way is to <code>unset MKL_NUM_THREADS</code> so that it defaults to <code>OMP_NUM_THREADS</code>, and set that.


	== Credits ==		== Credits ==

Hampel: /* OpenMP Offloading */

2026-04-13T07:29:49Z

OpenMP Offloading

@@ Line 142: / Line 142: @@
 * For GPU offloading, the OMP_STACKSIZE needs to be large. Set: <code>OMP_STACKSIZE=2048m</code>
 * Both MKL and ROCm can leverage CPU threads in library calls that run on the GPU. Make sure you have enough threads (4-8) available.
-** For AMD, the normal OpenMP threads are used: <code>OMP_NUM_THREADS=8</code>
-** For Intel MKL, <code>MKL_NUM_THREADS</code> might be set. If it is too small, it can hurt performance, The savest way is to <code>unset MKL_NUM_THREADS</code> so that it defaults to <code>OMP_NUM_THREADS</code>, and set that.
+AMD
 == Credits ==

Wolloch: Wolloch moved page OpenACC GPU port of VASP to GPU ports of VASP

2026-03-16T10:52:12Z

Wolloch moved page OpenACC GPU port of VASP to GPU ports of VASP

← Older revision	Revision as of 10:52, 16 March 2026
(No difference)

Wolloch at 10:50, 16 March 2026

2026-03-16T10:50:14Z

Show changes

Csheldon at 07:30, 28 July 2025

2025-07-28T07:30:05Z

← Older revision		Revision as of 07:30, 28 July 2025
Line 12:		Line 12:
	* To compile the OpenACC version of VASP you need a recent version of the [https://developer.nvidia.com/hpc-sdk NVIDIA HPC-SDK] (>=21.2).		* To compile the OpenACC version of VASP you need a recent version of the [https://developer.nvidia.com/hpc-sdk NVIDIA HPC-SDK] (>=21.2).
	:In principle, any compiler that supports at least OpenACC standard 2.6 should do the trick, but we have only tried and tested the aforementioned ones.		:In principle, any compiler that supports at least OpenACC standard 2.6 should do the trick, but we have only tried and tested the aforementioned ones.
	~~:'''N.B.''':~~ the NVIDIA HPC-SDK versions 22.1 and 22.2 have a serious bug that prohibits the execution of the OpenACC version in conjunction with OpenMP-threading. When using these compiler versions you should compile without OpenMP support. This bug is fixed as of NVIDIA HPC-SDK version 22.3.		{{NB\|warning\|the NVIDIA HPC-SDK versions 22.1 and 22.2 have a serious bug that prohibits the execution of the OpenACC version in conjunction with OpenMP-threading. When using these compiler versions you should compile without OpenMP support. This bug is fixed as of NVIDIA HPC-SDK version 22.3.\|:}}

	<u>''Libraries''</u>		<u>''Libraries''</u>
Line 29:		Line 29:
	* NVIDIA datacenter GPUs: P100 (Pascal), V100 (Volta), and A100 (Ampere).		* NVIDIA datacenter GPUs: P100 (Pascal), V100 (Volta), and A100 (Ampere).
	* NVIDIA Quadro GPUs: GP100 (Pascal), and GV100 (Volta).		* NVIDIA Quadro GPUs: GP100 (Pascal), and GV100 (Volta).
			{{NB\|important\|Running VASP on other NVIDIA GPUs (e.g. "gaming" hardware) is technically possible but not advisable: these GPUs are not well suited since they do not offer fast double-precision floating-point arithmetic (FP64) performance and in general have smaller memories without error correction code (ECC) capabilities.\|:}}
	~~'''N.B.''':~~ Running VASP on other NVIDIA GPUs (e.g. "gaming" hardware) is technically possible but not advisable: these GPUs are not well suited since they do not offer fast double-precision floating-point arithmetic (FP64) performance and in general have smaller memories without error correction code (ECC) capabilities.

	== Building ==		== Building ==
Line 54:		Line 53:
	Use OpenMP threads in addition to MPI ranks to leverage more of the available CPU power. The OpenACC version is currently limited to the use of 1 MPI-rank/GPU, which means that potentially quite a bit of CPU power remains unused. Since there are still parts of the code that run CPU-side it can be beneficial to allow for the use of multiple OpenMP threads per MPI rank:		Use OpenMP threads in addition to MPI ranks to leverage more of the available CPU power. The OpenACC version is currently limited to the use of 1 MPI-rank/GPU, which means that potentially quite a bit of CPU power remains unused. Since there are still parts of the code that run CPU-side it can be beneficial to allow for the use of multiple OpenMP threads per MPI rank:
	* To see how to build VASP with OpenACC- ''and'' OpenMP-support have a look at the [[Makefile.include.nvhpc ompi mkl omp acc\|makefile.include.nvhpc_ompi_mkl_omp_acc]] file.		* To see how to build VASP with OpenACC- ''and'' OpenMP-support have a look at the [[Makefile.include.nvhpc ompi mkl omp acc\|makefile.include.nvhpc_ompi_mkl_omp_acc]] file.
	~~:'''N.B.''': here~~ we link against Intel's MKL library for CPU-sided FFTW, BLAS, LAPACK, and scaLAPACK calls and the Intel OpenMP runtime library (<tt>libiomp5.so</tt>). This is strongly recommended when compiling for Intel CPUs, especially when using multiple threads. To ensure that MKL uses the Intel OpenMP runtime library you need to set an environment variable, either by:		{{NB\|important\|Here we link against Intel's MKL library for CPU-sided FFTW, BLAS, LAPACK, and scaLAPACK calls and the Intel OpenMP runtime library (<tt>libiomp5.so</tt>). This is strongly recommended when compiling for Intel CPUs, especially when using multiple threads. To ensure that MKL uses the Intel OpenMP runtime library you need to set an environment variable, either by:
	:<pre>export MKL_THREADING_LAYER=INTEL</pre>		:<pre>export MKL_THREADING_LAYER=INTEL</pre>
	:or by adding:		:or by adding:
	:<pre>-x MKL_THREADING_LAYER=INTEL</pre>		:<pre>-x MKL_THREADING_LAYER=INTEL</pre>
	:as an option to your <code>mpirun</code> command.		:as an option to your <code>mpirun</code> command.}}
	* Correct [[Combining MPI and OpenMP\|placement and pinning of OpenMPI ranks and OpenMP threads onto the CPU cores]] can be a bit tricky, and depends on the particular flavor of MPI one uses.		* Correct [[Combining MPI and OpenMP\|placement and pinning of OpenMPI ranks and OpenMP threads onto the CPU cores]] can be a bit tricky, and depends on the particular flavor of MPI one uses.
	</li>		</li>
Line 65:		Line 64:
	* Set {{TAG\|KPAR}} to the number of GPUs (= MPI-ranks) you are going to use. This only makes sense, though, when the number of irreducible '''k'''-points in your calculation is more or less evenly dividable by {{TAG\|KPAR}}, otherwise the distribution of the work over the MPI-ranks will be strongly imbalanced. This means your options in choosing this parameter are somewhat limited.		* Set {{TAG\|KPAR}} to the number of GPUs (= MPI-ranks) you are going to use. This only makes sense, though, when the number of irreducible '''k'''-points in your calculation is more or less evenly dividable by {{TAG\|KPAR}}, otherwise the distribution of the work over the MPI-ranks will be strongly imbalanced. This means your options in choosing this parameter are somewhat limited.
	* {{TAG\|NSIM}} determines the number of bands that are optimized simultaneously in many of the electronic solvers (e.g RMM-DIIS and blocked-Davidson). As a rule, one should choose this parameter larger to get good performance on GPUs than one would for CPU-sided execution.		* {{TAG\|NSIM}} determines the number of bands that are optimized simultaneously in many of the electronic solvers (e.g RMM-DIIS and blocked-Davidson). As a rule, one should choose this parameter larger to get good performance on GPUs than one would for CPU-sided execution.
	~~'''N.B.''':~~ For optimal CPU-sided execution of VASP one would normally experiment with different settings for {{TAG\|NCORE}} as well. When running on GPUs anything different from {{TAG\|NCORE}}=1 will adversely affect performance, and VASP will automatically switch to {{TAG\|NCORE}}=1, even if otherwise specified in the {{FILE\|INCAR}} file.		{{NB\|warning\|For optimal CPU-sided execution of VASP one would normally experiment with different settings for {{TAG\|NCORE}} as well. When running on GPUs anything different from {{TAG\|NCORE}}{{=}}1 will adversely affect performance, and VASP will automatically switch to {{TAG\|NCORE}}{{=}}1, even if otherwise specified in the {{FILE\|INCAR}} file.}}
	</li>		</li>
	</ol>		</ol>

Wolloch: /* Requirements */

2025-04-30T11:44:18Z

Requirements

← Older revision		Revision as of 11:44, 30 April 2025
Line 17:		Line 17:
	* Numerical libraries: FFTW, BLAS, LAPACK, and scaLAPACK. In case you are using the [https://developer.nvidia.com/hpc-sdk NVIDIA HPC-SDK] the only numerical library you will have to install yourself is [http://www.fftw.org FFTW]. The latter three (BLAS, LAPACK, and scaLAPACK) are shipped with the SDK. Alternatively, you can link against an installation of [https://www.intel.com/content/www/us/en/developer/tools/oneapi/onemkl.html Intel's oneAPI MKL] library that provides all four.		* Numerical libraries: FFTW, BLAS, LAPACK, and scaLAPACK. In case you are using the [https://developer.nvidia.com/hpc-sdk NVIDIA HPC-SDK] the only numerical library you will have to install yourself is [http://www.fftw.org FFTW]. The latter three (BLAS, LAPACK, and scaLAPACK) are shipped with the SDK. Alternatively, you can link against an installation of [https://www.intel.com/content/www/us/en/developer/tools/oneapi/onemkl.html Intel's oneAPI MKL] library that provides all four.
	* [https://developer.nvidia.com/cuda-toolkit The NVIDIA CUDA Toolkit] (>=10.0). All necessary CUDA Toolkit components are shipped as part of the NVIDIA HPC-SDK.		* [https://developer.nvidia.com/cuda-toolkit The NVIDIA CUDA Toolkit] (>=10.0). All necessary CUDA Toolkit components are shipped as part of the NVIDIA HPC-SDK.
	* A CUDA-aware version of MPI. The OpenMPI installations that are packaged with the NVIDIA HPC-SDK are CUDA-aware.		* A CUDA-aware version of MPI. The OpenMPI installations that are packaged with the NVIDIA HPC-SDK are CUDA-aware. MPICH will also work on Cray machines, but the precompiler flag [[Precompiler_options#-DCRAY_MPICH\|-DCRAY_MPICH]] has to be added to the [[Makefile.include#NVIDIA_HPC-SDK_for_CPU_and_GPU\|makefile.include]].
	* [https://developer.nvidia.com/nccl The NVIDIA Collective Communications Library (NCCL)] (>=2.7.8). This library is not a strict requirement but its use is highly recommended for performance reasons. Suitable installations of NCCL are shipped as part of the NVIDIA HPC-SDK.		* [https://developer.nvidia.com/nccl The NVIDIA Collective Communications Library (NCCL)] (>=2.7.8). This library is not a strict requirement but its use is highly recommended for performance reasons. Suitable installations of NCCL are shipped as part of the NVIDIA HPC-SDK.

Huebsch at 08:55, 21 February 2024

2024-02-21T08:55:38Z

← Older revision		Revision as of 08:55, 21 February 2024
Line 86:		Line 86:
	----		----

	[[Category:VASP]][[Category:Installation]][[Category:Performance]][[Category:GPU]][[Category:Parallelization~~]][[Category:VASP6~~]]		[[Category:VASP]][[Category:Installation]][[Category:Performance]][[Category:GPU]][[Category:Parallelization]]