Combining MPI and OpenMP - Revision history

Wolloch at 08:54, 16 March 2026

2026-03-16T08:54:55Z

← Older revision		Revision as of 08:54, 16 March 2026
Line 6:		Line 6:

	#On nodes with many cores, e.g., 64 or more. On such nodes, the memory bandwidth and cache size per core may limit the parallel efficiency of VASP. These problems can be (partly) alleviated by the use of OpenMP.		#On nodes with many cores, e.g., 64 or more. On such nodes, the memory bandwidth and cache size per core may limit the parallel efficiency of VASP. These problems can be (partly) alleviated by the use of OpenMP.
	#When running the [[~~OpenACC_GPU_port_of_VASP\|OpenACC port~~ of VASP]] ~~on GPUs~~. Execution of VASP on GPUs is most efficient when using only a single MPI rank per GPU. Therefore, only a few MPI ranks are running on the CPU in most cases. It is helpful to run with multiple OpenMP threads per MPI rank to leverage the CPU's remaining computational power for those parts of VASP that still run on the CPU side.		#When running the [[GPU ports of VASP]]. Execution of VASP on GPUs is most efficient when using only a single MPI rank per GPU. Therefore, only a few MPI ranks are running on the CPU in most cases. It is helpful to run with multiple OpenMP threads per MPI rank to leverage the CPU's remaining computational power for those parts of VASP that still run on the CPU side.
	{{NB\|important\|When running with a single OpenMP thread per MPI rank, there is no appreciable difference between a VASP run with an MPI+OpenMP executable and an MPI-only one. The inactive OpenMP constructs incur very little overhead. In that sense, no strong argument speaks against building VASP with OpenMP support per default.}}		{{NB\|important\|When running with a single OpenMP thread per MPI rank, there is no appreciable difference between a VASP run with an MPI+OpenMP executable and an MPI-only one. The inactive OpenMP constructs incur very little overhead. In that sense, no strong argument speaks against building VASP with OpenMP support per default.}}

Line 141:		Line 141:
	[[:Category:Parallelization\|Parallelization]],		[[:Category:Parallelization\|Parallelization]],
	[[Installing_VASP.6.X.X\|Installing VASP.6.X.X]],		[[Installing_VASP.6.X.X\|Installing VASP.6.X.X]],
	[[~~OpenACC_GPU_port_of_VASP\|OpenACC~~ GPU ~~Port~~ of VASP]]		[[GPU ports of VASP]]

Vaspmaster: /* Using IntelMPI */

2022-09-14T15:29:38Z

Using IntelMPI

Vaspmaster: /* Using OpenMPI */

2022-09-14T15:29:09Z

Using OpenMPI

← Older revision		Revision as of 15:29, 14 September 2022
Line 71:		Line 71:

	=== Using OpenMPI ===		=== Using OpenMPI ===
			Now start 8 MPI ranks (<code>-np 8</code>), with the following placement specification: 2 ranks/socket, assigning 4 subsequent cores to each rank (<code>--map-by ppr:2:socket:PE=4</code>), and bind them to their physical cores (<code>--bind-to core</code>):

	~~:Now start 8 MPI ranks (~~<~~code~~>-np 8~~</code>), with the following placement specification: 2 ranks/socket, assigning 4 subsequent cores to each rank (<code>~~--map-by ppr:2:socket:PE=4~~</code>), and bind them to their physical cores (<code>~~--bind-to core</~~code~~>):		<pre>mpirun -np 8 --map-by ppr:2:socket:PE=4 --bind-to core <your-vasp-executable></pre>

	:~~<pre>mpirun -np 8 --map-by ppr:2:socket:PE=4 --bind-to core <your-vasp-executable></pre>~~		Or all of the above wrapped into a single command:

	:~~Or all of the above wrapped into a single command~~:		<pre>mpirun -np 8 --map-by ppr:2:socket:PE=4 --bind-to core \
			-x OMP_NUM_THREADS=4 -x OMP_STACKSIZE=512m \
			-x OMP_PLACES=cores -x OMP_PROC_BIND=close \
			--report-bindings <your-vasp-executable>
			</pre>

	:<~~pre~~>~~mpirun -np 8 --map-by ppr:2:socket:PE=4 --bind-to core \~~		where the <code>--report-bindings</code> is optional but a good idea to use at least once to check whether the rank and thread placement is as intended.
	~~:: -x OMP_NUM_THREADS=4 -x OMP_STACKSIZE=512m \~~
	~~:: -x OMP_PLACES=cores -x OMP_PROC_BIND=close \~~
	:: --report-bindings ~~<your-vasp-executable>~~
	::</~~pre~~>

	~~:where~~ the ~~<code>--report-bindings</code> is optional but a good idea to use at least once to check whether~~ the rank and ~~thread placement~~ is ~~as intended~~.		In our example, the above will assure that the OpenMP threads each MPI rank spawns reside on the same package/socket, and pins both the MPI ranks as well as the OpenMP threads to specific cores. This is crucial for performance.

	:In our example, the above will assure that the OpenMP threads each MPI rank spawns reside on the same package/socket, and pins both the MPI ranks as well as the OpenMP threads to specific cores. This is crucial for performance.
	=== Using IntelMPI ===		=== Using IntelMPI ===

Vaspmaster: /* For the OpenMP runtime */

2022-09-14T15:28:49Z

For the OpenMP runtime

← Older revision		Revision as of 15:28, 14 September 2022
Line 48:		Line 48:
	and that it should bind the threads to the physical cores, and put them onto cores that are as close as possible to the core that is running the corresponding MPI rank (and OpenMP master thread):		and that it should bind the threads to the physical cores, and put them onto cores that are as close as possible to the core that is running the corresponding MPI rank (and OpenMP master thread):

	<pre>export OMP_PLACES=cores ; export OMP_PROC_BIND=close</pre>		<pre>export OMP_PLACES=cores
			export OMP_PROC_BIND=close</pre>

	In addition to taking care of thread placement, it is often necessary to increase the size of the private stack of the OpenMP threads (to 256 or even 512 Mbytes), since the default is in many cases too small for VASP to run, and will cause segmentation faults:		In addition to taking care of thread placement, it is often necessary to increase the size of the private stack of the OpenMP threads (to 256 or even 512 Mbytes), since the default is in many cases too small for VASP to run, and will cause segmentation faults:

Vaspmaster: /* For the OpenMP runtime */

2022-09-14T15:28:32Z

For the OpenMP runtime

← Older revision		Revision as of 15:28, 14 September 2022
Line 42:		Line 42:
	=== For the OpenMP runtime ===		=== For the OpenMP runtime ===

	:Tell the OpenMP runtime it may spawn 4 threads per MPI rank:		Tell the OpenMP runtime it may spawn 4 threads per MPI rank:

	:<pre>export OMP_NUM_THREADS=4</pre>		<pre>export OMP_NUM_THREADS=4</pre>

	:and that it should bind the threads to the physical cores, and put them onto cores that are as close as possible to the core that is running the corresponding MPI rank (and OpenMP master thread):		and that it should bind the threads to the physical cores, and put them onto cores that are as close as possible to the core that is running the corresponding MPI rank (and OpenMP master thread):

	:<pre>export OMP_PLACES=cores ; export OMP_PROC_BIND=close</pre>		<pre>export OMP_PLACES=cores ; export OMP_PROC_BIND=close</pre>

	:In addition to taking care of thread placement, it is often necessary to increase the size of the private stack of the OpenMP threads (to 256 or even 512 Mbytes), since the default is in many cases too small for VASP to run, and will cause segmentation faults:		In addition to taking care of thread placement, it is often necessary to increase the size of the private stack of the OpenMP threads (to 256 or even 512 Mbytes), since the default is in many cases too small for VASP to run, and will cause segmentation faults:

	:<pre>export OMP_STACKSIZE=512m</pre>		<pre>export OMP_STACKSIZE=512m</pre>
	{{NB\|mind\|[https://software.intel.com/content/www/us/en/develop/documentation/cpp-compiler-developer-guide-and-reference/top/optimization-and-programming-guide/openmp-support/openmp-library-support/thread-affinity-interface-linux-and-windows.html The Intel OpenMP-runtime library (<tt>libiomp5.so</tt>) offers an alternative set of environment variables to control OpenMP-thread placement, stacksize ''etc''].\|:}}		{{NB\|mind\|[https://software.intel.com/content/www/us/en/develop/documentation/cpp-compiler-developer-guide-and-reference/top/optimization-and-programming-guide/openmp-support/openmp-library-support/thread-affinity-interface-linux-and-windows.html The Intel OpenMP-runtime library (<tt>libiomp5.so</tt>) offers an alternative set of environment variables to control OpenMP-thread placement, stacksize ''etc''].\|:}}
	<!--		<!--

Vaspmaster: /* For the OpenMP runtime */

2022-09-14T15:21:01Z

For the OpenMP runtime

← Older revision		Revision as of 15:21, 14 September 2022
Line 48:		Line 48:
	:and that it should bind the threads to the physical cores, and put them onto cores that are as close as possible to the core that is running the corresponding MPI rank (and OpenMP master thread):		:and that it should bind the threads to the physical cores, and put them onto cores that are as close as possible to the core that is running the corresponding MPI rank (and OpenMP master thread):

	:<pre>export OMP_PLACES=cores		:<pre>export OMP_PLACES=cores ; export OMP_PROC_BIND=close</pre>
	:: export OMP_PROC_BIND=close</pre>

	:In addition to taking care of thread placement, it is often necessary to increase the size of the private stack of the OpenMP threads (to 256 or even 512 Mbytes), since the default is in many cases too small for VASP to run, and will cause segmentation faults:		:In addition to taking care of thread placement, it is often necessary to increase the size of the private stack of the OpenMP threads (to 256 or even 512 Mbytes), since the default is in many cases too small for VASP to run, and will cause segmentation faults:

Huebsch: Huebsch moved page Construction:Combining MPI and OpenMP to Combining MPI and OpenMP

2022-05-25T09:22:01Z

Huebsch moved page Construction:Combining MPI and OpenMP to Combining MPI and OpenMP

← Older revision	Revision as of 09:22, 25 May 2022
(No difference)

Huebsch at 17:28, 12 April 2022

2022-04-12T17:28:08Z

← Older revision		Revision as of 17:28, 12 April 2022
Line 3:		Line 3:
	== When to use MPI + OpenMP ==		== When to use MPI + OpenMP ==

	When is it beneficial to run with multiple OpenMP threads per MPI rank? ~~In fact, there~~ are not so many cases, but we can discern at least two:		When is it beneficial to run with multiple OpenMP threads per MPI rank? There are not so many cases, but we can discern at least two:

	#On nodes with many cores ~~(''~~e.g.'' 64 or more). On such nodes, ~~the parallel efficiency of VASP may be limited by~~ the memory bandwidth and cache size per core. These problems can be (partly) alleviated by the use of OpenMP.		#On nodes with many cores, e.g., 64 or more. On such nodes, the memory bandwidth and cache size per core may limit the parallel efficiency of VASP. These problems can be (partly) alleviated by the use of OpenMP.
	#When running the [[OpenACC_GPU_port_of_VASP\|OpenACC port of VASP]] on GPUs. Execution of VASP on GPUs is most efficient when using only a single MPI rank per GPU. ~~In most cases~~, ~~this means one will have~~ only a few MPI ranks running on the CPU. To leverage the remaining computational power ~~of the CPU~~ for those parts of VASP that still run CPU-side ~~it is helpful to run with multiple OpenMP threads per MPI rank~~.		#When running the [[OpenACC_GPU_port_of_VASP\|OpenACC port of VASP]] on GPUs. Execution of VASP on GPUs is most efficient when using only a single MPI rank per GPU. Therefore, only a few MPI ranks are running on the CPU in most cases. It is helpful to run with multiple OpenMP threads per MPI rank to leverage the CPU's remaining computational power for those parts of VASP that still run on the CPU side.
	{{NB\|important\|When running with a single OpenMP thread per MPI rank there is no appreciable difference between a VASP run with ~~a hybrid~~ MPI+OpenMP executable and an MPI-only one. The inactive OpenMP constructs incur very little overhead. In that sense, ~~there is~~ no strong argument ~~that~~ speaks against building VASP with OpenMP support per default.}}		{{NB\|important\|When running with a single OpenMP thread per MPI rank, there is no appreciable difference between a VASP run with an MPI+OpenMP executable and an MPI-only one. The inactive OpenMP constructs incur very little overhead. In that sense, no strong argument speaks against building VASP with OpenMP support per default.}}

	== Compilation ==		== Compilation ==
Line 15:		Line 15:
	CPP_OPTIONS += -D_OPENMP		CPP_OPTIONS += -D_OPENMP

	In addition, you ~~will have~~ to add some compiler-specific options to [[Installing_VASP.6.X.X#Compiler_variables\|the command that invokes your Fortran compiler (and sometimes to the linker as well)]].		In addition, you need to add some compiler-specific options to [[Installing_VASP.6.X.X#Compiler_variables\|the command that invokes your Fortran compiler (and sometimes to the linker as well)]]. For instance, when using an Intel toolchain (ifort + Intel MPI), it is

	~~When~~ using an Intel toolchain (ifort + Intel MPI), ~~for instance:~~

	FC = mpiifort -qopenmp		FC = mpiifort -qopenmp
			{{NB\|important\| Base your {{FILE\|makefile.include}} file on one of the archetypical <tt>/arch/makefile.include.*_omp</tt> files that are provided with your VASP.6.X.X release.}}
	~~It is probably best to base~~ your {{FILE\|makefile.include}} file on one of the archetypical <tt>/arch/makefile.include.*_omp</tt> files that are provided with your VASP.6.X.X release.
	To adapt these to the particulars of your system (if necessary) please read the [[Installing_VASP.6.X.X\|instructions on the installation of VASP.6.X.X]].		To adapt these to the particulars of your system (if necessary) please read the [[Installing_VASP.6.X.X\|instructions on the installation of VASP.6.X.X]].
	{{NB\|~~important~~\|When you compile VASP with OpenMP support and you are '''not''' using the FFTs from the Intel-MKL library, you should [[Makefile.include#fftlib_.28recommended_when_using_OpenMP.29\|compile VASP with <tt>fftlib</tt>]]. Otherwise, the costs of (planning) the OpenMP-threaded FFTs will become prohibitively large at higher thread counts.}}		{{NB\|mind\|When you compile VASP with OpenMP support and you are '''not''' using the FFTs from the Intel-MKL library, you should [[Makefile.include#fftlib_.28recommended_when_using_OpenMP.29\|compile VASP with <tt>fftlib</tt>]]. Otherwise, the costs of (planning) the OpenMP-threaded FFTs will become prohibitively large at higher thread counts.}}
	<!--		<!--
	{{NB\|tip\|When compiling for Intel CPUs we strongly recommend using an all Intel toolchain (Intel compiler + Intel MPI + MKL libraries) since this will yield the best performance by far (and especially so for the hybrid MPI/OpenMP version of VASP). The aforementioned compilers and libraries are freely available in the form of [https://software.intel.com/content/www/us/en/develop/tools/oneapi/all-toolkits.html the Intel oneAPI base + HPC toolkits].}}		{{NB\|tip\|When compiling for Intel CPUs we strongly recommend using an all Intel toolchain (Intel compiler + Intel MPI + MKL libraries) since this will yield the best performance by far (and especially so for the hybrid MPI/OpenMP version of VASP). The aforementioned compilers and libraries are freely available in the form of [https://software.intel.com/content/www/us/en/develop/tools/oneapi/all-toolkits.html the Intel oneAPI base + HPC toolkits].}}
Line 30:		Line 27:
	== Running multiple OpenMP threads per MPI rank ==		== Running multiple OpenMP threads per MPI rank ==

	In principle, running VASP on ''n'' MPI-ranks with ''m'' OpenMP threads per rank is as simple as~~. Unfortunately the latter depends on the flavor of MPI one uses (below we will only discuss the use of OpenMPI and IntelMPI).~~		In principle, running VASP on ''n'' MPI ranks with ''m'' OpenMP threads per rank is as simple as:

	export OMP_NUM_THREADS=<m> ; mpirun -np <n> <your-vasp-executable>		export OMP_NUM_THREADS=<m> ; mpirun -np <n> <your-vasp-executable>

	~~To get good performance~~, ~~however, is often not that simple.~~		Here, the <tt>mpirun</tt> part of the command depends on the flavor of MPI one uses and has to be replaced appropriately. Below, we will only discuss the use of OpenMPI and IntelMPI.
	~~The key is to ensure that~~ the MPI ~~ranks,~~ and ~~the associated OpenMP threads they spawn~~, ~~are placed optimally onto~~ the ~~''physical'' cores~~ of ~~the node(s),~~ and ~~are pinned to these cores~~.

	As an example (a typical Intel Xeon-like architecture): Let us assume we plan to run on 2 nodes, each with 16 physical cores. These 16 cores per node are further divided into ~~two~~ ''packages'' (aka ''sockets'') of 8 cores each. The cores on a ~~package~~ share access to a block of memory and in addition, they may access the memory associated with the other package on their node via a so-called ''crossbar-switch''. The latter, however, comes at a (slight) performance penalty.		For proper performance, it is crucial to ensure that the MPI ranks, and the associated OpenMP threads they spawn, are placed optimally onto the ''physical'' cores of the node(s), and are pinned to these cores.
			As an example (for a typical Intel Xeon-like architecture): Let us assume we plan to run on 2 nodes, each with 16 physical cores. These 16 cores per node are further divided into 2 ''packages'' (aka ''sockets'') of 8 cores each. The cores on a socket share access to a block of memory and in addition, they may access the memory associated with the other package on their node via a so-called ''crossbar switch''. The latter, however, comes at a (slight) performance penalty.

	In the aforementioned situation, a possible placement of MPI ranks and OpenMP threads would for instance be the following: place 2 MPI ranks on each package (''i.e.'', 8 MPI ranks in total) and have each MPI rank spawn 4 OpenMP threads on the same package. These OpenMP threads will all have fast access to the memory associated with their package, and will not have to access memory through the crossbar switch.		In the aforementioned situation, a possible placement of MPI ranks and OpenMP threads would for instance be the following: place 2 MPI ranks on each package (''i.e.'', 8 MPI ranks in total) and have each MPI rank spawn 4 OpenMP threads on the same package. These OpenMP threads will all have fast access to the memory associated with their package, and will not have to access memory through the crossbar switch.
Line 63:		Line 60:
	export KMP_AFFINITY=verbose,granularity=fine,compact,1,0		export KMP_AFFINITY=verbose,granularity=fine,compact,1,0

	{{NB\|tip\|As far as we are aware, this level of fine-grained control over the thread placement is only available from Intel's OpenMP runtime. In light of this fact, and since ''oversubscribing'', i.e., starting more OpenMP-threads than there are physical cores on a node seldomly brings a (~~large~~) performance gain, we recommend disabling hyperthreading altogether.}}		{{NB\|tip\|As far as we are aware, this level of fine-grained control over the thread placement is only available from Intel's OpenMP runtime. In light of this fact, and since ''oversubscribing'', i.e., starting more OpenMP-threads than there are physical cores on a node seldomly brings a (significant) performance gain, we recommend disabling hyperthreading altogether.}}

	All of the above may be combined into a single command, as follows:		All of the above may be combined into a single command, as follows:

Huebsch at 08:40, 12 April 2022

2022-04-12T08:40:41Z

Vaspmaster at 14:55, 11 April 2022

2022-04-11T14:55:34Z

← Older revision		Revision as of 14:55, 11 April 2022
Line 1:		Line 1:
	VASP can be built ~~with~~ a combination of OpenMP threading and parallelization of MPI ranks. This is beneficial ~~for~~ some ~~calculations and requires passing the desired number of threads at the time of submission~~.		VASP can be built to use a combination of OpenMP threading and parallelization over MPI ranks. This is beneficial on some hardware.

	== When to use MPI + OpenMP ==		== When to use MPI + OpenMP ==

← Older revision		Revision as of 15:29, 14 September 2022
Line 89:		Line 89:
	=== Using IntelMPI ===		=== Using IntelMPI ===

	:Tell MPI to reserve a domain of <tt>OMP_NUM_THREADS</tt> cores for each rank		Tell MPI to reserve a domain of <tt>OMP_NUM_THREADS</tt> cores for each rank

	:<pre>export I_MPI_PIN_DOMAIN=omp</pre>		<pre>export I_MPI_PIN_DOMAIN=omp</pre>

	:and pin the MPI ranks to the cores		and pin the MPI ranks to the cores

	:<pre>export I_MPI_PIN=yes</pre>		<pre>export I_MPI_PIN=yes</pre>

	:Then start VASP on 8 MPI ranks		Then start VASP on 8 MPI ranks

	:<pre>mpirun -np 8 <your-vasp-executable></pre>		<pre>mpirun -np 8 <your-vasp-executable></pre>

	:In case one uses Intel MPI, things are fortunately a bit less involved. Distributing 8 MPI-ranks over 2 nodes with 16 physical cores each (2 sockets per node) allowing for 4 OpenMP threads per MPI-rank is as simple as:		In case one uses Intel MPI, things are fortunately a bit less involved. Distributing 8 MPI-ranks over 2 nodes with 16 physical cores each (2 sockets per node) allowing for 4 OpenMP threads per MPI-rank is as simple as:

	:<pre>mpirun -np 8 -genv I_MPI_PIN=yes -genv I_MPI_PIN_DOMAIN=omp -genv I_MPI_DEBUG=4</pre>		<pre>mpirun -np 8 -genv I_MPI_PIN=yes -genv I_MPI_PIN_DOMAIN=omp -genv I_MPI_DEBUG=4</pre>

	:Or all of the above wrapped up into a single command:		Or all of the above wrapped up into a single command:

	:<pre> mpirun -np 8 -genv I_MPI_PIN_DOMAIN=omp -genv I_MPI_PIN=yes -genv OMP_NUM_THREADS=4 -genv OMP_STACKSIZE=512m \		<pre> mpirun -np 8 -genv I_MPI_PIN_DOMAIN=omp -genv I_MPI_PIN=yes -genv OMP_NUM_THREADS=4 -genv OMP_STACKSIZE=512m \
	:: -genv OMP_PLACES=cores -genv OMP_PROC_BIND=close -genv I_MPI_DEBUG=4 <your-vasp-executable></pre>		-genv OMP_PLACES=cores -genv OMP_PROC_BIND=close -genv I_MPI_DEBUG=4 <your-vasp-executable></pre>

	:where the <code>-genv I_MPI_DEBUG=4</code> is optional but a good idea to use at least once to check whether the rank and thread placement is as intended.		where the <code>-genv I_MPI_DEBUG=4</code> is optional but a good idea to use at least once to check whether the rank and thread placement is as intended.

	:In our example, the above will assure that the OpenMP threads each MPI rank spawns reside on the same package/socket, and pins both the MPI ranks as well as the OpenMP threads to specific cores. This is crucial for performance.		In our example, the above will assure that the OpenMP threads each MPI rank spawns reside on the same package/socket, and pins both the MPI ranks as well as the OpenMP threads to specific cores. This is crucial for performance.

	== MPI versus MPI/OpenMP: the main difference ==		== MPI versus MPI/OpenMP: the main difference ==

← Older revision		Revision as of 08:40, 12 April 2022
Line 28:		Line 28:
	-->		-->

	== Running multiple OpenMP-threads per MPI-rank ==		== Running multiple OpenMP threads per MPI rank ==

	In principle, running VASP on ''n'' MPI-ranks with ''m'' OpenMP threads per rank is as simple as. Unfortunately the latter depends on the flavor of MPI one uses (below we will only discuss the use of OpenMPI and IntelMPI).		In principle, running VASP on ''n'' MPI-ranks with ''m'' OpenMP threads per rank is as simple as. Unfortunately the latter depends on the flavor of MPI one uses (below we will only discuss the use of OpenMPI and IntelMPI).
Line 45:		Line 45:
	=== For the OpenMP runtime ===		=== For the OpenMP runtime ===

	Tell the OpenMP runtime it may spawn 4 threads per MPI rank:		:Tell the OpenMP runtime it may spawn 4 threads per MPI rank:

	export OMP_NUM_THREADS=4		:<pre>export OMP_NUM_THREADS=4</pre>

	and that it should bind the threads to the physical cores, and put them onto cores that are as close as possible to the core that is running the corresponding MPI rank (and OpenMP master thread):		:and that it should bind the threads to the physical cores, and put them onto cores that are as close as possible to the core that is running the corresponding MPI rank (and OpenMP master thread):

	export OMP_PLACES=cores		:<pre>export OMP_PLACES=cores
	export OMP_PROC_BIND=close		:: export OMP_PROC_BIND=close</pre>

	In addition to taking care of thread placement, it is often necessary to increase the size of the private stack of the OpenMP threads (to 256 or even 512 Mbytes), since the default is in many cases too small for VASP to run, and will cause segmentation faults:		:In addition to taking care of thread placement, it is often necessary to increase the size of the private stack of the OpenMP threads (to 256 or even 512 Mbytes), since the default is in many cases too small for VASP to run, and will cause segmentation faults:

	export OMP_STACKSIZE=512m		:<pre>export OMP_STACKSIZE=512m</pre>
	{{NB\|mind\|[https://software.intel.com/content/www/us/en/develop/documentation/cpp-compiler-developer-guide-and-reference/top/optimization-and-programming-guide/openmp-support/openmp-library-support/thread-affinity-interface-linux-and-windows.html The Intel OpenMP-runtime library (<tt>libiomp5.so</tt>) offers an alternative set of environment variables to control OpenMP-thread placement, stacksize ''etc''].}}		{{NB\|mind\|[https://software.intel.com/content/www/us/en/develop/documentation/cpp-compiler-developer-guide-and-reference/top/optimization-and-programming-guide/openmp-support/openmp-library-support/thread-affinity-interface-linux-and-windows.html The Intel OpenMP-runtime library (<tt>libiomp5.so</tt>) offers an alternative set of environment variables to control OpenMP-thread placement, stacksize ''etc''].\|:}}
	<!--		<!--
	When your CPU supports ''hyperthreading'' (and if this is enabled in the BIOS) there are more ''logical'' cores than ''physical'' cores (typically a factor 2). In this case, one should make sure the threads are placed on consecutive physical cores instead of consecutive logical cores. When using Intel's OpenMP runtime library (<tt>libiomp5.so</tt>) this can be achieved by:		When your CPU supports ''hyperthreading'' (and if this is enabled in the BIOS) there are more ''logical'' cores than ''physical'' cores (typically a factor 2). In this case, one should make sure the threads are placed on consecutive physical cores instead of consecutive logical cores. When using Intel's OpenMP runtime library (<tt>libiomp5.so</tt>) this can be achieved by:
Line 75:		Line 75:
	=== Using OpenMPI ===		=== Using OpenMPI ===

	Now start 8 MPI ranks (<code>-np 8</code>), with the following placement specification: 2 ranks/socket, assigning 4 subsequent cores to each rank (<code>--map-by ppr:2:socket:PE=4</code>), and bind them to their physical cores (<code>--bind-to core</code>):		:Now start 8 MPI ranks (<code>-np 8</code>), with the following placement specification: 2 ranks/socket, assigning 4 subsequent cores to each rank (<code>--map-by ppr:2:socket:PE=4</code>), and bind them to their physical cores (<code>--bind-to core</code>):

	mpirun -np 8 --map-by ppr:2:socket:PE=4 --bind-to core <your-vasp-executable>		:<pre>mpirun -np 8 --map-by ppr:2:socket:PE=4 --bind-to core <your-vasp-executable></pre>

	Or all of the above wrapped into a single command:		:Or all of the above wrapped into a single command:

	mpirun -np 8 --map-by ppr:2:socket:PE=4 --bind-to core \		:<pre>mpirun -np 8 --map-by ppr:2:socket:PE=4 --bind-to core \
	-x OMP_NUM_THREADS=4 -x OMP_STACKSIZE=512m \		:: -x OMP_NUM_THREADS=4 -x OMP_STACKSIZE=512m \
	-x OMP_PLACES=cores -x OMP_PROC_BIND=close \		:: -x OMP_PLACES=cores -x OMP_PROC_BIND=close \
	--report-bindings <your-vasp-executable>		:: --report-bindings <your-vasp-executable>
			::</pre>

	where the <code>--report-bindings</code> is optional but a good idea to use at least once to check whether the rank and thread placement is as intended.		:where the <code>--report-bindings</code> is optional but a good idea to use at least once to check whether the rank and thread placement is as intended.

	In our example, the above will assure that the OpenMP threads each MPI rank spawns reside on the same package/socket, and pins both the MPI ranks as well as the OpenMP threads to specific cores. This is crucial for performance.		:In our example, the above will assure that the OpenMP threads each MPI rank spawns reside on the same package/socket, and pins both the MPI ranks as well as the OpenMP threads to specific cores. This is crucial for performance.
	=== Using IntelMPI ===		=== Using IntelMPI ===

	Tell MPI to reserve a domain of <tt>OMP_NUM_THREADS</tt> cores for each rank		:Tell MPI to reserve a domain of <tt>OMP_NUM_THREADS</tt> cores for each rank

	export I_MPI_PIN_DOMAIN=omp		:<pre>export I_MPI_PIN_DOMAIN=omp</pre>

	and pin the MPI ranks to the cores		:and pin the MPI ranks to the cores

	export I_MPI_PIN=yes		:<pre>export I_MPI_PIN=yes</pre>

	Then start VASP on 8 MPI ranks		:Then start VASP on 8 MPI ranks

	mpirun -np 8 <your-vasp-executable>		:<pre>mpirun -np 8 <your-vasp-executable></pre>

	In case one uses Intel MPI things are fortunately a bit less involved. Distributing 8 MPI-ranks over 2 nodes with 16 physical cores each (2 sockets per node) allowing for 4 OpenMP threads per MPI-rank is as simple as:		:In case one uses Intel MPI, things are fortunately a bit less involved. Distributing 8 MPI-ranks over 2 nodes with 16 physical cores each (2 sockets per node) allowing for 4 OpenMP threads per MPI-rank is as simple as:

	mpirun -np 8 -genv I_MPI_PIN=yes -genv I_MPI_PIN_DOMAIN=omp -genv I_MPI_DEBUG=4		:<pre>mpirun -np 8 -genv I_MPI_PIN=yes -genv I_MPI_PIN_DOMAIN=omp -genv I_MPI_DEBUG=4</pre>

	Or all of the above wrapped up into a single command:		:Or all of the above wrapped up into a single command:

	mpirun -np 8 -genv I_MPI_PIN_DOMAIN=omp -genv I_MPI_PIN=yes \		:<pre> mpirun -np 8 -genv I_MPI_PIN_DOMAIN=omp -genv I_MPI_PIN=yes -genv OMP_NUM_THREADS=4 -genv OMP_STACKSIZE=512m \
	-genv OMP_NUM_THREADS=4 -genv OMP_STACKSIZE=512m \		:: -genv OMP_PLACES=cores -genv OMP_PROC_BIND=close -genv I_MPI_DEBUG=4 <your-vasp-executable></pre>
	-genv OMP_PLACES=cores -genv OMP_PROC_BIND=close \
	-genv I_MPI_DEBUG=4 <your-vasp-executable>

	where the <code>-genv I_MPI_DEBUG=4</code> is optional but a good idea to use at least once to check whether the rank and thread placement is as intended.		:where the <code>-genv I_MPI_DEBUG=4</code> is optional but a good idea to use at least once to check whether the rank and thread placement is as intended.

	In our example, the above will assure that the OpenMP threads each MPI rank spawns reside on the same package/socket, and pins both the MPI ranks as well as the OpenMP threads to specific cores. This is crucial for performance.		:In our example, the above will assure that the OpenMP threads each MPI rank spawns reside on the same package/socket, and pins both the MPI ranks as well as the OpenMP threads to specific cores. This is crucial for performance.

	== MPI versus MPI/OpenMP: the main difference ==		== MPI versus MPI/OpenMP: the main difference ==
Line 143:		Line 142:
	== Related tags and articles ==		== Related tags and articles ==

			[[:Category:Parallelization\|Parallelization]],
	[[Installing_VASP.6.X.X\|Installing VASP.6.X.X]],		[[Installing_VASP.6.X.X\|Installing VASP.6.X.X]],
	[[OpenACC_GPU_port_of_VASP\|OpenACC GPU Port of VASP]]		[[OpenACC_GPU_port_of_VASP\|OpenACC GPU Port of VASP]]


	----		----

	[[Category:VASP]][[Category:Installation]][[Category:Parallelization]]		[[Category:VASP]][[Category:Installation]][[Category:Parallelization]]