Category:Parallelization - Revision history

Csheldon at 14:23, 24 March 2026

2026-03-24T14:23:10Z

← Older revision		Revision as of 14:23, 24 March 2026
Line 54:		Line 54:
	<!--; Patches-->		<!--; Patches-->
	{{NB\|warning\|The terminology of nodes, cores, CPUs, threads, etc. is not universal. For instance, some refer to a single core as a CPU, others refer to an entire node as a CPU.}}		{{NB\|warning\|The terminology of nodes, cores, CPUs, threads, etc. is not universal. For instance, some refer to a single core as a CPU, others refer to an entire node as a CPU.}}
	{{NB\|tip\|There is a lecture on ~~[https~~:~~//youtu.be/KzIuL_e0zz8~~ high-performance computing (HPC)] in VASP available on our YouTube channel.}}		{{NB\|tip\|There is a lecture on {{Video\|hpc:martin:2022\|high-performance computing (HPC)}} in VASP available on our YouTube channel.}}

	==How to==		==How to==

Wolloch at 09:03, 16 March 2026

2026-03-16T09:03:40Z

← Older revision		Revision as of 09:03, 16 March 2026
Line 11:		Line 11:
	</math>		</math>

	In addition to the parallelization using MPI, VASP can make use of [[Combining MPI and OpenMP\|OpenMP threading]] and/or [[~~OpenACC_GPU_port_of_VASP~~\|~~OpenACC (for the GPU port)~~]]. Note that running on multiple OpenMP threads and/or GPUs switches off the {{TAG\|NCORE}} parallelization.		In addition to the parallelization using MPI, VASP can make use of [[Combining MPI and OpenMP\|OpenMP threading]] and/or [[GPU_ports_of_VASP\|offloading to GPUs]]. Note that running on multiple OpenMP threads and/or GPUs switches off the {{TAG\|NCORE}} parallelization.

	===Communication patterns===		===Communication patterns===
Line 69:		Line 69:

	===OpenMP/OpenACC===		===OpenMP/OpenACC===
	Both [[Combining MPI and OpenMP\|OpenMP]] and [[~~OpenACC_GPU_port_of_VASP~~\|~~OpenACC~~]] parallelize the FFTs and therefore disregard any conflicting specification of {{TAG\|NCORE}}.		Both [[Combining MPI and OpenMP\|OpenMP]] and [[GPU_ports_of_VASP\|offloading to GPUs]] parallelize the FFTs and therefore disregard any conflicting specification of {{TAG\|NCORE}}.
	When ~~combining these methods OpenACC~~ takes precedence but any code not ported to ~~OpenACC~~ benefits from the additional OpenMP threads.		When running any of the [[GPU ports of VASP]], the offloaded code path takes precedence, but any code not ported to GPU benefits from the additional OpenMP threads on the CPU.
	This approach is relevant because the recommended NVIDIA Collective Communications Library requires a single MPI rank per GPU.		This approach is relevant because the recommended NVIDIA Collective Communications Library requires a single MPI rank per GPU (as does RCCL for AMD).
	Learn more about the OpenMP and OpenACC parallelization in these sections		Learn more about the OpenMP and OpenACC parallelization in these sections
	* How to [[Combining MPI and OpenMP\|parallelize with multiple OpenMP threads per MPI rank]]		* How to [[Combining MPI and OpenMP\|parallelize with multiple OpenMP threads per MPI rank]]
	* How to [[~~OpenACC_GPU_port_of_VASP~~\|run on GPUs]]		* How to [[GPU_ports_of_VASP\|run on GPUs]]

	==Additional parallelization options==		==Additional parallelization options==

Hampel: /* Optimizing the parallelization */

2025-10-16T08:49:17Z

Optimizing the parallelization

← Older revision		Revision as of 08:49, 16 October 2025
Line 61:		Line 61:
	Different algorithms ([[:Category:Electronic minimization\|density-functional theory]], [[:Category:Many-body perturbation theory\|many-body perturbation theory]], or [[:Category:Molecular dynamics\|molecular dynamics]]) require a separate optimization of the parallel setup.		Different algorithms ([[:Category:Electronic minimization\|density-functional theory]], [[:Category:Many-body perturbation theory\|many-body perturbation theory]], or [[:Category:Molecular dynamics\|molecular dynamics]]) require a separate optimization of the parallel setup.
	To obtain publishable results, many projects require performing many similar calculations, i.e., calculations with similar input and using the same algorithms.		To obtain publishable results, many projects require performing many similar calculations, i.e., calculations with similar input and using the same algorithms.
	Therefore, we recommend optimizing the parallelization to make the most of the available compute time.		Therefore, we recommend optimizing the parallelization to make the most of the available compute time. This optimization process depends greatly on the hardware and [[:Toolchains\|compiler toolchain]] on which you run your calculation. Hence, make sure to verify your setup when switching one of these.
	{{NB\|tip\|Run a few test calculations varying the parallel setup and use the optimal choice of parameters for the rest of the calculations.\|}}		{{NB\|tip\|Run a few test calculations varying the parallel setup and use the optimal choice of parameters for the rest of the calculations.\|}}
	For more detailed advice, check the following:		For more detailed advice, check the following:

Hampel: /* Terminology in high-performance computing (HPC) */

2025-10-16T08:45:27Z

Terminology in high-performance computing (HPC)

← Older revision		Revision as of 08:45, 16 October 2025
Line 44:		Line 44:
	; Core: When a ''CPU'' has the option to execute multiple tasks in parallel, we refer to this as a multi-core CPU. Because these computational cores are physically close, they typically exhibit a fast communication between them.		; Core: When a ''CPU'' has the option to execute multiple tasks in parallel, we refer to this as a multi-core CPU. Because these computational cores are physically close, they typically exhibit a fast communication between them.
	; Node: The node constitutes a physical entity consisting of one or more ''CPUs'' potentially accelerated by ''GPUs''. The communication between nodes is much slower compared to the communication within a single node.		; Node: The node constitutes a physical entity consisting of one or more ''CPUs'' potentially accelerated by ''GPUs''. The communication between nodes is much slower compared to the communication within a single node.
			; NUMA (Non-Uniform Memory Access) domain: these are distinct regions within a computer system where each core or group of cores has its own local memory that can be accessed faster than memory connected to other cores. In a NUMA architecture, memory access times vary depending on the proximity of the memory to the CPU core requesting it — local memory access is quicker, while remote access across domains incurs latency.
	; Socket: ''Processes'' communicate via sockets. Each socket corresponds to an endpoint in this communication.		; Socket: ''Processes'' communicate via sockets. Each socket corresponds to an endpoint in this communication.
	; Process: A process is a program executing on one or more ''cores''. Multiple processes may distribute work via communication. Each process may spawn multiple ''threads'' to execute their task.		; Process: A process is a program executing on one or more ''cores''. Multiple processes may distribute work via communication. Each process may spawn multiple ''threads'' to execute their task.

Csheldon: /* Terminology in high-performance computing (HPC) */

2025-09-22T10:10:16Z

Terminology in high-performance computing (HPC)

← Older revision		Revision as of 10:10, 22 September 2025
Line 53:		Line 53:
	<!--; Patches-->		<!--; Patches-->
	{{NB\|warning\|The terminology of nodes, cores, CPUs, threads, etc. is not universal. For instance, some refer to a single core as a CPU, others refer to an entire node as a CPU.}}		{{NB\|warning\|The terminology of nodes, cores, CPUs, threads, etc. is not universal. For instance, some refer to a single core as a CPU, others refer to an entire node as a CPU.}}
			{{NB\|tip\|There is a lecture on [https://youtu.be/KzIuL_e0zz8 high-performance computing (HPC)] in VASP available on our YouTube channel.}}

	==How to==		==How to==

Huebsch: /* Theory */

2024-10-16T06:24:39Z

Theory

← Older revision		Revision as of 06:24, 16 October 2024
Line 11:		Line 11:
	</math>		</math>

	In addition to the parallelization using MPI, VASP can make use of [[Combining MPI and OpenMP\|OpenMP threading]] and/or [[OpenACC_GPU_port_of_VASP\|OpenACC (for the GPU port)]]. Note that running on multiple OpenMP threads and/or GPUs switches off the {{TAG\|NCORE}} parallelization.		In addition to the parallelization using MPI, VASP can make use of [[Combining MPI and OpenMP\|OpenMP threading]] and/or [[OpenACC_GPU_port_of_VASP\|OpenACC (for the GPU port)]]. Note that running on multiple OpenMP threads and/or GPUs switches off the {{TAG\|NCORE}} parallelization.

	===Communication patterns===		===Communication patterns===

Schlipf: /* MPI setup */

2022-07-26T07:56:18Z

MPI setup

← Older revision		Revision as of 07:56, 26 July 2022
Line 22:		Line 22:
	===MPI setup===		===MPI setup===

	The MPI setup determines the placement of the ranks onto the nodes. VASP assumes the ranks first fill up a node before the next node is occupied. As an example when running with 8 ranks on two nodes, VASP expects rank ~~1–4~~ on node 1 and rank ~~5–8~~ on node 2. If the ranks are placed differently, communication between the nodes occurs for every parallel FFT. Because FFTs are essential to VASP's speed, this deteriorates the performance of the calculation. A manifestation is an increase in computing time when the number of nodes is increased from 1 to 2. If {{TAG\|NCORE}} is not used this issue is less severe but will still reduce the performance.		[[File:good_process_binding.png\|x300px\|All ranks associated with a particular band group are on the same node. All communication for the FFTs is intranode.]]
			[[File:bad_process_binding.png\|x300px\|Ranks of some band groups extend over multiple nodes. Every FFT requires internode communication.]]

			The MPI setup determines the placement of the ranks onto the nodes. VASP assumes the ranks first fill up a node before the next node is occupied. As an example when running with 20 ranks on two nodes, VASP expects rank 1–10 on node 1 and rank 11–20 on node 2. If the ranks are placed differently, communication between the nodes occurs for every parallel FFT. Because FFTs are essential to VASP's speed, this deteriorates the performance of the calculation. A manifestation is an increase in computing time when the number of nodes is increased from 1 to 2. If {{TAG\|NCORE}} is not used this issue is less severe but will still reduce the performance.

	To address this issue, please check the setup of the MPI library and the submitted job script. It is usually possible to overwrite the placement by setting environment variables or command-line arguments. When in doubt, contact the HPC administration of your machine to investigate the behavior.		To address this issue, please check the setup of the MPI library and the submitted job script. It is usually possible to overwrite the placement by setting environment variables or command-line arguments. When in doubt, contact the HPC administration of your machine to investigate the behavior.

Schlipf: /* Communication patterns */

2022-07-26T07:48:51Z

Communication patterns

← Older revision		Revision as of 07:48, 26 July 2022
Line 15:		Line 15:
	===Communication patterns===		===Communication patterns===

	The aforementioned parallelization levels directly map onto MPI communicators. Initially, VASP divides all available ranks into {{TAG\|IMAGES}} equally-sized images. Setting {{TAG\|KPAR}} splits each of these images further into groups of '''k''' points. Properties involving all '''k''' points (e.g. density, Fermi energy) require then a communication over these '''k'''-point groups. VASP divides the '''k'''-point group into ~~inter-~~band groups. Communication within the inter-band group occurs to evaluate properties involving two bands (e.g. Hamiltonian, orthonormalization). Therefore it is more frequent than the communication among the '''k'''-point group. The size ~~of the inter-band group is chosen such that~~ {{TAG\|NCORE}} ~~ranks form the intra-band group~~. The ~~intra-~~band group parallelizes the FFTs and triggers the most frequent communications.		[[File:communication.png\|x300px\|VASP divides the available MPI ranks into images, k-point groups, and band groups]]
			[[File:communication2.png\|x300px\|VASP divides the available MPI ranks into images, k-point groups, and band groups]]

			The aforementioned parallelization levels directly map onto MPI communicators. Initially, VASP divides all available ranks into {{TAG\|IMAGES}} equally-sized images. Setting {{TAG\|KPAR}} splits each of these images further into groups of '''k''' points. Properties involving all '''k''' points (e.g. density, Fermi energy) require then a communication over these '''k'''-point groups. VASP divides the '''k'''-point group into band groups of size {{TAG\|NCORE}}. The band group parallelizes the FFTs and triggers the most frequent communications. To evaluate properties involving multiple bands (e.g. Hamiltonian, orthonormalization) communication between the band groups occurs.

	===MPI setup===		===MPI setup===

Schlipf: /* File handling */

2022-07-19T15:21:21Z

File handling

← Older revision		Revision as of 15:21, 19 July 2022
Line 26:		Line 26:
	When VASP is started it reads the file {{TAG\|INCAR}} in the root directory.		When VASP is started it reads the file {{TAG\|INCAR}} in the root directory.
	Because the MPI setup needs to happen early in the general setup of the calculation the following tags are processed before any of the other settings:		Because the MPI setup needs to happen early in the general setup of the calculation the following tags are processed before any of the other settings:
	{{TAG\|NCORE_IN_IMAGE1}}, {{TAG\|IMAGES~~}}, {{TAG\|KIMAGES}}, {{TAG\|FOURORBIT~~}}, {{TAG\|KPAR}}, {{TAG\|NCORE}}, {{TAG\|~~NCORES_PER_BAND~~}}, {{TAG\|~~NPAR~~}}, {{TAG\|~~NCSHMEM~~}}.		{{TAG\|NCORE_IN_IMAGE1}}, {{TAG\|IMAGES}}, {{TAG\|KPAR}}, {{TAG\|NCORE}}, {{TAG\|NPAR}}, {{TAG\|NCSHMEM}}, {{TAG\|LUSENCCL}}.
	When {{TAG\|IMAGES}} are used, subsequently any input given in the root directory is superseded by the same file in subdirectories 01, 02, ...		When {{TAG\|IMAGES}} are used, subsequently any input given in the root directory is superseded by the same file in subdirectories 01, 02, ...
	Any file not present in these subdirectories will be read from the root directory.		Any file not present in these subdirectories will be read from the root directory.
	The output ~~files are~~ files are always written to the subdirectories.		The output files are always written to the subdirectories.
			<!-- The following tags are also read, but the methods are not documented anywhere KIMAGES, FOURORBIT -->

	===Terminology in high-performance computing (HPC)===		===Terminology in high-performance computing (HPC)===

Schlipf: /* Theory */

2022-07-19T15:09:56Z

Theory

← Older revision		Revision as of 15:09, 19 July 2022
Line 22:		Line 22:

	To address this issue, please check the setup of the MPI library and the submitted job script. It is usually possible to overwrite the placement by setting environment variables or command-line arguments. When in doubt, contact the HPC administration of your machine to investigate the behavior.		To address this issue, please check the setup of the MPI library and the submitted job script. It is usually possible to overwrite the placement by setting environment variables or command-line arguments. When in doubt, contact the HPC administration of your machine to investigate the behavior.

			===File handling===
			When VASP is started it reads the file {{TAG\|INCAR}} in the root directory.
			Because the MPI setup needs to happen early in the general setup of the calculation the following tags are processed before any of the other settings:
			{{TAG\|NCORE_IN_IMAGE1}}, {{TAG\|IMAGES}}, {{TAG\|KIMAGES}}, {{TAG\|FOURORBIT}}, {{TAG\|KPAR}}, {{TAG\|NCORE}}, {{TAG\|NCORES_PER_BAND}}, {{TAG\|NPAR}}, {{TAG\|NCSHMEM}}.
			When {{TAG\|IMAGES}} are used, subsequently any input given in the root directory is superseded by the same file in subdirectories 01, 02, ...
			Any file not present in these subdirectories will be read from the root directory.
			The output files are files are always written to the subdirectories.

	===Terminology in high-performance computing (HPC)===		===Terminology in high-performance computing (HPC)===