NCORE parallelization not functioning correctly in VASP 6.5.1 CPU version

[chenggoj]

Problem Description
I am experiencing an issue with the NCORE tag in VASP 6.5.1 CPU version compiled with GNU compilers. Despite explicitly setting NCORE = 8 in the INCAR file, the calculation appears to default to NCORE = 1, which significantly impacts computational efficiency.

System and Compilation Details

VASP Version: 6.5.1
Compiler: GNU
Hardware: 128 CPU cores on 1 node
MPI Configuration: 128 mpi-ranks with 2 threads/rank

INCAR Setting:

NCORE = 8

From the OUTCAR header:

running 128 mpi-ranks, with 2 threads/rank, on 1 nodes
distrk: each k-point on 128 cores, 1 groups
distr: one band on NCORE= 1 cores, 128 groups

When I checked

grep NCORE OUTCAR

I will get

distr: one band on NCORE= 1 cores, 128 groups
NCORE = 8

The OUTCAR confirms that NCORE = 8 is being read from the INCAR file, but the parallelization is not working as expected. The calculation completes successfully without errors, but I am not sure whether this is a bug or not.

I would appreciate any guidance on resolving this issue.
Thank you for your assistance.

[ahampel]

Hi,

thank you for reaching out on the official VASP forum. Usually there is a good reason why VASP overwrites the provided NCORE . Are you using the OpenMP enabled version VASP and set OMP_NUM_THREADS>1 ? I can see the following output:

Code: Select all

MPI Configuration: 128 mpi-ranks with 2 threads/rank

pointing to the fact that you used multiple threads per MPI rank. This is fine, but overwrites any NCORE setting. Internally, the code can either use NCORE parallelism or threading but not both. See also https://www.vasp.at/wiki/index.php/Comb ... difference .

Can you maybe share with the input files, the full OUTCAR, and the exact command you used to execute VASP? But depending on your job it could be more efficient to use OMP_NUM_THREADS>1 and NCORE=1 or use OMP_NUM_THREADS=1 and NCORE=8.

Thank you!

[chenggoj]

Hi ahampel，

Thanks so much for your reply. Yes, you are right. I used

export OMP_NUM_THREADS=2

and

srun -n 128 -c 2 --cpu-bind=cores vasp_gam

in my submission script. Now, I understand them.

As you said, I can use

OMP_NUM_THREADS>1 and NCORE=1

or

OMP_NUM_THREADS=1 and NCORE=8

, right?

I think it might be related to my computing node specifications.

System Specifications:

Total CPUs: 256 logical cores
Online CPU list: 0-255
Vendor: AuthenticAMD
Model: AMD EPYC 7763 64-Core Processor
CPU family: 25
Model: 1
Threads per core: 2
Cores per socket: 64
Sockets: 2

Based on this, it seems that I should keep OMP_NUM_THREADS=2 and NCORE = 1? I'm just worried if this might cause a serious performance decrease compared with NCORE = 8.

Thanks a lot.

[ahampel]

What kind of calculation are you performing? For normal DFT SCF or relaxation calculations I would suggest to try a bit yourself what works best. Usually keeping OMP_NUM_THREADS=1 and using NCORE for improved performance is a good starting point. Typically using OMP_NUM_THREADS instead of NCORE will not get faster for normal DFT jobs. This will be different for GW, BSE or other more advanced calculations.

Try also to leverage KPAR to parallelize over kpoints (if you have enough kpoints to split them). I have a small table here to compare tome combinations of NCORE/KPAR for different systems against each other: https://github.com/flatironinstitute/cc ... /README.md . But you should do your own testing for your system.

Best,
Alex