Performance regression in OpenACC+OMP CPU threading

[aohara]

Dear VASP team,

I'm not sure if this is a bug per se or a necessary consequence of adding the OpenMP off-loading for AMD/Intel GPUs. However, I have noticed a performance regression in using OpenACC+OMP. I first noticed it in going from version 6.5.1 to version 6.6.0, but I didn't get a chance to run more thorough tests until I was compiling version 6.6.1 over the past week.

In all cases, compilation was done using the same library modules and basic makefile.include. This was based on the makefile.include.nvhpc_omp_acc provided in the arch directory. I have included a list of what the cluster I use has installed for related modules loaded. Hardware wise, the nodes being used are dual socket AMD EPYC 7713 Processors (128 total cores) with 512 GB memory and 4x A100 SXM GPUs with 81920 MB memory. The compiler toolchain was Nvidia SDK 23.7 with Cuda 11.8 selected due to drivers.

As an example, I am running an HSEsol calculation on a 4x4x4 supercell of the silicon conventional cell with the gamma point flavor. I have seen similar changes in other systems as well though.

The first comparison is the baseline 6.5.1 run with no omp compared to the 6.5.1 run with omp. In this case, combining OpenACC+OMP provides a speed up of ~10% per electronic loop.

Compiling 6.6.1 the same way with OpenACC+OMP and using the same jobscript yields similar timings to the baseline 6.5.1 run.

After this, I tried checking other environmental variables. First adding to the jobscript:
OPENBLAS_NUM_THREADS = $SLURM_CPUS_PER_TASK
GOTO_NUM_THREADS = $SLURM_CPUS_PER_TASK

And eventually also adding:
SCALAPACK_NUM_THREADS = $SLURM_CPUS_PER_TASK
FFTW_NUM_THREADS = $SLURM_CPUS_PER_TASK
MPI_PARAM_MAX_THREADS = $SLURM_CPUS_PER_TASK

However, all of these gave similar timings as the original baseline 6.5.1 run and the above 6.6.1 run.

Next, I tested changing the bindings on the mpiexec line to:
--bind-to core --map-by slot:PE=32
and this also ran at the baseline speed.

I did try a version where I compiled without explicit OpenMP linking in the makefile.include (removed -D_OPENMP precompiler flag, -mp flags, and -lfftw_omp) to test if the OpenBLAS threads could be activated separately and saw no performance gain return.

Lastly, I tried seeing if I could get the Nvidia compilers to do the threading by adding:
-stdpar=multicore
to the FFLAGS variable. However, this failed the final linking step in creating an executable due to conflicts (nvlink error: Multiple definition of 'kernel1_779_gpu' in 'rmm-diis_lr.o', first defined in 'hamil_lr.o').

As I said, I don't know if this is really a bug or a necessity due to code refactoring for the OpenMP offloading version, but the small (10% here, closer to 15% in some other systems) performance boost with OpenACC+OMP has come in handy for certain work flows involving many ionic steps.

I've attached files that include the loaded modules environment, the makefile.include used, the basic VASP input files, and the jobscript/OUTCAR for the 6.5.1 no OpenMP baseline, the 6.5.1 OpenMP run, and the 6.6.1 OpenMP run.

Thank you for your help,
Andy

[ahampel]

Dear Andy,

thank you for the detailed report and sorry for the regression. Indeed the problem was known but we misjudged the collateral / performance impact the regression could have. The issue is fixed on our development branch but we missed to bake the fix into the recent 6.6.1 release. The problem is a single routine: SET_DD_PAW which is covered by the timer SETDIJ in your OUTCAR flat profiles. Note, that this is the only timer impacted by the number of OMP threads. There is basically no other part in the OpenACC port with active GPUs that really needs OMP threads anymore (there might be some smallish lapack, fft calls on the host but they do not show up even in these larger sized jobs you have). The workload in in SET_DD_PAW is efficiently parallelized over ions on the CPU. This is the calculation of the one center terms of the PAW projectors and since it involves a lot of small operations it is not ported to GPUs. It is called once per SCF step. Here are the timings from your flat profiles extracted:

Code: Select all

| section       | 6.5.1, no OMP | 6.5.1, OMP=32 | 6.6.1, OMP=32 |
|---------------|---------------|---------------|---------------|
| POTLOK real   |   2.56 s      |   2.54 s      |   0.04 s      |
| SETDIJ real   |  12.23 s      |   **0.88 s**  |  **12.14 s**  |
| SETDIJ cpu    |  12.23 s      |  13.74 s      |  13.07 s      |
| EDDAV real    |  89.76 s      |  89.96 s      |  90.65 s      |
| **LOOP real** | **104.55 s**  | **93.38 s**   | **102.82 s**  |

you can see that the rest like EDDAV is untouched by this. This went unnoticed because even for a 64 atom system the compute time in this routine is 0.1sec - negligible for a total time of minutes. But sure enough here in hybrids with many channels etc this counts. Sorry for that. A change was made intended to increase the performance on systems with NIONS<OMP_NUM_THREADS but it went wrong. The fix is simple:

Code: Select all

--- a/src/paw.F
+++ b/src/paw.F
@@ -1499,7 +1499,7 @@ __GPU USE moffload
       LOGICAL LSKIP
 
 !$    INTEGER IDONP
-!$    INTEGER, EXTERNAL :: OMP_GET_NUM_THREADS,OMP_GET_THREAD_NUM
+!$    INTEGER, EXTERNAL :: OMP_GET_NUM_THREADS,OMP_GET_THREAD_NUM,OMP_GET_MAX_THREADS
 
       PROFILING_START('set_dd_paw')
 
@@ -1600,7 +1600,7 @@ __GPU    CALL OFFLOAD_ON_POP
       PP => NULL()
 
 ! reduce number of threads spawned to minimize overhead
-!$OMP PARALLEL DEFAULT(NONE) NUM_THREADS(MIN(T_INFO%NIONS, OMP_GET_NUM_THREADS())) &
+!$OMP PARALLEL DEFAULT(NONE) NUM_THREADS(MIN(T_INFO%NIONS, OMP_GET_MAX_THREADS())) &
 !$OMP PRIVATE(NI, NIP, NT, LSKIP, ISP, LYMAX, RNMAX, LMAX_TAU, LMMAX_TAU, LMMAX, &
 !$OMP TAUAE, TAUPS, MUAE, MUPS, KINDENSCOL, RHOLM, RHOLM_, COCC, COCC_IM, ISIZE, IADD, ITMP, &
 !$OMP RHOAE, RHO, RHOCOL, &

change these two lines. It is just that OMP_GET_NUM_THREADS is simply wrong and always gives 1 outside an OMP region. OMP_GET_MAX_THREADS is the right library function call to OMP. For a smaller 64 atom hyb test I get:

Code: Select all

| version         | thr | SETDIJ real | LOOP real | LOOP+ real |
|-----------------|-----|-------------|-----------|------------|
| 6.5.1           |  1  |   0.311 s   |   8.41 s  |  48.46 s   |
| 6.5.1           | 20  |   0.169 s   |   8.26 s  |  46.09 s   |
| 6.6.1           |  1  |   0.311 s   |   6.37 s  |  30.42 s   |
| 6.6.1           | 20  |   0.444 s   |   5.92 s  |  31.24 s   |
| 6.6.1 + fix     |  1  |   0.314 s   |   5.64 s  |  29.66 s   |
| 6.6.1 + fix     | 20  |   0.170 s   |   5.39 s  |  27.69 s   |

Same timings as in 6.5.1 . Done. Next version of VASP has this shipped. In general you should see some speedup in hybrids on Nvidia GPUs in 6.6.1 compared to 6.5.1 (see table LOOP+ timings) . Especially for mid sized systems. Your system is probably already FFT saturated in the hybrid and thus you see the same timings between 6.5. and 6.6 (apart from that bug)

Best regards,
Alex