Intel GPU - Slow EDDAV - after update oneapi 2026.1

[alvaro_vazquez-mayagoitia1]

Dear vasp-dev team,

We recently update the OS and Intel oneapi to 2026.1 on Aurora Intel GPU/PVC (driver is the same). After the update I noticed a regression in the performance of EDDAV times.

I did a fresh install of vasp 6.6.1, I am running with 12 mpi ranks, 1 rank per tile, using one full node (6GPU, 2Tiles per GPU).

These are my findings,

Code: Select all

# EDDAV timing evidence (same 216-atom MoS2 cell/temp, same INCAR, 1 node/12 tiles)
# Source: MoS2 Chain_1 1400K OUTCAR(.full). 'EDDAV' = blocked-Davidson eigensolver = ~99% of each SCF iteration.

## FAST  (pre-upgrade, 2026-09; OUTCAR.full) -- cpu ~= real ~34 s
     EDDAV:  cpu time     34.0924: real time     34.0350
     EDDAV:  cpu time     43.7086: real time     43.7164
     EDDAV:  cpu time     32.6944: real time     32.6392
     EDDAV:  cpu time     41.0601: real time     40.9939
     EDDAV:  cpu time     42.0169: real time     41.9469
     EDDAV:  cpu time     39.4240: real time     39.3546

## SLOW  (post-upgrade, 2026-10-01; OUTCAR) -- cpu ~500 s, real ~900-1200 s
     EDDAV:  cpu time    453.7851: real time    915.3841
     EDDAV:  cpu time    600.2974: real time   1206.4452
     EDDAV:  cpu time    481.5902: real time    968.8751
     EDDAV:  cpu time    569.9442: real time   1158.6036
     EDDAV:  cpu time    591.2253: real time   1200.4863
     EDDAV:  cpu time    573.7501: real time   1144.7010

## Full per-iteration LOOP time for reference
FAST LOOP:
      LOOP:  cpu time     37.5732: real time     37.5240
      LOOP:  cpu time     43.7100: real time     43.7177
      LOOP:  cpu time     32.6954: real time     32.6403
      LOOP:  cpu time     41.0611: real time     40.9949
SLOW LOOP:
      LOOP:  cpu time    457.9436: real time    922.9618
      LOOP:  cpu time    600.2982: real time   1206.4460
      LOOP:  cpu time    481.5914: real time    968.8918
      LOOP:  cpu time    569.9454: real time   1158.6056

While I ran I checked the XPU memory, it verified it was used.

I'd appreciate any help here,
Thank you

PS>
I had to change to remove compilation errors in david_full.F, the following line:

Code: Select all

CPROJ(1:NPRO,N) = GCIJP(1:NPRO)

to

Code: Select all

CPROJ(1:NPRO,N) = GCIJP

[michael_wolloch]

Dear Alvaro,

I could reproduce these very slow timings with DAV on Aurora.

It looks like this is all very slow MPI communication, as the timers for

Code: Select all

m_alltoall_d

Code: Select all

m_alltoallv_z

Code: Select all

m_bvast_z

and

Code: Select all

m_symb_d

are the most costly parts of the code in the new compile. Compared to an old run of the same benchmark (different VASP version, so not a properly fair comparison), these timers use 1415s in the new run, but only 26s in the old run.

I will investigate further and hope that I can give you a solution soon. I hope it is just a matter of settings on Aurora.

Cheers, and thanks for the regression report,

Michael

[alvaro_vazquez-mayagoitia1]

Dear Michael,

Thank you very much for your quick assessment, this gave us a good lead to where to look at.

We switched to the module mpich/prd/5.0.0.aurora_test.51a94741 and now we see a performance very similar as before the update, about 50 sec for each EDDAV timing.

Best wishes,
Álvaro

[michael_wolloch]

Dear Alvaro,

I have a fix for you to try.

Please set these two environment variables with the default modules:

Code: Select all

export MPIR_CVAR_CH4_IPC_GPU_CACHE_SIZE=unlimited
export LIBOMPTARGET_LEVEL_ZERO_MEMORY_POOL=device,64,8,1024

By default, the first MPICH variable is set to limited, which restricts the MPICH cache to 16 IPC handles. VASP needs many more device buffers, so communication handles are constantly closed and reopened instead of cached, and for the new modules, that takes a lot of time.

The LIBOMPTARGET variable concerns the OpenMP runtime's device memory pool. By default, it is set to device,1,4,256, which means that only allocations smaller than 1MB are pooled, and VASP needs significantly larger allocations.

Both environment variables are needed together for me to get performance to the level before the update. Indeed, performance is a bit better now, since the LAPACK calls get faster in OneAPI 2026.1!
Note that I did not exhaustively test this yet, so it might lead to other problems, but I am quite optimistic.

I also reached out to our contacts at Intel, who are working closely with Aurora. They report that a new image that should be deployed today should fix these problems out of the box, but so far I cannot see this new image on Aurora yet.

Let me know if this fix works for you, and I will update this thread if I can confirm that a new image or set of modules is deployed and working without changing these two environment variables.

Cheers, Michael

EDIT: Sorry Alvaro, I missed your message before posting mine. I will test the module you mentioned ASAP and report back!

[michael_wolloch]

Hi again, Alvaro,

First, I want to apologize for not realizing initially that you reported this problem as an Aurora admin, and for missing your post about the new module. For some reason, I did not get a notification that you replied.

I have now confirmed that it indeed works without setting the two env vars. However, it is not the default module being loaded, so it might be a good idea to change that?

Cheers, Michael