cannot calculate lage system in 6.5.1

[wujun_shi2]

Hi all,
I encountered an issue when running VASP 6.5.1 for a bilayer graphene system containing 5044 carbon atoms. The calculation stops with the following error message:

FATAL ERROR: data in use_device clause was not found on device 4: host:0x15382206b630

The detailed error output is provided in the attached file.

Notably, the same setup works correctly for smaller systems. VASP was compiled using Intel oneAPI 2025.3.0, MKL 2025.3, and NVHPC 25.9 on Rocky Linux 9.6.

Any insights into the possible cause of this issue and how to resolve it would be greatly appreciated.

Thanks a lot.

[pedro_melo]

Dear wujun_shi2,

Does this calculation produce any OUTCAR or standard output at all? If so could you share them? From the error file you sent it is quite hard to know here the code stops.

Kind regards,
Pedro

[pedro_melo]

Also, please upload here the submission script that you used for this job.

Kind regards,
Pedro

[wujun_shi2]

Dear Pedro,
Please find the attachment of all the files.

Thank you.

Best.
Wujun

[pedro_melo]

Dear Wujun,

We spoke with NVIDIA's team and got the following patch. Unfortunately we cannot test it on our side since we do not have the same amount of resources available. Could you edit the file src/wave_mpi.F as indicated below, recompile VASP with the same toolchain and report to us if it solved your issue?

Kind regards,
Pedro

Code: Select all

diff --git a/src/wave_mpi.F b/src/wave_mpi.F
index acefcc8c3..310419af1 100644
--- a/src/wave_mpi.F
+++ b/src/wave_mpi.F
@@ -943,6 +943,7 @@ __GPU USE moffload
       TYPE (wavedes1)  WDES1
       INTEGER NBANDS
       COMPLEX(q) :: CW(WDES1%NRPLWV*NBANDS)
+      INTEGER(kind=8)  :: OFFSET
 #if defined(T3D_SMA)
       INTEGER MALLOC_DONE
       INTEGER, EXTERNAL :: ISHM_CHECK
@@ -981,8 +982,9 @@ GPACC !$ACC DATA CREATE(CWORK) IF(OFF_ACTIVE) ASYNC(ACC_ASYNC_Q)
 GPOMP !$OMP TARGET ENTER DATA MAP(ALLOC:CWORK)  IF(OFF_ACTIVE)
 #endif
       DO NB=0,NBANDS-1
-         CALL M_alltoall_z(WDES1%COMM_INTER, NRPLWV, CW(NB*NRPLWV+1), CWORK(1))
-         CALL WZCOPY( NRPLWV, CWORK(1), 1, CW(NB*NRPLWV+1), 1)
+         OFFSET=INT(NB,kind=8)*INT(NRPLWV,kind=8) + 1
+         CALL M_alltoall_z(WDES1%COMM_INTER, NRPLWV, CW(OFFSET), CWORK(1))
+         CALL WZCOPY( NRPLWV, CWORK(1), 1, CW(OFFSET), 1)
       ENDDO
 #ifndef T3D_SMA
 GPACC !$ACC WAIT(ACC_ASYNC_Q) IF(OFF_ACTIVE)
 

[wujun_shi2]

Dear Pedro,

After recompiling VASP with the same toolchain, it works.

Thank you so much.

Best,
Wujun