Memory leak on MI300A

[cmelton]

Hi,

When running long molecular dynamics calculations, I seem to be running out of memory after hundreds or thousands of ionic steps depending on the system. This surprised me since naively I would expect the memory usage to be roughly constant over the simulation so it seemed odd that it would crash only after many ionic steps. From my limited analysis, I seem to be encountering what seems to be a memory leak on a MI300a machine using v6.6.0 and v6.6.1. I've also done tests on H100s and do not see any sign of a leak, so I think it is specific to the OpenMP offload implementation. I'm using a single node, 4 ranks per node (i.e. 1 rank per GPU), and have tried multiple values of OMP threads.

I compiled with the cray CCE 20.0.0, rocm 6.4.2 (and 6.4.3), cray-mpich 9.0.1

It also looks like I can make the crash happen sooner if I increase NSIM, and delay it if I decrease NSIM. However, there is a persistent growth of memory usage over time that eventually uses up all memory and kills my job.

Since MI300a have shared memory, I set up my submission script to call `free -m ` every 60s and plot the memory usage over time and it is attached. Clearly, memory is growing linearly over time. However, I also monitor rocm-smi over the entire simulation and query that every minute, and the VRAM percentage is flat across the entire runtime. So the GPU memory usage isn't increasing over time, so I expect them is something on the host side.

I've also attached a tar file with an example INCAR, POSCAR, POTCAR, KPOINTS, submission file with memory logging calls, and those outputs. The one I uploaded didn't run long enough to trigger the OOM condition, but it eventually will. This doesn't seem related to anything I've seen on the known issues page, so I'm hoping I can get some help on this. Its also worth noting this specific case is using thermodynamic integration. However, I have removed the SCALEE in other tests and get the same behavior.

Thanks,
Cody Melton

[ahampel]

Dear Cody,

thank you for reaching out to us on the Forum and thanks for testing our OMP GPU port! This is all new territory for us and I hope you could get some calculations done with it!

I could reproduce the memory leak you are facing with your input files:

• release.6.6, NSIM=16 : leaks around 24 MB per ionic step
• release.6.6, NSIM=128: leaks around 44 MB per ionic step
• our development master branch leaks even more

Importantly, this is with cce19 on AMD MI210 cards which have dedicated memory. Hence, this is not system dependent nor is it specific to cce20. The leak must be in our code. I will try to hunt it down and give you a report once I find something and hopefully also a workaround or patch fix.

Best regards,
Alex

[ahampel]

Hi Cody,

I run now your job with 3 different toolchains: 1) cray with GPU offloading 2) cray CPU only 3) gfortran CPU + openmpi . For both cray CPU and gfortran the memory is basically flat (numbers in GiB, for 4 ranks in total):

Cray Fortran (posix_memalign) 48.75 48.49 48.75 48.75 48.75 48.75 48.75
slope steps 3-7: 0.0 MB/ionic step

gfortran (plain malloc) 12.14 12.15 12.15 12.15 12.15
slope steps 3-5: 1.9 MB/ionic step

same cray binary WITH offload: 164-202 MB/ionic step ( which is somehow far away from your 500 MB/ionic step)

For me with offloading enabled interestingly the mem plateau's at around 50 GiB per rank:

forum20702_rss_vs_inuse.png

Now comes the weird part. I put in the code malloc info calls to check what glibc accounts for and the amount of memory in use is flat (red line)! It does not growth. This suggests that there is no leak in the code, which is supported by the fact that the CPU only run does not growth. The real growths comes from unclaimed space on the heap. The RSS growths exactly when VASP allocates its new wavefunction objects. The memory is correctly freed but somehow the offloading engine clings on to that memory or does not want to allow new mem to allocate its space.

Mitigations: one can do manual malloc_trim calls that basically flatten the memory requirement but it costs roughly 10% performance. Or what I would like you to test is to set NSIM=16 (btw. in DAV on MI300 a value larger than 32 will not help you, it will rather make things slower). Can you make a mem used plot for NSIM=16 of that run for me? We want to check my hypothesis that it does flatten out as it does for me.

In parallel I will ask our contacts at HPE for advice. Maybe there is some flag that one can tune. My suspect: the runtime tries to hold on to that memory in case the same data comes again and then it could reuse these chunks of memory.

Best,
Alex

[cmelton]

Hi Alex,

Attached is the memory for the same input files on the MI300s with NSIM = 16.

Screenshot 2026-08-14 at 9.44.38 PM.png

So it seems I also see the memory saturate. The reason I have been running with much larger NSIM is because the wiki suggests it helps performance https://vasp.at/wiki/GPU_ports_of_VASP

NSIM determines the number of bands that are optimized simultaneously in many of the electronic solvers (e.g RMM-DIIS and blocked-Davidson). As a rule, one should choose this parameter larger to get good performance on GPUs than one would for CPU-based execution.

Because of that, I started doing smaller MD simulations while scanning NSIM to get better performance. Attached is an example (I don't remember the system, but this was a plot from some of my initial tests on our MI300 machine).

Screenshot 2026-08-14 at 9.56.42 PM.png

So I typically have been running with values larger than 32, but from now on can limit nsim to 16 or 32.

-Cody

[ahampel]

Hi Cody,

update from my side. I identified the cause of this even though I did not find a good workaround. The problem seems to be some mechanism in the cray OpenACC/OpenMP runtime to reuse memory instead of allocating new memory: https://cpe.ext.hpe.com/docs/latest/cce ... nmp.7.html search for CRAY_ACC_REUSE_MEM_LIMIT:

CRAY_ACC_REUSE_MEM_LIMIT
Specify the maximum number of bytes that the Cray accelerator runtime will hold for later reuse.

By default, the Cray accelerator runtime for NVIDIA GPUs does not release memory back to the CUDA runtime, but instead optimizes performance by holding memory allocations for later reuse. Use this environment variable to specify the maximum number of bytes the runtime will hold. To disable this feature, set CRAY_ACC_REUSE_MEM_LIMIT to 0.

If I do compare a untuned run with setting export CRAY_ACC_REUSE_MEM_LIMIT=0 it looks like this:

forum20702_reuse_cache_fix.png

this basically removes the memory growth for me but: performance drops significantly with average LOOP+ time (ionic step) from 15.2 s to 28.2s . That is pretty bad. I tried now with various values for this:

Code: Select all

config                   LOOP+  RSS@end   VRAM/gpu   SCF
unlimited (default)      15.10*    38.0     46.3   134
LIMIT=1 GiB              28.12     44.6     59.0   134
LIMIT=4 GiB              26.45     42.6     61.0   134
LIMIT=16 GiB             25.60     38.7     45.0   134
LIMIT=32 GiB             25.65     38.3     46.4   134
LIMIT=0                  28.16     31.0     59.1   134

none of this removes the performance penalty. This seems to me like a big problem in the OpenMP runtime. I will contact the cray team to look into this. So unfortunately I can only give you the advice to lower NSIM so that things fit into the memory for now. The penalty is probably especially severe on MI300A because you pay twice memory for this, memory for the CPU and GPU which are separate as we currently cannot activate shared memory on the software level.

To confirm you can maybe run with large NSIM, set export CRAY_ACC_REUSE_MEM_LIMIT=0 and see if this works without running OOM.

Regarding the NSIM value. Your graph looks very interesting. What system might this be. I redid this on your system (NSW=1) and get:

Code: Select all

OUTCAR.4r_8thr_NSIM8:       LOOP+:  cpu time     35.6069: real time     23.4645
OUTCAR.4r_8thr_NSIM16:     LOOP+:  cpu time     34.3311: real time     22.4664
OUTCAR.4r_8thr_NSIM32:     LOOP+:  cpu time     35.6437: real time     23.0284
OUTCAR.4r_8thr_NSIM48:     LOOP+:  cpu time     33.2556: real time     21.8084
OUTCAR.4r_8thr_NSIM64:     LOOP+:  cpu time     30.1427: real time     19.9252
OUTCAR.4r_8thr_NSIM96:     LOOP+:  cpu time     32.1101: real time     21.5926
OUTCAR.4r_8thr_NSIM128:   LOOP+:  cpu time     30.5461: real time     20.3310
OUTCAR.4r_8thr_NSIM256:   LOOP+:  cpu time     34.2418: real time     23.5181

So for me these values do not move much but clearly NSIM=64 is best. Larger values get worse again. The reason is known to us: some GPU kernels scale favorable with NSIM, but the allocation of the NSIM sized wavefunctions and some repointering takes time with larger NSIM. That is what we found for most systems. But of course there might be systems where there is a different trend.

Best,
Alex

[cmelton]

Hi Alex,

For the NSIM scan, this was for a completely different system I was testing for a colleague of mine when I first built the code for the MI300s. Looks like it was a 256 atom simulation of 303 stainless steel with PBE. One kpoint, NVT at 300K. It used a very large number of bands (2400), basically to stress test the system.

I reran the problem using NSIM = 128. I'm seeing a much larger performance penalty by disabling the memory reuse, but it does look like the linear growth and OOM won't be a problem. I haven't run this out very long, but here is what I'm seeing.

Screenshot 2026-08-17 at 11.02.36 AM.png

For performance, I typically throw out the first ionic step since and look at the average ionic step time. For NSIM=128, with default settings I was getting 16+/-4 sec per ionic step. However, once I set cray_acc_reuse_mem_limit=0 (which fixes the memory growth issue) the ionic step time is 54 +/- 2 s. So it seems like the performance penalty is significantly worse on the MI300.

-Cody

[cmelton]

Actually, I noticed there was still a slightly positive slope to the memory as I ran it longer. Its now been going for 17 hours with that environment variable, and its about to hit the OOM condition.

Screenshot 2026-08-18 at 9.18.52 AM.png

it has made it to about 600 ionic iterations, whereas the default settings with NSIM=128 is getting to around 450, but its running much much slower with this environment variable.

-Cody

[ahampel]

I agree this looks like a real problem in the runtime. I also tried now with the OMP Offloading / GPU branch for intel GPUs. This basically uses 1:1 the src code that is used for Cray OMP GPU offloading (apart from the blas/fft/lapack wrappers, which I hope are not the problem) and it shows no RSS growths whatsoever:

Code: Select all

┌───────────────────────────┬───────────────────────┬──────────────────┬─────┐
│                           │ floor mem (GB).       │ mem slope        │ SCF │  
├───────────────────────────┼───────────────────────┼──────────────────┼─────┤
│ Cray ENCUT=1500, 2 ranks  │ 11.4 → 13.7           │ 70.6 MB/step     │ 158 │
├───────────────────────────┼───────────────────────┼──────────────────┼─────┤
│ Intel ENCUT=1500, 2 ranks │ 4.8 → 5.0             │ 1.1 MB/step      │ 266 │
└───────────────────────────┴───────────────────────┴──────────────────┴─────┘

I think 1 MB is quite unsuspicious. Right now I do not have any other suggestion than keeping NSIM small. I think the ball is now at the cray team side (they are looking into it but so far none of the ideas really helped).

However, I do have to say that there is a code design change we could keep in the back of our mind as an idea. The memory increase fits pretty much with the per scf step done allocation in the EDDAV call for the local wavefunctions via calls to NEWWAVA and NEWWAV . So something there is making the runtime trip. We could only do the allocation once and reuse the memory across subroutine calls but this is a big code change. I will keep you posted.

Best,
Alex