Memory leak on MI300A

[cmelton]

Hi,

When running long molecular dynamics calculations, I seem to be running out of memory after hundreds or thousands of ionic steps depending on the system. This surprised me since naively I would expect the memory usage to be roughly constant over the simulation so it seemed odd that it would crash only after many ionic steps. From my limited analysis, I seem to be encountering what seems to be a memory leak on a MI300a machine using v6.6.0 and v6.6.1. I've also done tests on H100s and do not see any sign of a leak, so I think it is specific to the OpenMP offload implementation. I'm using a single node, 4 ranks per node (i.e. 1 rank per GPU), and have tried multiple values of OMP threads.

I compiled with the cray CCE 20.0.0, rocm 6.4.2 (and 6.4.3), cray-mpich 9.0.1

It also looks like I can make the crash happen sooner if I increase NSIM, and delay it if I decrease NSIM. However, there is a persistent growth of memory usage over time that eventually uses up all memory and kills my job.

Since MI300a have shared memory, I set up my submission script to call `free -m ` every 60s and plot the memory usage over time and it is attached. Clearly, memory is growing linearly over time. However, I also monitor rocm-smi over the entire simulation and query that every minute, and the VRAM percentage is flat across the entire runtime. So the GPU memory usage isn't increasing over time, so I expect them is something on the host side.

I've also attached a tar file with an example INCAR, POSCAR, POTCAR, KPOINTS, submission file with memory logging calls, and those outputs. The one I uploaded didn't run long enough to trigger the OOM condition, but it eventually will. This doesn't seem related to anything I've seen on the known issues page, so I'm hoping I can get some help on this. Its also worth noting this specific case is using thermodynamic integration. However, I have removed the SCALEE in other tests and get the same behavior.

Thanks,
Cody Melton

[ahampel]

Dear Cody,

thank you for reaching out to us on the Forum and thanks for testing our OMP GPU port! This is all new territory for us and I hope you could get some calculations done with it!

I could reproduce the memory leak you are facing with your input files:

• release.6.6, NSIM=16 : leaks around 24 MB per ionic step
• release.6.6, NSIM=128: leaks around 44 MB per ionic step
• our development master branch leaks even more

Importantly, this is with cce19 on AMD MI210 cards which have dedicated memory. Hence, this is not system dependent nor is it specific to cce20. The leak must be in our code. I will try to hunt it down and give you a report once I find something and hopefully also a workaround or patch fix.

Best regards,
Alex

[ahampel]

Hi Cody,

I run now your job with 3 different toolchains: 1) cray with GPU offloading 2) cray CPU only 3) gfortran CPU + openmpi . For both cray CPU and gfortran the memory is basically flat (numbers in GiB, for 4 ranks in total):

Cray Fortran (posix_memalign) 48.75 48.49 48.75 48.75 48.75 48.75 48.75
slope steps 3-7: 0.0 MB/ionic step

gfortran (plain malloc) 12.14 12.15 12.15 12.15 12.15
slope steps 3-5: 1.9 MB/ionic step

same cray binary WITH offload: 164-202 MB/ionic step ( which is somehow far away from your 500 MB/ionic step)

For me with offloading enabled interestingly the mem plateau's at around 50 GiB per rank:

forum20702_rss_vs_inuse.png

Now comes the weird part. I put in the code malloc info calls to check what glibc accounts for and the amount of memory in use is flat (red line)! It does not growth. This suggests that there is no leak in the code, which is supported by the fact that the CPU only run does not growth. The real growths comes from unclaimed space on the heap. The RSS growths exactly when VASP allocates its new wavefunction objects. The memory is correctly freed but somehow the offloading engine clings on to that memory or does not want to allow new mem to allocate its space.

Mitigations: one can do manual malloc_trim calls that basically flatten the memory requirement but it costs roughly 10% performance. Or what I would like you to test is to set NSIM=16 (btw. in DAV on MI300 a value larger than 32 will not help you, it will rather make things slower). Can you make a mem used plot for NSIM=16 of that run for me? We want to check my hypothesis that it does flatten out as it does for me.

In parallel I will ask our contacts at HPE for advice. Maybe there is some flag that one can tune. My suspect: the runtime tries to hold on to that memory in case the same data comes again and then it could reuse these chunks of memory.

Best,
Alex

[cmelton]

Hi Alex,

Attached is the memory for the same input files on the MI300s with NSIM = 16.

Screenshot 2026-08-14 at 9.44.38 PM.png

So it seems I also see the memory saturate. The reason I have been running with much larger NSIM is because the wiki suggests it helps performance https://vasp.at/wiki/GPU_ports_of_VASP

NSIM determines the number of bands that are optimized simultaneously in many of the electronic solvers (e.g RMM-DIIS and blocked-Davidson). As a rule, one should choose this parameter larger to get good performance on GPUs than one would for CPU-based execution.

Because of that, I started doing smaller MD simulations while scanning NSIM to get better performance. Attached is an example (I don't remember the system, but this was a plot from some of my initial tests on our MI300 machine).

Screenshot 2026-08-14 at 9.56.42 PM.png

So I typically have been running with values larger than 32, but from now on can limit nsim to 16 or 32.

-Cody