Memory leak on MI300A
Hi,
When running long molecular dynamics calculations, I seem to be running out of memory after hundreds or thousands of ionic steps depending on the system. This surprised me since naively I would expect the memory usage to be roughly constant over the simulation so it seemed odd that it would crash only after many ionic steps. From my limited analysis, I seem to be encountering what seems to be a memory leak on a MI300a machine using v6.6.0 and v6.6.1. I've also done tests on H100s and do not see any sign of a leak, so I think it is specific to the OpenMP offload implementation. I'm using a single node, 4 ranks per node (i.e. 1 rank per GPU), and have tried multiple values of OMP threads.
I compiled with the cray CCE 20.0.0, rocm 6.4.2 (and 6.4.3), cray-mpich 9.0.1
It also looks like I can make the crash happen sooner if I increase NSIM, and delay it if I decrease NSIM. However, there is a persistent growth of memory usage over time that eventually uses up all memory and kills my job.
Since MI300a have shared memory, I set up my submission script to call `free -m ` every 60s and plot the memory usage over time and it is attached. Clearly, memory is growing linearly over time. However, I also monitor rocm-smi over the entire simulation and query that every minute, and the VRAM percentage is flat across the entire runtime. So the GPU memory usage isn't increasing over time, so I expect them is something on the host side.
I've also attached a tar file with an example INCAR, POSCAR, POTCAR, KPOINTS, submission file with memory logging calls, and those outputs. The one I uploaded didn't run long enough to trigger the OOM condition, but it eventually will. This doesn't seem related to anything I've seen on the known issues page, so I'm hoping I can get some help on this. Its also worth noting this specific case is using thermodynamic integration. However, I have removed the SCALEE in other tests and get the same behavior.
Thanks,
Cody Melton