NaN in Machine Learning of liquid surfacs

Problems running VASP: crashes, internal errors, "wrong" results.

Moderators: Global Moderator, Moderator

Locked
Message
Author
julien_steffen
Newbie
Newbie
Posts: 20
Joined: Wed Feb 23, 2022 10:18 am

NaN in Machine Learning of liquid surfacs

#1 Post by julien_steffen » Mon Oct 24, 2022 11:51 am

Hello,

I am running simulations of liquid metal surfaces (e.g., Gallium with other metals solved in it) and generating Machine Learning Force Fields for those systems as well. In most cases, this works well (both training and application); sometimes, however, a single atom detaches from the surface to the gas phase during the training. When the distance of that atom is (presumably) larger than the given cutoff values, the next energy and gradient of the system is calculated to be NaN and the calculation holds (but the process is not aborted until the walltime is exceeded).
I assume that this error occurs, because the environment of the cannot be defined and referenced?
Since the OUTCAR and vasp.out (stdout) files are too large to add them fully (almost 1 GB for the OUTCAR), I have cut a large part of the intermediate lines, leaving the start and the sections before the ending.

Best wishes,
Julien
You do not have the required permissions to view the files attached to this post.

jonathan_lahnsteiner2
Global Moderator
Global Moderator
Posts: 149
Joined: Fri Jul 01, 2022 2:17 pm

Re: NaN in Machine Learning of liquid surfacs

#2 Post by jonathan_lahnsteiner2 » Mon Oct 24, 2022 3:24 pm

Dear Julien,

Could you please also send me the OSZICAR and the ML_LOGFILE. This will make it
much easier to figure out the problem.

All the best
Jonathan

julien_steffen
Newbie
Newbie
Posts: 20
Joined: Wed Feb 23, 2022 10:18 am

Re: NaN in Machine Learning of liquid surfacs

#3 Post by julien_steffen » Tue Oct 25, 2022 9:34 pm

Dear Jonathan,
I have now added the missing files to the zip-archive. Thank you in advance!
Best wishes,
Julien
You do not have the required permissions to view the files attached to this post.

jonathan_lahnsteiner2
Global Moderator
Global Moderator
Posts: 149
Joined: Fri Jul 01, 2022 2:17 pm

Re: NaN in Machine Learning of liquid surfacs

#4 Post by jonathan_lahnsteiner2 » Fri Oct 28, 2022 9:47 am

Dear Julien,

I was looking at your data. You are completely right that there
is an issue when the distance between the atoms is larger than the cutoff.
To debug this issue it would be very helpful if you could also
supply us with the ML_ABN file of this simulation.

All the best

Jonathan

julien_steffen
Newbie
Newbie
Posts: 20
Joined: Wed Feb 23, 2022 10:18 am

Re: NaN in Machine Learning of liquid surfacs

#5 Post by julien_steffen » Thu Nov 03, 2022 9:43 pm

Dear Jonathan,
I have now attached the ML_ABN file.
Another observation was that in most cases when a single atom left the surface, a "cluster" of two to three nearly merged atoms (distances less than 0.3 Angstrom) had been built near the surface. I don't know if one of the observations is a "reason" for the other, but at least there seems to be some correlation.
Best wishes,
Julien
You do not have the required permissions to view the files attached to this post.

ferenc_karsai
Global Moderator
Global Moderator
Posts: 422
Joined: Mon Nov 04, 2019 12:44 pm

Re: NaN in Machine Learning of liquid surfacs

#6 Post by ferenc_karsai » Fri Dec 02, 2022 1:22 pm

Thank you very much for reporting the problem.

There was a bug in the code, so that normalizations are calculated, even when there are no neighbors. This leads to division by zero, which results in some NaNs in some arrays.

I have fixed it now.
I can't post the fix here, because it's too much, but it will be in the next release.

Thanks again for posting the issue.

Locked