VASP MLFF select restart question

[jess_white]

Hi everyone,

I'm running a large MLFF selection calculation in VASP 6.6.0 using ML_MODE = select over 48 hours.

The dataset contains approximately 7415 structures, and the calculation is unable to finish within the available walltime. Before the job ended, the .log file showed progress such as:

processed 1566 of 7415 structures, new force field

and an updated ML_ABN file had been written.

After the job timed out, I resubmitted the calculation from the same directory, keeping all MLFF files (ML_AB, ML_ABN, ML_FFN, etc.). However, the new run appears to restart from:

processed 1 of 7415 structures

rather than continuing from structure 1567.

I have a few questions:

Is there a supported way to continue/restart a partially completed ML_MODE = select calculation?
Does the selection procedure checkpoint its progress, or is it expected to start from the first structure after a restart?
For very large datasets, is there a recommended workflow to avoid walltime issues during selection?

Any advice or examples from people who have run large MLFF selection jobs would be greatly appreciated.

Thanks!

[jonathan_lahnsteiner2]

Dear Jess White,

It is currently not possible to continue a ML_MODE=select run with VASP. We have already implemented an alternative version of the VASPml select in the new c++ code. This feature is not documented yet and does not have a VASP interface yet. To try and compile the feature nevertheless you have to go to the folder

Code: Select all

cd vasp_base/src/vaspml/

There you will find an arch directory. From this directory you can select a makefile.include which you have to copy to the vaspml base folder.

Code: Select all

cp arch/makefile.include.gnu makefile.include

Then you can compile with

Code: Select all

make -j

Note you will need cblas and lapacke libraries for the code to compile successfully and a c++ compiler supporting c++-17. After compilation finished successfully you will find an executable in the ./bin folder called vaspml-select. You can run this executable for example as

Code: Select all

mpirun -np 16 ./bin/vaspml/select path_to_your_INCAR/INCAR path_to_ML_AB/ML_AB.

A minimal INCAR would look like

Code: Select all

ML_LMLFF = .TRUE.
ML_MODE        = Select
ML_SALGO       = FPSN
ML_SMETRIC     = l2norm;
ML_SNCONF      = 12000 12000 12000 1538  # here you have to set the number of local reference configurations for every atom type. If you have 4 atom types place for numbers here.

Please note this feature is still experimental and you should use with care.

Otherwise I can only recommend you to redo the select calculation and increasing the maximum job time in your slurm script. The available compute time can be set by the command

Code: Select all

#SBATCH --time=4-00:00:00

in slurm. With this command you would request 4 days for your computation.

All the Best Jonathan