vasp 6.5.1 for gpu's

[jpg]

Hi,

I am compiling vasp 6.5.1 with nvhpc 23.9 on rocky 9 and I am getting the following errors (6.4.3 with nvhpc 23.9 works fine)

mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.8 -mp -Mfree -w -tp host -tp host -fast -I/cm/shared/apps/spack/0.21.2/gpu/opt/spack/linux-rocky9-skylake_avx512/gcc-11.4.1/nvhpc-23.9-x2of3yawshudrxnvhzpej2t3iw2yatd5/Linux_x86_64/23.9/compilers/extras/qd/include/qd -I/cm/shared/apps/spack/0.21.2/gpu/opt/spack/linux-rocky9-skylake_avx512/nvhpc-23.9/intel-oneapi-mkl-2023.2.0-clvkwap7fetfdz5j5hv2brut2jxhkrkc/mkl/2023.2.0/include/fftw -c incar_reader.f90
NVFORTRAN-S-0026-Unmatched quote (incar_reader.F: 173)
NVFORTRAN-S-0146-Expression must be character type (incar_reader.F: 178)
NVFORTRAN-S-0457-Illegal expression in initialization (incar_reader.F: 178)
nvfortran-Fatal-/cm/shared/apps/spack/0.21.2/gpu/opt/spack/linux-rocky9-skylake_avx512/

[ahampel]

Hi jpg,

This is a problem with the C preprocessor when processing the src/incar_reader.F into incar_reader.f90 . The problem is the line 173:

Code: Select all

escape = "\", ignore_between = '"'

which is treated by the C preprocessor as an escape character. To fix this you have to pass the option -Mbackslash to the pre-processor call. Do this simply by adding:

Code: Select all

FFLAGS      += -Mbackslash

as far as I can see all our provided makefile.includes in arch/ include this option for nvhpc. Please try this.

Best,
Alex

[jpg]

Thanks!

Now I am getting the following error:

mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.8 -mp -Mfree -Mbackslash -Mlarge_arrays -tp host -fast -I/cm/shared/apps/spack/0.21.2/gpu/opt/spack/linux-rocky9-skylake_avx512/gcc-11.4.1/nvhpc-23.9-x2of3yawshudrxnvhzpej2t3iw2yatd5/Linux_x86_64/23.9/compilers/extras/qd/include/qd -I/cm/shared/apps/spack/0.21.2/gpu/opt/spack/linux-rocky9-skylake_avx512/nvhpc-23.9/intel-oneapi-mkl-2023.2.0-clvkwap7fetfdz5j5hv2brut2jxhkrkc/mkl/2023.2.0/include/fftw -c mpi.f90
NVFORTRAN-S-0038-Symbol, lusenccl, has not been explicitly declared (mpi.F: 882)

( here is my makefile.include generated by my spack package; please forgive me: the stupid macos finder would not let my upload the file even if I added extensions):

# Default precompiler options
CPP_OPTIONS ?= -DHOST=\"LinuxNV\" \
-DMPI -DMPI_INPLACE -DMPI_BLOCK=8000 -Duse_collective \
-DCACHE_SIZE=4000 \
-Davoidalloc \
-Dvasp6 \
-Dtbdyn \
-Dqd_emulate \
-Dfock_dblbuf \
-D_OPENMP \
-DACC_OFFLOAD \
-DNVCUDA \
-DUSENCCL

CPP = nvfortran -Mpreprocess -Mfree -Mextend -E $(CPP_OPTIONS) $*$(FUFFIX) > $*$(SUFFIX)

# N.B.: you might need to change the cuda-version here
# to one that comes with your NVIDIA-HPC SDK
CC = mpicc -acc -gpu=cc60,cc70,cc80,cuda11.8 -mp
FC = mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.8 -mp
FCL = mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.8 -mp -c++libs

FREE = -Mfree

FFLAGS = -Mbackslash -Mlarge_arrays

OFLAG = -fast

DEBUG = -Mfree -O0 -traceback

LLIBS = -cudalib=cublas,cusolver,cufft,nccl -cuda

# Redefine the standard list of O1 and O2 objects
SOURCE_O1 := pade_fit.o minimax_dependence.o wave_window.o
SOURCE_O2 := pead.o

# For what used to be vasp.5.lib
CPP_LIB = $(CPP)
FC_LIB = $(FC)
CC_LIB = $(CC)
CFLAGS_LIB = -O -w
FFLAGS_LIB = -O1 -Mfixed
FREE_LIB = $(FREE)

OBJECTS_LIB = linpack_double.o getshmem.o

# For the parser library
CXX_PARS = nvc++ --no_warnings

##
## Customize as of this point! Of course you may change the preceding
## part of this file as well if you like, but it should rarely be
## necessary ...
##
# When compiling on the target machine itself , change this to the
# relevant target when cross-compiling for another architecture
VASP_TARGET_CPU ?= -tp host
FFLAGS += $(VASP_TARGET_CPU)

# Specify your NV HPC-SDK installation (mandatory)
#... first try to set it automatically
NVROOT =$(shell which nvfortran | awk -F /compilers/bin/nvfortran '{ print $$1 }')

# If the above fails, then NVROOT needs to be set manually
#NVHPC ?= /opt/nvidia/hpc_sdk
#NVVERSION = 21.11
#NVROOT = $(NVHPC)/Linux_x86_64/$(NVVERSION)

## Improves performance when using NV HPC-SDK >=21.11 and CUDA >11.2
#OFLAG_IN = -fast -Mwarperf
#SOURCE_IN := nonlr.o

# Software emulation of quadruple precsion (mandatory)
QD ?= $(NVROOT)/compilers/extras/qd
LLIBS += -L$(QD)/lib -lqdmod -lqd
INCS += -I$(QD)/include/qd

# Intel MKL for FFTW, BLAS, LAPACK, and scaLAPACK
MKLROOT ?= /path/to/your/mkl/installation
#MKLLIBS = -Mmkl
MKLLIBS = -lmkl_intel_lp64 -lmkl_pgi_thread -lmkl_core -pgf90libs -mp -lpthread -lm -ldl

# If you want to use scaLAPACK from MKL
LLIBS_MKL = -L$(MKLROOT)/lib -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 $(MKLLIBS)

# Use a separate scaLAPACK installation (optional but recommended in combination with OpenMPI)
# Comment out the two lines below if you want to use scaLAPACK from MKL instead
#SCALAPACK_ROOT ?= /path/to/your/scalapack/installation
#LLIBS_MKL = -L$(SCALAPACK_ROOT)/lib -lscalapack $(MKLLIBS)

LLIBS += $(LLIBS_MKL)

INCS += -I$(MKLROOT)/include/fftw

# Use cusolvermp (optional)
# supported as of NVHPC-SDK 24.1 (and needs CUDA-11.8)
#CPP_OPTIONS+= -DCUSOLVERMP -DCUBLASMP
#LLIBS += -cudalib=cusolvermp,cublasmp -lnvhpcwrapcal

# HDF5-support (optional but strongly recommended, and mandatory for some features)
#CPP_OPTIONS+= -DVASP_HDF5
#HDF5_ROOT ?= /path/to/your/hdf5/installation
#LLIBS += -L$(HDF5_ROOT)/lib -lhdf5_fortran
#INCS += -I$(HDF5_ROOT)/include

# For the VASP-2-Wannier90 interface (optional)
#CPP_OPTIONS += -DVASP2WANNIER90
#WANNIER90_ROOT ?= /path/to/your/wannier90/installation
#LLIBS += -L$(WANNIER90_ROOT)/lib -lwannier

# For the fftlib library (hardly any benefit for the OpenACC GPU port, especially in combination with MKL's FFTs)
#CPP_OPTIONS+= -Dsysv
#FCL += fftlib.o
#CXX_FFTLIB = nvc++ -mp --no_warnings -std=c++11 -DFFTLIB_USE_MKL -DFFTLIB_THREADSAFE
#INCS_FFTLIB = -I./include -I$(MKLROOT)/include/fftw
#LIBS += fftlib
#LLIBS += -ldl

# For machine learning library vaspml (experimental)
#CPP_OPTIONS += -Dlibvaspml
#CPP_OPTIONS += -DVASPML_USE_CBLAS
#CPP_OPTIONS += -DVASPML_DEBUG_LEVEL=3
#CXX_ML = mpic++ -mp
#CXXFLAGS_ML = -O3 -std=c++17 -Wall -Wextra
#INCLUDE_ML =

# Add -gpu=tripcount:host to compiler commands for NV HPC-SDK > 25.1
NVFORTRAN_VERSION := $(shell nvfortran --version | sed -n '2s/^nvfortran \([0-9.]*\).*/\1/p')
define greater_or_equal
$(shell printf '%s\n%s\n' '$(1)' '$(2)' | sort -V | head -n1 | grep -q '$(2)' && echo true || echo false)
endef
ifeq ($(call greater_or_equal,$(NVFORTRAN_VERSION),25.1),true)
CC += -gpu=tripcount:host
FC += -gpu=tripcount:host
endif

[ahampel]

This is a bit weird. Lets unpack this.

First thing I notice in your makefile.include is a missing -DscaLAPACK pre-compiler option. Please add this. Scalapack is pretty much recommended and since you link against MKL scalapack anyway please compile with this option. Then let's update your MKL link line (might not be a problem but still):

Code: Select all

MKLROOT    ?= /path/to/your/mkl/installation
LLIBS_MKL   = -Mmkl -L$(MKLROOT)/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64
INCS       += -I$(MKLROOT)/include/fftw
LLIBS      += $(LLIBS_MKL)

this will use all libraries from MKL including scalapack. -Mmkl I found much safer than manually specifying the mkl libs. Please try this. Also make sure to run make veryclean before building again. If the error persists I would need to see pre-compiler command. If you run make std first the PP is done and you should see something like:

Code: Select all

nvfortran -Mpreprocess -Mfree -Mextend -E -DHOST=\"LinuxNV\" -DMPI -DMPI_INPLACE -DMPI_BLOCK=8000 -Duse_collective -DscaLAPACK -Davoidalloc -DCACHE_SIZE=4000 -Dvasp6 -Dtbdyn -Dqd_emulate -Dfock_dblbuf -DACC_OFFLOAD -DNVCUDA -D_OPENMP -DUSENCCL  mpi.F > mpi.f90

Make sure that -DUSENCCL is in this line. if not somehow the -DUSENCCL from your makefile.include is not recognized. In the end in the mpi.f90 should be usenccl . This seems to be a bug by your precompiler.

Maybe also try makefile.include.nvhpc_ompi_mkl_omp_acc from arch and see if that just works.

Best,
Alex

[jpg]

The way the vasp spack package works is the the CPP_OPTIONS environment variable is set and -DScalapack and -DUSENCCL are passed along:

nvfortran -Mpreprocess -Mfree -Mextend -E -DMPI -DMPI_BLOCK=8000 -Duse_collective -DCACHE_SIZE=4000 -Davoidalloc -Duse_bse_te -Dtbdyn -Duse_shmem -Dfock_dblbuf -D_OPENMP -DHOST=\"LinuxGNU\" -Dvasp6 -DMPI_INPLACE -D_OPENACC -DUSENCCL -DUSENCCLP2P -Dqd_emulate -DscaLAPACK mpi.F > mpi.f90 -DNGZhalf

Also the spack package is using makefile.include.nvhpc_ompi_mkl_omp_acc

Should I try a newer version of nvhpc ?

Thanks,

Jerry

[ahampel]

Ah here we see what is wrong. If you do add -DUSENCCL (or spack, whoever it was). You have to add also -DACC_OFFLOAD. If I remove in my pre-processing line -DACC_OFFLOAD but keep -DUSENCCL I can reproduce your error.

Currently our pre-processing relies on the GPU port of VASP being active if you want to use NCCL. I do believe that you want to use the GPU port, so please do add -DACC_OFFLOAD. Your generated makefile.include had this but your pre-processing command does not. So this is a problem. Can you point me to the spack package (currently we are not maintainer of it). I can have a look and try to see what is going wrong. But you have to add this. I do not understand why spack is not using all defined CPP_OPTIONS but it looks like it messed up something.

Best,
Alex

[jpg]

Thanks!

I added -DACC_OFFLOAD but now it's complaing about a missing mod file. I attached the output and the spack package.py file with a ".gz" appended to the name so that the macOS finder would let me upload it. (Note that this is my modification of the spack package: I see in the latest version of spack that they finally are supporting 6.5.1, (spack 1.2.0.dev), but I am working with spack 0.21.2 for one of our clusters). Note that the idea of setting CPP_OPTIONS as an environment variable ( so that you don't actually see the real value of it in the makefile.include file ) is not my idea: that is the way spack does it.

Jerry

[ahampel]

Hi Jerry,

well now -DNVCUDA is still missing. Sorry I missed that one. Also currently you do not have "DCUSOLVERMP -DCUBLASMP -DVASP_HDF5" . The first two are optional for BSE workloads, but the HDF5 support may come handy. Anyway adding -DNVCUDA should fix this issue.

So modifying:

Code: Select all

        if '+acc' in spec:
            cpp_options.extend(['-DMPI_INPLACE','-D_OPENACC','-DUSENCCL','-DUSENCCLP2P','-DACC_OFFLOAD'])

to

Code: Select all

        if '+acc' in spec:
            cpp_options.extend(['-DMPI_INPLACE','-D_OPENACC','-DUSENCCL','-DNVCUDA','-DACC_OFFLOAD'])

the other -DUSENCCLP2P is not used I believe. At least I am not using it.

Note that the idea of setting CPP_OPTIONS as an environment variable ( so that you don't actually see the real value of it in the makefile.include file ) is not my idea: that is the way spack does it.

Well that explains the problems. From 6.4.x to 6.5.x the pre-compiler flags used internally have been changed quite a bit.

Best,
Alex

[jpg]

I also had the spack package add the line:

%.o: %.mod

to makefile.include after it complained about "m2c not found"

and now it works.

Thanks!

Jerry