request for functional makefile.include for GPU and 6.3.0

Questions regarding the compilation of VASP on various platforms: hardware, compilers and libraries, etc.

Moderators: Global Moderator, Moderator

Locked
Message
Author
User avatar
paulfons
Jr. Member
Jr. Member
Posts: 72
Joined: Sun Nov 04, 2012 2:40 am
License Nr.: 5-1405
Location: Yokohama, Japan
Contact:

request for functional makefile.include for GPU and 6.3.0

#1 Post by paulfons » Tue Mar 15, 2022 4:34 am

I am having trouble with undefined symbols trying to compile vasp 6.3.0 using the OpenACC GPU-port of VASP. I am under the assumption that someone must have already successfully compiled the GPU version. My system is CentOS 7.9 and have intel oneapi installed. I have an Ampere GPU and Cuda version 11.6 is installed. The Nvidia hpc_sdk version is 22.2 (the latest). Can someone send me a copy of a functional makefile.include (or offer suggestions as to what I am doing incorrectly). Thanks.
nvidia-smi
Tue Mar 15 13:27:14 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A30 Off | 00000000:CA:00.0 Off | 0 |
| N/A 31C P0 31W / 165W | 0MiB / 24576MiB | 3% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+


The makefile.include is attached below (attachments didn't work)
# Default precompiler options
CPP_OPTIONS = -DHOST=\"LinuxNV\" \
-DMPI -DMPI_BLOCK=8000 -Duse_collective \
-DscaLAPACK \
-DCACHE_SIZE=4000 \
-Davoidalloc \
-Dvasp6 \
-Duse_bse_te \
-Dtbdyn \
-Dqd_emulate \
-Dfock_dblbuf \
-D_OPENMP \
-D_OPENACC \
-DUSENCCL -DUSENCCLP2P

CPP = nvfortran -Mpreprocess -Mfree -Mextend -E $(CPP_OPTIONS) $*$(FUFFIX) > $*$(SUFFIX)

# N.B.: you might need to change the cuda-version here
# to one that comes with your NVIDIA-HPC SDK
FC = mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.6 -mp
FCL = mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.6 -mp -c++libs

FREE = -Mfree

FFLAGS = -Mbackslash -Mlarge_arrays

OFLAG = -fast

DEBUG = -Mfree -O0 -traceback

OBJECTS = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o

LLIBS = -cudalib=cublas,cusolver,cufft,nccl -cuda

# Redefine the standard list of O1 and O2 objects
SOURCE_O1 := pade_fit.o
SOURCE_O2 := pead.o

# For what used to be vasp.5.lib
CPP_LIB = $(CPP)
FC_LIB = nvfortran
CC_LIB = nvc -w
CFLAGS_LIB = -O
FFLAGS_LIB = -O1 -Mfixed
FREE_LIB = $(FREE)

OBJECTS_LIB = linpack_double.o

# For the parser library
CXX_PARS = nvc++ --no_warnings

##
## Customize as of this point! Of course you may change the preceding
## part of this file as well if you like, but it should rarely be
## necessary ...
##

# Specify your NV HPC-SDK installation (mandatory)
#... first try to set it automatically
#NVROOT =$(shell which nvfortran | awk -F /compilers/bin/nvfortran '{ print $$1 }')

# If the above fails, then NVROOT needs to be set manually
NVHPC = /opt/nvidia/hpc_sdk
NVVERSION = 22.2
NVROOT = $(NVHPC)/Linux_x86_64/$(NVVERSION)

# Software emulation of quadruple precsion (mandatory)
QD ?= $(NVROOT)/compilers/extras/qd
LLIBS += -L$(QD)/lib -lqdmod -lqd
INCS += -I$(QD)/include/qd

# Intel MKL (FFTW, BLAS, LAPACK, and scaLAPACK)
MKLROOT ?= /path/to/your/mkl/installation
LLIBS += -Mmkl -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64
INCS += -I$(MKLROOT)/include/fftw

# HDF5-support (optional but strongly recommended)
#CPP_OPTIONS+= -DVASP_HDF5
#HDF5_ROOT ?= /path/to/your/hdf5/installation
#LLIBS += -L$(HDF5_ROOT)/lib -lhdf5_fortran
#INCS += -I$(HDF5_ROOT)/include

# For the VASP-2-Wannier90 interface (optional)
#CPP_OPTIONS += -DVASP2WANNIER90
#WANNIER90_ROOT ?= /path/to/your/wannier90/installation
WANNIER90_ROOT = /opt/wannier
LLIBS += -L$(WANNIER90_ROOT)/lib -lwannier

# For the fftlib library (experimental)
#CPP_OPTIONS+= -Dsysv
#FCL += fftlib.o
#CXX_FFTLIB = nvc++ -mp --no_warnings -std=c++11 -DFFTLIB_USE_MKL -DFFTLIB_THREADSAFE
#INCS_FFTLIB = -I./include -I$(MKLROOT)/include/fftw
#LIBS += fftlib
#LLIBS += -ldl

marie-therese.huebsch
Full Member
Full Member
Posts: 175
Joined: Tue Jan 19, 2021 12:01 am

Re: request for functional makefile.include for GPU and 6.3.0

#2 Post by marie-therese.huebsch » Wed Mar 30, 2022 6:39 am

This issue has been resolved here:
Linking error compiling GPU version of Vasp 6.3.0

In summary, the makefile.include file provided in the arch directory is functional. One needs to appropriately set the system library path LD_LIBRARY_PATH, so that the linked libraries can be found at runtime. Additionally, it is necessary to consistently call the same OpenMPI installation (in case multiple installations are present on the same machine).

Cheers,
Marie-Therese

Locked