VASP6.3.2 forrtl: severe (174): SIGSEGV, segmentation fault occurred

Questions regarding the compilation of VASP on various platforms: hardware, compilers and libraries, etc.

Moderators: Global Moderator, Moderator

Locked
Message
Author
jeff_macdonald7
Newbie
Newbie
Posts: 5
Joined: Wed Aug 31, 2022 12:53 pm

VASP6.3.2 forrtl: severe (174): SIGSEGV, segmentation fault occurred

#1 Post by jeff_macdonald7 » Wed Jan 25, 2023 4:44 pm

I used instructions from here to build VASP https://www.intel.com/content/www/us/en ... ilers.html


lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal

1 TB of RAM 10 GB of swap 128 cores.

ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 4126170
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4126170
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

uname -a
Linux hostname 5.15.0-52-generic #58~20.04.1-Ubuntu SMP Thu Oct 13 13:09:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux



head /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 25
model : 1
model name : AMD EPYC 7773X 64-Core Processor
stepping : 2
microcode : 0xa001229
cpu MHz : 2195.894


SCRIPT_THAT_STARTS_MPIRUN & VASP6.3.2
#!/bin/bash
cores=126
executable='/var/tmp/intel_vasp_build/vasp.6.3.2/bin/vasp_std'
source /opt/intel/oneapi/setvars.sh
echo "Running VASP with executable at: "$executable
export OMP_NUM_THREADS=1
/usr/local/spack/opt/spack/linux-ubuntu20.04-zen3/aocc-3.2.0/openmpi-4.1.1-y2ogf3bkk3syj5oiz5ep554fs2kuvjsz/bin/mpirun -np $cores $executable 2>&1 | tee vasp_try${1}.out
wait

source /opt/intel/oneapi/setvars.sh

:: initializing oneAPI environment ...
bash: BASH_VERSION = 5.0.17(1)-release
args: Using "$@" for setvars.sh arguments:
:: advisor -- latest
:: ccl -- latest
:: clck -- latest
:: compiler -- latest
:: dal -- latest
:: debugger -- latest
:: dev-utilities -- latest
:: dnnl -- latest
:: dpcpp-ct -- latest
:: dpl -- latest
:: inspector -- latest
:: ipp -- latest
:: ippcp -- latest
:: ipp -- latest
:: itac -- latest
:: mkl -- latest
:: mpi -- latest
:: tbb -- latest
:: vpl -- latest
:: vtune -- latest
:: oneAPI environment initialized ::

makefile.include started off as a copy of makefile.include.linux_intel

Makefile.include
CPP_OPTIONS = -DHOST=\"LinuxIFC\" \
-DMPI -DMPI_BLOCK=8000 -Duse_collective \
-DscaLAPACK \
-DCACHE_SIZE=4000 \
-Davoidalloc \
-Dvasp6 \
-Duse_bse_te \
-Dtbdyn \
-Dfock_dblbuf
CPP = fpp -f_com=no -free -w0 $*$(FUFFIX) $*$(SUFFIX) $(CPP_OPTIONS)
FC = mpiifort
FCL = mpiifort -mkl=sequential
FREE = -free -names lowercase
FFLAGS = -assume byterecl -w
OFLAG = -O2
OFLAG_IN = $(OFLAG)
DEBUG = -O0 -traceback
OBJECTS = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o /opt/intel/oneapi/mkl/2023.0.0/interfaces/fftw3xf/libfftw3xf_intel.a
OBJECTS_O1 += fftw3d.o fftmpi.o fftmpiw.o
OBJECTS_O2 += fft3dlib.o
CPP_LIB = $(CPP)
FC_LIB = $(FC)
CC_LIB = icc
CFLAGS_LIB = -O
FFLAGS_LIB = -O1
FREE_LIB = $(FREE)
OBJECTS_LIB = linpack_double.o
CXX_PARS = icpc
LLIBS = -lstdc++
VASP_TARGET_CPU ?= -march=core-avx2
FFLAGS += $(VASP_TARGET_CPU) -traceback -debug -g
FCL += -qmkl=sequential
MKLROOT ?= /opt/intel/oneapi/mkl/2023.0.0/lib/intel64

LLIBS += -L$(MKLROOT)/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64
INCS =-I$(MKLROOT)/include/fftw


70% of the time MPIRUN & VASP crash with this error

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libpthread-2.31.s 00007FADD5EA6420 Unknown Unknown Unknown
vasp_std 00000000008BF0D0 hamil_mp_hamiltmu 1626 hamil.F
vasp_std 00000000010BA5A9 david_mp_eddav_ 505 davidson.F
vasp_std 000000000115D063 elmin_ 499 electron.F
vasp_std 0000000001CB3E5F vamp_IP_electroni 5384 main.F
vasp_std 0000000001C8C1CD MAIN__ 3142 main.F
vasp_std 000000000040959D Unknown Unknown Unknown
libc-2.31.so 00007FADD5B75083 __libc_start_main Unknown Unknown
vasp_std 00000000004094BE Unknown Unknown Unknown
--------------------------------------------------------------------------


any help would be appreciated.

martin.schlipf
Global Moderator
Global Moderator
Posts: 455
Joined: Fri Nov 08, 2019 7:18 am

Re: VASP6.3.2 forrtl: severe (174): SIGSEGV, segmentation fault occurred

#2 Post by martin.schlipf » Thu Jan 26, 2023 9:15 am

Does this happen for any system, even very small ones, or do you have one big system that you want to calculate?

andreas.singraber
Global Moderator
Global Moderator
Posts: 231
Joined: Mon Apr 26, 2021 7:40 am

Re: VASP6.3.2 forrtl: severe (174): SIGSEGV, segmentation fault occurred

#3 Post by andreas.singraber » Thu Jan 26, 2023 10:10 am

Hello!

Please try setting

Code: Select all

ulimit -s unlimited
in the shell where your are trying to run VASP, maybe that solves the problem. Background: VASP allocates large amounts of memory on the stack. However, the default limit for stack allocations on Linux systems is often found to be only a few megabytes (8192 kB in your case, see output of ulimit -a). One can lift this restriction by using the command above. Please note that this change is not permanent and is lost as soon as you close the terminal. To make it permanent add the command to your .bashrc or comparable file.

Best,
Andreas Singraber

jeff_macdonald7
Newbie
Newbie
Posts: 5
Joined: Wed Aug 31, 2022 12:53 pm

Re: VASP6.3.2 forrtl: severe (174): SIGSEGV, segmentation fault occurred

#4 Post by jeff_macdonald7 » Thu Jan 26, 2023 12:23 pm

thanks andreas .. I updated /etc/security/limits. conf so stack size was unlimited.... will need to double / triple check my settings when I get back into work today.

jeff_macdonald7
Newbie
Newbie
Posts: 5
Joined: Wed Aug 31, 2022 12:53 pm

Re: VASP6.3.2 forrtl: severe (174): SIGSEGV, segmentation fault occurred

#5 Post by jeff_macdonald7 » Thu Jan 26, 2023 1:15 pm

stack size is unlimited, still SIGSEGV.

jeff_macdonald7
Newbie
Newbie
Posts: 5
Joined: Wed Aug 31, 2022 12:53 pm

Re: VASP6.3.2 forrtl: severe (174): SIGSEGV, segmentation fault occurred

#6 Post by jeff_macdonald7 » Thu Jan 26, 2023 3:10 pm

turns out I lied. PHD im working with had shell open for a long time, and updated /etc/security/limits.conf had no effect on his shell until
he logged out and logged in again. Testing again, this time with stacksize ulimited for sure.

jeff_macdonald7
Newbie
Newbie
Posts: 5
Joined: Wed Aug 31, 2022 12:53 pm

Re: VASP6.3.2 forrtl: severe (174): SIGSEGV, segmentation fault occurred

#7 Post by jeff_macdonald7 » Fri Jan 27, 2023 2:53 pm

setting stacksize to unlimited stopped vasp / mpirun from SIGSEGV'ing. Thanks so much for the help!!!

NOTE: its worth noting that VASP runs faster on AMD cpus when compiled with the intel's compiler than when compiled with AMDs compiler. Dont know why and totally unexpected result.

Locked