Problems installing OpenMPI locally for use with CUDA

Question

Short Version

I have CUDA code I need to run locally. Thus, I am trying to install OpenMPI following the OpenMPI directions. When I try to make my code, I receive very long error output, similar to what is described by the OpenMPI documentation. I tried to reinstall OpenMPI with the documentation's suggested fix, but now I get these errors during installation:

Making all in tools/ompi_info
make[2]: Entering directory '/home/hatfull/Downloads/openmpi-2.1.1/ompi/tools/ompi_info'
  CC       ompi_info.o
  CC       param.o
  CCLD     ompi_info
ld: warning: libimf.so, needed by ../../../ompi/.libs/libmpi.so, not found (try using -rpath or -rpath-link)
ld: warning: libsvml.so, needed by ../../../ompi/.libs/libmpi.so, not found (try using -rpath or -rpath-link)
ld: warning: libirng.so, needed by ../../../ompi/.libs/libmpi.so, not found (try using -rpath or -rpath-link)
ld: warning: libintlc.so.5, needed by ../../../ompi/.libs/libmpi.so, not found (try using -rpath or -rpath-link)
ld: .libs/ompi_info: hidden symbol `__intel_cpu_features_init_x' in /opt/intel/compilers_and_libraries_2017.4.196/linux/compiler/lib/intel64_lin/libirc.a(cpu_feature_disp.o) is referenced by DSO
ld: final link failed: Bad value
Makefile:1785: recipe for target 'ompi_info' failed
make[2]: *** [ompi_info] Error 1
make[2]: Leaving directory '/home/hatfull/Downloads/openmpi-2.1.1/ompi/tools/ompi_info'
Makefile:3353: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/hatfull/Downloads/openmpi-2.1.1/ompi'
Makefile:1806: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

What am I doing wrong? Please help! S.O.S.!

Long Version

I have CUDA code I need to run locally. Thus, I am trying to install OpenMPI following the OpenMPI directions. I have the tar stored as ~/Downloads/openmpi-2.1.1.tar.gz. Hance, I ran

$gunzip -c openmpi-2.1.1.tar.gz | tar xf -
$cd openmpi-2.1.1
$./configure --prefix=/opt/openmpi-2.1.1 &> configure_log1
$sudo make all install &> install_log_take1

with success. Note that I changed the --prefix=/usr/local in the directions to --prefix=/opt/openmpi-2.1.1. I have included configure_log1 and install_log_take1.

When I try to make my code using my makefile makefile.ulfhednar,

$make -f makefile.ulfhednar clean
$make -f makefile.ulfhednar &> make_log1

I receive a very long error output in make_log1 that looks similar to what is described by the OpenMPI documentation. It says the solution is to install OpenMPI with the configure options "./configure CC=icc CXX=icpc F77=ifort FC=ifort ...", so I reinstalled using the commands,

$cd ~/Downloads/openmpi-2.1.1
$sudo make uninstall
$sudo rm -r /opt/openmpi-2.1.1
$cd ..
$sudo rm -r openmpi-2.1.1

$gunzip -c openmpi-2.1.1.tar.gz | tar xf -
$cd openmpi-2.1.1

$which icc
/opt/intel/compilers_and_libraries_2017.4.196/linux/bin/intel64/icc
$which icpc
/opt/intel/compilers_and_libraries_2017.4.196/linux/bin/intel64/icpc
$which ifort
/opt/intel/compilers_and_libraries_2017.4.196/linux/bin/intel64/ifort

$./configure --prefix=/opt/openmpi-2.1.1 CC=/opt/intel/compilers_and_libraries_2017.4.196/linux/bin/intel64/icc CXX=/opt/intel/compilers_and_libraries_2017.4.196/linux/bin/intel64/icpc F77=/opt/intel/compilers_and_libraries_2017.4.196/linux/bin/intel64/ifort FC=/opt/intel/compilers_and_libraries_2017.4.196/linux/bin/intel64/ifort &> configure_log2
$sudo make all install &> install_log_take2

Here is configure_log2 and install_log_take2. Of note in install_log_take2 are the following lines:

Making all in tools/ompi_info
make[2]: Entering directory '/home/hatfull/Downloads/openmpi-2.1.1/ompi/tools/ompi_info'
  CC       ompi_info.o
  CC       param.o
  CCLD     ompi_info
ld: warning: libimf.so, needed by ../../../ompi/.libs/libmpi.so, not found (try using -rpath or -rpath-link)
ld: warning: libsvml.so, needed by ../../../ompi/.libs/libmpi.so, not found (try using -rpath or -rpath-link)
ld: warning: libirng.so, needed by ../../../ompi/.libs/libmpi.so, not found (try using -rpath or -rpath-link)
ld: warning: libintlc.so.5, needed by ../../../ompi/.libs/libmpi.so, not found (try using -rpath or -rpath-link)
ld: .libs/ompi_info: hidden symbol `__intel_cpu_features_init_x' in /opt/intel/compilers_and_libraries_2017.4.196/linux/compiler/lib/intel64_lin/libirc.a(cpu_feature_disp.o) is referenced by DSO
ld: final link failed: Bad value
Makefile:1785: recipe for target 'ompi_info' failed
make[2]: *** [ompi_info] Error 1
make[2]: Leaving directory '/home/hatfull/Downloads/openmpi-2.1.1/ompi/tools/ompi_info'
Makefile:3353: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/hatfull/Downloads/openmpi-2.1.1/ompi'
Makefile:1806: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

Sorry I had to upload all the log files to mediafire, pastebin didn't accept them because they were too large.

What am I doing wrong? Please help! S.O.S.!

Answer 1

I have found the solution!!!

I logged into root and ran through the installation steps without using sudo.

#gunzip -c openmpi-2.1.1.tar.gz | tar xf -
#cd openmpi-2.1.1
#./configure --prefix=/opt/openmpi-2.1.1 CC=icc CXX=icpc FC=ifort
#make all install

I was trying to use OpenMPI with the Intel Composer compilers and this was causing permissions issues. I followed the installation instructions found here, but I got the same problems I had before when trying to install. The issue was that the linker, ld "could not find" the proper libraries because those libraries are no longer provided in the $LD_LIBRARY_PATH variable when sudo is called. The only way to circumvent this is by logging into the root user and setting your $LD_LIBRARY_PATH variable to be identical to the one your normal user has.

I had been stuck on this for days and I'm jumping for joy now that it's sorted out! I hope this helps someone else in the future.