Installing Software: Library errors
April 08 2019
This blog post continues our series on software installation without root permissions. For other blog posts on this topic, look for the list under Software installation in this link.
If you are currently or have been installing programs, it is likely that in one of these attempts you would have run into a library error. This blog post walks through what these errors
So what do library errors look like? Here are a couple of examples to help,
$ pip uninstall drmaa
/N/home/i/u/iugalaxy/Karst/galaxy-17.05/.venv/bin/python2.7: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory
What do you think the error is in the above example?
Here is a hint:" error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory".
Let's take a look at another example,
$ python3 ~/bin/albacore/usr/lib/python3/dist-packages/albacore/pipeline.py
Traceback (most recent call last):
File "/N/u/cganote/Carbonate/bin/albacore/usr/lib/python3/dist-packages/albacore/pipeline.py", line 12, in <module>
from albacore.pipeline_core import PipelineCore, get_debug_level as get_core_debug_level
ImportError: libhdf5_cpp.so.11: cannot open shared object file: No such file or directory
What do you think the error is in the above example?
" ImportError: libhdf5_cpp.so.11: cannot open shared object file: No such file or directory"
In both the examples above, the error suggests the libraries - libpython2.7.so.1.0, libhdf5_cpp.so.11 are not found. This could mean either the library is not installed or that it's not in your path or environment (click here for more information).
Great, now hopefully that confirms the error you are running into is because of a missing library. If it is, let's understand what is happening here, in a little more detail and fix the error.
Why do we run into library errors
It really helps when designing software to reuse stuff written by someone else! That way you don't reinvent the wheel, so to speak. One way to reuse stuff written by someone else is to take their code, copy it all, and paste it right into yours. This is delightful when dealing with 100 lines or so, and a total nightmare at 1,000,000 lines. You can run into issues where you make a mistake copying, or you find out that the other person also has a variable called “i” or a function called “test” and there are conflicts. If the other software comes up with a bug fix or a cool new feature, you would have to update your code to match as well, if you want those nice things.
A way around this is to compile your code into a library that can be shared with other programmers. Shared libraries are files that end in .so and can be included in other C programs. Other languages have this concept too - Python and Perl call them packages or modules. You generally can't share code between programming languages without special help. Interpreted languages, like Perl and Python, create the machine-ready code at the same time it is run, as opposed to having a separate compiling step. This is done by invoking the
Linking and the art of figure-it-out-later
Great, so you can include code from other programs written in your language. In C, there's an additional consideration. Once you pulled the code from this other source into your repertoire, how do you combine these codes,
- do you integrate it into your binary?
- Or do you keep it aside as a reference, so that when your binary is run later, you can go and find it again?
This decision is made when you compile your code - you can choose to keep it flexible with Dynamic linking, or you can choose to package it all together with a Static build of your code. Generally, dynamic linking is preferred. This is because when you give your binary to someone else, they can use their spiffy, optimized-to-all-heck math libraries and make your code run fast on their machine. As another example, if you are including some security-sensitive code in your program, like SSL libraries, you don't want to wrap up the code in your binary that might have critical bug fixes later. If someone hacks the SSL library version that's in your code, you have to recompile it to include a new version. If you linked it dynamically, you just recompile SSL and all the code that uses it will use the updated version! Additionally, the more dependencies you statically link, the bigger your file gets. This might not be an issue for a few things, but if every program you install has its own version of the whole C library inside, that becomes a lot of redundant code you are housing. You will have to weigh these considerations against the convenience of putting it all in one place and distributing it as a complete package. If you want more gory details, read through this nice article.
How to compile dynamic linksUsually, you can add the -fPIC flag to your compiler. You can alter your CFLAGS environment variable to give options to the C compiler, but it's up to the Makefile if it even bothers to look at the environment. The same thing goes for your library paths. Note that CPP does not mean C
A few other things you can try:
- --enable-shared during configure
- CRAYPE_LINK_TYPE=dynamic if you are working on a Cray
- option (BUILD_SHARED_LIB "Build a shared library" ON) in CMakeLists.txt
Here's a blurb you might see at the end of your make install:
- add LIBDIR to the `LD_LIBRARY_PATH' environment variable during execution
- add LIBDIR to the `LD_RUN_PATH' environment variable during linking
- use the `-Wl,-
rpath-Wl,LIBDIR' linker flag
Do those things =)
You can take out the PIC flag, so no -fPIC in your Makefile or environment. Also, try –enable-static in your configure line.
Messing with the linkerIf you want to see which libraries are dynamically linked to a binary, type
If you see ⇒ not found on one of the lines, there's a good chance you'll have trouble running that program; the library isn't being found in your current environment, like the examples mentioned in the above section.
linux-vdso.so.1 => (0x00007fff6510c000)
libbfd-2.27-28.base.el7_5.1.so => /usr/lib64/libbfd-2.27-28.base.el7_5.1.so (0x00002b8ba591e000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x00002b8ba5c65000)
libc.so.6 => /usr/lib64/libc.so.6 (0x00002b8ba5e69000)
When searching for a library you need, you might ask the linker directly. For example, let's say I'm looking for the BLAS math library. The file I need will start with
If you want to link to
ld -lblas --verbose
Once you save that to a file, test compiling it while linking the
printf ("Hello Solar System\n");
If that happens with no error, that means that your library was able to be linked! You can then see where the library was located using
gcc -lblas -o testImport testImport.c
You can use
linux-vdso.so.1 => (0x00007ffd26acc000)
libblas.so.3 => /lib64/libblas.so.3 (0x00002b51542dc000)
libc.so.6 => /lib64/libc.so.6 (0x00002b5154535000)
Note, however, when I invoke
gcc -L/N/u/cganote/Carbonate/bin/rmats -lblas -o testImport testImport.c
linux-vdso.so.1 => (0x00007ffd4d174000)
libblas.so.3 => /lib64/libblas.so.3 (0x00002b0c21066000)
libc.so.6 => /lib64/libc.so.6 (0x00002b0c212bf000)
The location of
I can press the issue on
gcc -Wl,-R/N/u/cganote/Carbonate/bin/rmats -lblas -o testImport testImport.c
linux-vdso.so.1 => (0x00007ffde1d76000)
libblas.so.3 => /N/u/cganote/Carbonate/bin/rmats/libblas.so.3 (0x00002af206980000)
It remembered! However, this can cause issues if you move the directory with
mv rmats rmat5
linux-vdso.so.1 => (0x00007ffd27398000)
libblas.so.3 => /lib64/libblas.so.3 (0x00002aab0ad73000)
Since the -R path with
If you are still running into a library error, email us at help@@ncgas.org.