Confusion about linking boost library while compilation

Question

For a client I needed to add boost 1.54 to the system. So I downloaded the latest version (1.55) and built it within a special directory: /usr/local/lib/boost1.55/. This works. Then I had to adapt the Makefile in this way.

LIBS = $(SUBLIBS) -L/usr/lib/x86_64-linux-gnu -LC:/deps/miniupnpc -lminiupnpc -lqrencode -lrt -LC:/deps/boost/stage/lib -Lc:/deps/db/build_unix -Lc:/deps/ssl -LC:/deps/libqrencode/.libs -lssl -lcrypto -ldb_cxx -L/usr/local/lib/boost1.55/boost_system-mgw46-mt-sd-1_54 -L/usr/local/lib/boost1.55/boost_filesystem-mgw46-mt-sd-1_54 -L/usr/local/lib/boost1.55/boost_program_options-mgw46-mt-sd-1_54 -L/usr/local/lib/boost1.55/boost_thread-mgw46-mt-sd-1_54 -lQtDBus -lQtGui -lQtCore -lpthread -lboost_system -lboost_filesystem -lboost_program_options -lboost_thread

In the unmodified Makefile, the boost linkings looked like this:

-lboost_thread-mgw46-mt-sd-1_54

But this didn't work. I wasn't able to compile it because it was not found. So I added (as you can see above)

-L/usr/local/lib/boost1.55/boost_thread-mgw46-mt-sd-1_54

and

-lboost_thread

Otherwise it wouldn't compile either. After successful compilation, I executed ldd on the binary and it shows me:

libboost_system.so.1.53.0 => /usr/lib/x86_64-linux-gnu/libboost_system.so.1.53.0 (0x00007f416c169000) libboost_filesystem.so.1.53.0 => /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.53.0 (0x00007f416bf52000) libboost_program_options.so.1.53.0 => /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.53.0 (0x00007f416bce4000) libboost_thread.so.1.53.0 => /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.53.0 (0x00007f416bace000)

1.53 is the version installed by package manager. I don't understand why it links to this version. If I wouldn't have installed 1.55, it would not compiled, but now it doesn't link to this version. Any explanation for that?

Actually my target is to not use dynamic linked libraries and I didn't figured out yet how to do that, but I still want to know why the above doesn't work as expected.

Answer 1

You don't say what distribution of Linux this is, but often times there is a directory where you can add dynamically linkable libraries. On Redhat distros such as Fedora this directory is here, /etc/ld.so.conf.d/.

LD

You can add a file to this directory with the path to your newly installed library like so:

$ cat /etc/ld.so.conf.d/myboost.conf
/usr/local/lib/boost1.55

Then run this command:

$ ldconfig -v

This will process all the libraries and rebuild a "cache", /etc/ld.so.cache. This cache is what's used to locate libraries when they're specified like so: -lboost_thread-mgw46-mt-sd-1_54.

Example output

$ ldconfig -v
/usr/lib64/atlas:
        libclapack.so.3 -> libclapack.so.3.0
        libptcblas.so.3 -> libptcblas.so.3.0
        libf77blas.so.3 -> libf77blas.so.3.0
        libcblas.so.3 -> libcblas.so.3.0
        liblapack.so.3 -> liblapack.so.3.0
        libptf77blas.so.3 -> libptf77blas.so.3.0
        libatlas.so.3 -> libatlas.so.3.0
/usr/lib64/wxSmithContribItems:
        libwxflatnotebook.so.0 -> libwxflatnotebook.so.0.0.1
...

When adding paths to the setup, I like to confirm by going through this output to make sure things are getting picked up the way I expect them to.

LD's cache

You can always print the contents of the .cache file as well using this command:

$ ldconfig -p | head -10
2957 libs found in cache `/etc/ld.so.cache'
    lib3ds-1.so.3 (libc6,x86-64) => /usr/lib64/lib3ds-1.so.3
    libzvbi.so.0 (libc6,x86-64) => /usr/lib64/libzvbi.so.0
    libzvbi-chains.so.0 (libc6,x86-64) => /usr/lib64/libzvbi-chains.so.0
    libzrtpcpp-1.4.so.0 (libc6,x86-64) => /usr/lib64/libzrtpcpp-1.4.so.0
    libzmq.so.1 (libc6,x86-64) => /usr/lib64/libzmq.so.1
    libzmq.so (libc6,x86-64) => /usr/lib64/libzmq.so
    libzipios.so.0 (libc6,x86-64) => /usr/lib64/libzipios.so.0
    libzipios.so (libc6,x86-64) => /usr/lib64/libzipios.so
    libzip.so.1 (libc6,x86-64) => /usr/lib64/libzip.so.1

Why's my ldd output still using 1.53?

This is because your binary is using dynamic libraries. So when the binary was compiled, it was against the 1.55 versions of the libraries. However when you interrogate the binary using ldd, it's within the environment that's using the contents of the .cache file. So the library within the cache that is associated with the symbols used by this binary match those for 1.53, hence you're seeing those libraries.

Your environment knows nothing of the 1.55 libraries, only your build environment, i.e. your Makefile, is aware of this.

Dynamic libraries

Think of the functions within these as symbols. The symbols are a name, and so these names often don't change from one version of a library to another. So if you were to look at a library such as boost, you can use the tool readelf to get a list of the symbols within one of these .so files.

Example

$ readelf -Ws /usr/lib64/libboost_date_time-mt.so | head -10

Symbol table '.dynsym' contains 261 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND 
     1: 000000335aa096a8     0 SECTION LOCAL  DEFAULT    9 
     2: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSt8bad_castD2Ev@GLIBCXX_3.4 (2)
     3: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSt6locale5_ImplD1Ev@GLIBCXX_3.4 (2)
     4: 0000000000000000     0 OBJECT  GLOBAL DEFAULT  UND _ZTINSt6locale5facetE@GLIBCXX_3.4 (2)
     5: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND wcslen@GLIBC_2.2.5 (3)
     6: 0000000000000000     0 OBJECT  GLOBAL DEFAULT  UND _ZTISt11logic_error@GLIBCXX_3.4 (2)

In the above output you can see some of the FUNC definitions, these are the names that are used to "link" a function in an executable with a function from some .so library.

I'm over simplifying this, and probably explaining things slightly off, but I'm only trying to give you a general idea of how the mechanics of how the machinery under the hood works.

References

• How do I list the symbols in a .so file

Answer 2

When compiling software that uses dynamic libraries, unless you hard-code an RPATH into the compiled binary, when you run the executable (or run ldd against it), it looks for the libraries in the defined search paths for your OS (often set in /etc/ld.so.conf).

If you have compiled your own software and installed it outside of the library search paths, you'll need to compile the binaries with a hard-coded RPATH. This is done by passing -rpath to your linker. However, most of the time, your Makefile doesn't run the linker directly, you need to tell gcc to pass it to the linker, typically by adding -Wl,-rpath,/usr/local/whatever/lib to the CFLAGS.

If you don't want to hard-code paths, you can update your $LD_LIBRARY_PATH environment variable to have a comma delimited collection of library directories you want to prepend to the runtime library search path. Setting $LD_LIBRARY_PATH is ephemeral, it does nothing to the binary, so you'd need to set it any time you need to run the executable. (See Environment Modules if you want to do this in a regular manner)