10

Our computation cluster runs a very old version of CentOS, with an old Kernel (2.6.18) and, of course, old libs and binaries. Because updating the whole thing requires a lot of work on all the nodes, this is not an option.

I am trying to compile and use a program that requires C++11 and therefore newer versions of gcc (and/or clang). Because I do not want to mess around with the system at all, I want to do this as a non-root user in some local directory tree.

The problem is, that gcc requires a newer glibc than the one that is present on the machine(s). Hence, I need to maintain a separate, newer version of glibc in my local lib/ tree, probably as described here.

Where I am lost is, how do I "hardcode" the paths of my local libs into all the required binaries, i.e., gcc, g++ etc.? Setting LD_LIBRARY_PATH to my local lib/ tree causes all system binaries not to work anymore (ELF file OS ABI invalid) because they want to use my new libm.so/libc.so against which they have not been compiled.

So, to wrap it up: What is the proper way to maintain a newer, local development stack (containing glibc, gcc etc.) in parallel to an old system without messing around as root?

As a side question: Setting LD_LIBRARY_PATH is posted as a solution all over SE when it comes to separate glibc. For me, it causes the errors above when I try to execute any system binary (like ls). How come? Did I do something wrong or is this the intended behaviour?

1 Answer 1

10

You have basically three options:

  1. Use a wrapper around your libraries, that will set LD_LIBRARY_PATH appropriately and then execute the desired library - something like:

    #!/bin/sh
    export LD_LIBRARY_PATH="path/goes/here"
    exec "$@"
    
  2. link with -rpath (-Wl,rpath) which adds search path for dynamic linker into the binary (see also SO answer - it also mentions the wrapper).

  3. You won't like reading this one: update your cluster (note the emphasis on "your"). It will have to be done one day or another, so why not today. "Not an option" is a bit strong in most cases. Other users probably have the same issues.

As for the old binaries having problems - binaries have their preferred dynamic linker embedded in them. And the old dynamic linker doesn't understand the newer ABI. Try calling the binaries like this: path/to/your/ld-linux-<arch>.so binary.

Building GCC: you can always try exporting CFLAGS in the GCC's build environment - but I'm sure they get propagated. Buildscripts of various distributions may give you some clues (e.g.: for openSUSE look around line 1880 in the .spec file).

6
  • Hi peterph, thanks for the answer. I prefer option 2. However, how would I hardcode the rpath into, for example, gcc, without having to change the Makefile etc.? As for updating the cluster: This of course is on our agenda, but right now too many people are using it to justify longer downtimes. Also, for updating it we are going to hire the company again that set it up originally. There is nobody in our group that has sufficient knowledge and experience for this.
    – janoliver
    May 27, 2014 at 9:34
  • For GCC see update of my answer. For cluster upgrade my personal recommendation is to get an internal (or at least long-term external) guru. It's usually cheaper and more flexible in the longer run than just outsourcing it completely. The most important advantage is getting the knowledge home through interactions between system admin and users.
    – peterph
    May 27, 2014 at 9:49
  • As a small research group in the hopelessly underfinanced, German scientific system, we are unfortunately not able to hire somebody for this task or invest the time ourselves. You may be right, that in the long run this might pay off, but right now the budgets are too constrained for something like this. This is the downside of not being in a company. ;)
    – janoliver
    May 27, 2014 at 9:56
  • Get IT/CS students from your university - a good one might even be able to use it for his/her BSc/Msc thesis. And regarding finances: be glad you are in Germany and not in one of your eastern neighbours (maybe with the exception of Austria). :)
    – peterph
    May 27, 2014 at 10:53
  • As an alternative to passing -rpath (-Wl,rpath) to the linker, the linker will also add the paths from the LD_RUN_PATH environment variable to the search path
    – rakslice
    Oct 15, 2015 at 7:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .