3

I know that snipersim isn't a very typical "project" but this is more a linux/linking problem than anything else, so I think it goes here. I have also contacted the developers, but have yet to receive an answer.

First, for quick explanation of what I'm trying to do:

For my master thesis I am using the architectural simulator snipersim (http://www.snipersim.org). I downloaded it to my local machine (running Linux Mint 17.2), built it, and started working with it. Everything works perfectly fine.

Seeing as I need to make a few hundred simulations, each of which takes hours, I was given access to an university computing cluster using HTCondor, on x86_64 OpenSUSE 13.1 machines. Obviously, I do not have root access to the cluster.

Due to them having different distributions, I can't simply copy over the binaries (I ended up trying later, but the code behaves erratically), so I wanted to recompile snipersim.

My compilation process

In the cluster's access machine (from where you can submit the parallel jobs using condor_submit), I cloned my snipersim fork.

SQLite3

I checked whether the necessary libraries were installed, and noticed libsqlite3 was missing. In order to fix this, I downloaded sqlite-autoconf-3090200.tar.gz from the SQLite.org website, configured it to install to ~/sqlite (./configure -prefix=~/sqlite), and did make && make install.

I then configured SQLITE_PATH to point to ~/sqlite, and both LIBRARY_PATH and LD_LIBRARY_PATH to point to ~/sqlite/lib

So far, so good.

(For future reference, yes, SQLite3 was compiled with -fPIC)

Sniper

With all the libraries out of the way, I set out to compile sniper. Changed to the main directory, and typed make. Everything seems fine, exactly like on my home machine. It goes through all the dependencies, source files, etc, and then gets to the last step, which is the linking of the main executable sniper. Here, it suddenly gives an error, and halts:

[LD    ] lib/sniper
/usr/lib/gcc/x86_64-pc-linux-gnu/4.8.4/../../../../x86_64-pc-linux-gnu/bin/ld: ~/sniper/standalone/../standalone/standalone.o: relocation R_X86_6ldrelocationssnipecompilation4_32S against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
~/sniper/standalone/../standalone/standalone.o: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
Makefile:34: recipe for target '~/sniper/standalone/../lib/sniper' failed
make: *** [~/sniper/standalone/../lib/sniper] Error 1

This error message has me stumped. Everything is compiled with -fPIC, so the error must have to do with some libraries that get pulled in from outside.

The command executed here for the linking (for those of you who never used sniper) (It actually uses full paths, instead of '~', but I replaced those with '~' as they contain a ton of personally identifying information):

g++ -L~/sniper/standalone/../lib -L~/sniper/standalone/../sift -L~/sniper/standalone/../pin_kit/extras/xed-intel64/lib -L~/sqlite/lib -L~/sniper/standalone/../pin_kit/extras/xed2-intel64/lib -o ~/sniper/standalone/../lib/sniper ~/sniper/standalone/../standalone/exceptions.o ~/sniper/standalone/../standalone/standalone.o -lcarbon_sim -lpthread -lsift -lxed -L~/sniper/standalone/../python_kit/intel64/lib -lpython2.7 -lrt -lz -lsqlite3 -lxed -O2 -g -std=c++0x

In particular, -lcarbon_sim -sift reference custom libraries that were compiled previously during the make process, and -lxed references the Intel PIN library (used for Processor instrumentation).

My thoughts / What I've tried

This is a surprising error message. For some reason, it's stating that standalone.o is position dependent and can't be compiled into a shared object, when all the compilation steps have -fPIC (I triple-checked). Again, the only thing that comes to mind is that one of the libraries being pulled in was compiled without -fPIC, which is unlikely, I must be overlooking something. Is there any way to have ld print a list of all the libraries it pulls in, as it does so? If I could figure out where the problem lies, I might be able to pull in a manually-compiled library, for example.

In addition, I checked relocations on my home machine, and the exact same relocation ld complains about (R_X86_64_32S against .rodata.str1.1) exists in the standalone.o file, but everything works fine there.

I thought that maybe it could be due to my custom install of SQLite3 (I had it installed through a package manager in my home machine), as such I tried installing a copy on my home machine through the exact same process as on the cluster. Everything still works, and I confirmed through ldd that it was actually linking against my copy (instead of the system version).

I also compared gcc, g++ and ld versions between the two machines, and they match.

In addition, one weird thing I have noticed: The file ~/sniper/lib/pin_sim.so (compiled from some sniper code that pulls in the Intel PIN library) is a 64-bit dynamically linked ELF executable (as expected), but running ldd pin_sim.so simply prints not a dynamic executable, while in my home machine it prints all the used shared libraries. I tried copying pin_sim.so from my computer to the cluster, and ldd is also not able to read it. readelf -d pin_sim.so still works on both machines.

This is very weird. The only reference I could find to ldd failing when readelf/objdump don't was when calling it on a 32-bit executable under a 64-bit system. In this case, both the executable and the system are 64-bit, so that's not it.

I'm completely out of ideas on what to do. I spent about 5 hours today scouring the web for solutions to similar issues and trying them all, to no avail. Hopefully someone here has some ideas?

Edit 1: Comparison of linker parameters between both machines

As suggested by siblynx, I ran g++ on both machines using the -v parameter to try and figure out what the differences were when it calls the linker (collect2). I cleaned them up slightly (only the relevant file names, not the whole path), and removed the library directories (-L):

Common Parameters:

--eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o lib/sniper
crti.o crtn.o
standalone/exceptions.o standalone/standalone.o
-lcarbon_sim -lpthread -lsift -lxed -lpython2.7 -lrt -lz -lsqlite3 -lxed -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc

Parameters unique to my home machine (Linux Mint 17.2, where sniper works):

--sysroot=/ --build-id -z relro
crt1.o crtbegin.o crtend.o

Parameters unique to the cluster machine (where it doesn't work):

-pie -z now
Scrt1.o crtbeginS.o crtendS.o

I'm not very knowledgeable when it comes to linking, but I did notice that my home setup includes crt1.o crtbegin.o crtend.o, while the cluster includes Scrt1.o crtbeginS.o crtendS.o (notice the extra 'S'). What exactly do these files do, and what does the S in the filename mean? (one of "Shared" or "Static", I assume?)

Improve this question
11
  • 1
    You can try running g++ commandline with -v flag to see what exactly it runs during link time. There are usually hidden objects involved, and they maybe erroneus too, although unlikely, because ld clearly complains about standalone.o.
    – user140866
    Nov 13, 2015 at 4:35
  • Just did a comparison between the g++ linker calls on both my home machine and the cluster machine, and added it to my main post. Nothing particular stands out to me, but I'm not very knowledgeable about linking. Any ideas?
    – ruipin
    Nov 13, 2015 at 5:09
  • 1
    Well first I'd try to do manual linking with flags that are working, i.e. from your home installation. At least that can show off another errors that can lead to more precise comparison of your two build environments. And it's right that -pie only used on cluster? By the way what stops you from copying result binaries from your host to cluster? Different glibc versions?
    – user140866
    Nov 13, 2015 at 5:18
  • 1
    And (just guess!) probably opensuse enforces pie executables while your Mint don't. You can verify that by running file $(which gcc) on both machines and pasting result here.
    – user140866
    Nov 13, 2015 at 5:27
  • 1
    @rpinheiro if you solved the problem you should post it as an answer (and accept it when you can) instead of editing it into the question. This is so if someone else has the same issue they can see that this problem has a solution.
    – casey
    Nov 13, 2015 at 14:22

1 Answer 1

Reset to default
1

Cause

It seems the '-pie' parameter breaks Sniper compilation. I tried adding it to my home machine and it fails with the exact same error. Removing it from the cluster line and the linker succeeds.

As the user siblynx mentioned, OpenSUSE (at least the one in the cluster) enforces executables to use PIE when being linked, while Linux Mint doesn't.

Solution

Simply adding -fno-pie to the $(TARGET) linker call in the file standalone/Makefile overrides the COLLECT_GCC_OPTIONS -pie and everything seems to work correctly.

ldd pin_sim.so still doesn't work, but that's a different issue entirely. I might post a separate question for this, actually.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .