I know that snipersim isn't a very typical "project" but this is more a linux/linking problem than anything else, so I think it goes here. I have also contacted the developers, but have yet to receive an answer.
First, for quick explanation of what I'm trying to do:
For my master thesis I am using the architectural simulator snipersim (http://www.snipersim.org). I downloaded it to my local machine (running Linux Mint 17.2), built it, and started working with it. Everything works perfectly fine.
Seeing as I need to make a few hundred simulations, each of which takes hours, I was given access to an university computing cluster using HTCondor, on x86_64 OpenSUSE 13.1 machines. Obviously, I do not have root access to the cluster.
Due to them having different distributions, I can't simply copy over the binaries (I ended up trying later, but the code behaves erratically), so I wanted to recompile snipersim.
My compilation process
In the cluster's access machine (from where you can submit the parallel jobs using condor_submit), I cloned my snipersim fork.
I checked whether the necessary libraries were installed, and noticed libsqlite3 was missing. In order to fix this, I downloaded sqlite-autoconf-3090200.tar.gz from the SQLite.org website, configured it to install to ~/sqlite (./configure -prefix=~/sqlite), and did
make && make install.
I then configured
SQLITE_PATH to point to
~/sqlite, and both
LD_LIBRARY_PATH to point to
So far, so good.
(For future reference, yes, SQLite3 was compiled with -fPIC)
With all the libraries out of the way, I set out to compile sniper. Changed to the main directory, and typed
make. Everything seems fine, exactly like on my home machine. It goes through all the dependencies, source files, etc, and then gets to the last step, which is the linking of the main executable
sniper. Here, it suddenly gives an error, and halts:
[LD ] lib/sniper /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.4/../../../../x86_64-pc-linux-gnu/bin/ld: ~/sniper/standalone/../standalone/standalone.o: relocation R_X86_6ldrelocationssnipecompilation4_32S against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC ~/sniper/standalone/../standalone/standalone.o: error adding symbols: Bad value collect2: error: ld returned 1 exit status Makefile:34: recipe for target '~/sniper/standalone/../lib/sniper' failed make: *** [~/sniper/standalone/../lib/sniper] Error 1
This error message has me stumped. Everything is compiled with -fPIC, so the error must have to do with some libraries that get pulled in from outside.
The command executed here for the linking (for those of you who never used sniper) (It actually uses full paths, instead of '~', but I replaced those with '~' as they contain a ton of personally identifying information):
g++ -L~/sniper/standalone/../lib -L~/sniper/standalone/../sift -L~/sniper/standalone/../pin_kit/extras/xed-intel64/lib -L~/sqlite/lib -L~/sniper/standalone/../pin_kit/extras/xed2-intel64/lib -o ~/sniper/standalone/../lib/sniper ~/sniper/standalone/../standalone/exceptions.o ~/sniper/standalone/../standalone/standalone.o -lcarbon_sim -lpthread -lsift -lxed -L~/sniper/standalone/../python_kit/intel64/lib -lpython2.7 -lrt -lz -lsqlite3 -lxed -O2 -g -std=c++0x
-lcarbon_sim -sift reference custom libraries that were compiled previously during the
make process, and
-lxed references the Intel PIN library (used for Processor instrumentation).
My thoughts / What I've tried
This is a surprising error message. For some reason, it's stating that standalone.o is position dependent and can't be compiled into a shared object, when all the compilation steps have -fPIC (I triple-checked). Again, the only thing that comes to mind is that one of the libraries being pulled in was compiled without -fPIC, which is unlikely, I must be overlooking something. Is there any way to have ld print a list of all the libraries it pulls in, as it does so? If I could figure out where the problem lies, I might be able to pull in a manually-compiled library, for example.
In addition, I checked relocations on my home machine, and the exact same relocation ld complains about (
R_X86_64_32S against .rodata.str1.1) exists in the standalone.o file, but everything works fine there.
I thought that maybe it could be due to my custom install of SQLite3 (I had it installed through a package manager in my home machine), as such I tried installing a copy on my home machine through the exact same process as on the cluster. Everything still works, and I confirmed through ldd that it was actually linking against my copy (instead of the system version).
I also compared gcc, g++ and ld versions between the two machines, and they match.
In addition, one weird thing I have noticed: The file
~/sniper/lib/pin_sim.so (compiled from some sniper code that pulls in the Intel PIN library) is a 64-bit dynamically linked ELF executable (as expected), but running
ldd pin_sim.so simply prints
not a dynamic executable, while in my home machine it prints all the used shared libraries. I tried copying
pin_sim.so from my computer to the cluster, and ldd is also not able to read it.
readelf -d pin_sim.so still works on both machines.
This is very weird. The only reference I could find to ldd failing when readelf/objdump don't was when calling it on a 32-bit executable under a 64-bit system. In this case, both the executable and the system are 64-bit, so that's not it.
I'm completely out of ideas on what to do. I spent about 5 hours today scouring the web for solutions to similar issues and trying them all, to no avail. Hopefully someone here has some ideas?
Edit 1: Comparison of linker parameters between both machines
As suggested by siblynx, I ran g++ on both machines using the
-v parameter to try and figure out what the differences were when it calls the linker (collect2). I cleaned them up slightly (only the relevant file names, not the whole path), and removed the library directories (-L):
--eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o lib/sniper crti.o crtn.o standalone/exceptions.o standalone/standalone.o -lcarbon_sim -lpthread -lsift -lxed -lpython2.7 -lrt -lz -lsqlite3 -lxed -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc
Parameters unique to my home machine (Linux Mint 17.2, where sniper works):
--sysroot=/ --build-id -z relro crt1.o crtbegin.o crtend.o
Parameters unique to the cluster machine (where it doesn't work):
-pie -z now Scrt1.o crtbeginS.o crtendS.o
I'm not very knowledgeable when it comes to linking, but I did notice that my home setup includes
crt1.o crtbegin.o crtend.o, while the cluster includes
Scrt1.o crtbeginS.o crtendS.o (notice the extra 'S'). What exactly do these files do, and what does the S in the filename mean? (one of "Shared" or "Static", I assume?)