3

(This question is an indirect continuation of my previous question which has been solved.)

Small summary of what I'm trying to do

I've got a .so file named pin_sim.so for the architectural simulator snipersim (http://www.snipersim.org). This library basically implements a wrapper around Intel's PIN instrumentation library used by the simulation.

This file is dynamically loaded before the start of the simulation by the main application. As the simulation target executable must be loaded with the same library versions as pin_sim.so, and pin_sim.so packages some versions of the libraries, before a simulation starts, it checks which version of libraries are loaded by pin_sim.so using ldd.

Problem

On my home desktop (running Linux Mint 17.2), this works correctly. ldd pin_sim.so returns:

linux-vdso.so.1 =>  (0x00007ffcd75a3000)
libpython2.7.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 (0x00007fdd19d08000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fdd19b00000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fdd198e6000)
libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007fdd1962d000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fdd19429000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fdd19124000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fdd18e1e000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fdd18c08000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdd18842000)
/lib64/ld-linux-x86-64.so.2 (0x00007fdd1ae55000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fdd18624000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fdd18421000)

However, on a computing cluster I have access to which runs OpenSUSE 13.1, this command returns (no matter if I copy over the file, or re-compile it):

not a dynamic executable

In addition, I am unable to run sniper correctly (the simulation never executes, which I assume has to do with pin_sim.so not loading correctly).

If I compile the file in the cluster, and then copy to my home machine, ldd works correctly as before. All other files I've tried ldd on (even the main sniper binary) work correctly, only pin_sim.so fails.

More information

file pin_sim.so on my home machine (same on the cluster, if not recompiled):

pin_sim.so: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, BuildID[sha1]=4ac6397fefb655c475b5a941cef726183d3e3710, not stripped

file pin_sim.so on the cluster (re-compiled):

pin_sim.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

objdump -p pin_sim.so | grep NEEDED on both machines (even if recompiled on the cluster):

NEEDED               libpython2.7.so.1.0
NEEDED               librt.so.1
NEEDED               libz.so.1
NEEDED               libsqlite3.so.0
NEEDED               libdl.so.2
NEEDED               libstdc++.so.6
NEEDED               libm.so.6
NEEDED               libgcc_s.so.1
NEEDED               libc.so.6
NEEDED               ld-linux-x86-64.so.2

readelf -d pin_sim.so | grep NEEDED on both machines (even if recompiled on the cluster):

0x0000000000000001 (NEEDED)             Shared library: [libpython2.7.so.1.0]
0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
0x0000000000000001 (NEEDED)             Shared library: [libsqlite3.so.0]
0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]

ldd --version on my home machine:

ldd (Ubuntu EGLIBC 2.19-0ubuntu6.6) 2.19

ldd --version on the cluster:

ldd (Gentoo 2.20-r2 p4) 2.20

What I've tried / My thoughts

I'm not very familiar with the linux dynamic linking process, so I haven't tried much.

I looked around the web, and I could only find references to ldd being mistaken on very old binaries (compiled before ldd support was added to the ELF format), or when running ldd on 32-bit binaries under a 64-bit system without the 32-bit system libraries installed. Since this binary was compiled by me, and is 64-bit, I have no idea why ldd might be failing.

I'm thinking of switching the version checker in sniper from using ldd to using readelf/objdump instead, but I'm still surprised that ldd fails like this.

Any ideas? I do not have root access in the cluster.

Edit 1: Strace

I did two syscall traces on the cluster machines. One on the sniper binary (where ldd succeeds) and one on the pin_sim.so binary (where ldd fails).

I cut off the beginning, until the interesting part where they start to diverge.

EDIT 2: Now strace -f

strace -f ldd sniper:

faccessat(AT_FDCWD, "./sniper", R_OK)   = 0
faccessat(AT_FDCWD, "./sniper", X_OK)   = 0
faccessat(AT_FDCWD, "/lib/ld-linux.so.2", X_OK) = -1 ENOENT (No such file or directory)
faccessat(AT_FDCWD, "/lib64/ld-linux-x86-64.so.2", X_OK) = 0
pipe([3, 4])                            = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
lseek(255, -52, SEEK_CUR)               = 5706
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x359287389d0) = 21112
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigaction(SIGCHLD, {0x6f679b3710, [], SA_RESTORER|SA_RESTART, 0x35927cffc20}, {0x6f679b3710, [], SA_RESTORER|SA_RESTART, 0x35927cffc20}, 8) = 0
close(4)                                = 0
read(3, Process 21112 attached
 <unfinished ...>
[pid 21112] close(255)                  = 0
[pid 21112] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 21112] rt_sigaction(SIGTSTP, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, {SIG_DFL, [], 0}, 8) = 0
[pid 21112] rt_sigaction(SIGTTIN, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, {SIG_DFL, [], 0}, 8) = 0
[pid 21112] rt_sigaction(SIGTTOU, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, {SIG_DFL, [], 0}, 8) = 0
[pid 21112] rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, 8) = 0
[pid 21112] rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, {SIG_IGN, [], SA_RESTORER, 0x35927cffc20}, 8) = 0
[pid 21112] rt_sigaction(SIGCHLD, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x35927cffc20}, {0x6f679b3710, [], SA_RESTORER|SA_RESTART, 0x35927cffc20}, 8) = 0
[pid 21112] rt_sigaction(SIGCHLD, {0x6f679b3710, [], SA_RESTORER|SA_RESTART, 0x35927cffc20}, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x35927cffc20}, 8) = 0
[pid 21112] rt_sigaction(SIGINT, {0x6f679ce1a0, [], SA_RESTORER, 0x35927cffc20}, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, 8) = 0
[pid 21112] dup2(4, 1)                  = 1
[pid 21112] close(4)                    = 0
[pid 21112] close(3)                    = 0
[pid 21112] rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
[pid 21112] rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, {0x6f679ce1a0, [], SA_RESTORER, 0x35927cffc20}, 8) = 0
[pid 21112] rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, 8) = 0
[pid 21112] rt_sigaction(SIGCHLD, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x35927cffc20}, {0x6f679b3710, [], SA_RESTORER|SA_RESTART, 0x35927cffc20}, 8) = 0
[pid 21112] execve("/lib64/ld-linux-x86-64.so.2", ["/lib64/ld-linux-x86-64.so.2", "--verify", "./sniper"], [/* 51 vars */]) = 0
[pid 21112] brk(0)                      = 0x2d9db43c910
[pid 21112] open("./sniper", O_RDONLY|O_CLOEXEC) = 3
[pid 21112] read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\2\0>\0\1\0\0\0\321\307@\0\0\0\0\0"..., 832) = 832
[pid 21112] fstat(3, {st_mode=S_IFREG|0775, st_size=46853390, ...}) = 0
[pid 21112] getcwd("/path/to/sniper/lib", 128) = 52
[pid 21112] mmap(0x400000, 3837952, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x400000
[pid 21112] mmap(0x9a8000, 626688, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3a8000) = 0x9a8000
[pid 21112] mmap(0xa41000, 32872, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xa41000
[pid 21112] close(3)                    = 0
[pid 21112] exit_group(0)               = ?
[pid 21111] <... read resumed> "", 128) = 0
[pid 21112] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=21112, si_uid=169971, si_status=0, si_utime=0, si_stime=0} ---
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 21112
wait4(-1, 0x3e3bd435d80, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn()                          = 0
close(3)                                = 0
*** TRACE CONTINUES, no longer important because the other one has already errored out by this point ***

strace ldd -f pin_sim.so:

faccessat(AT_FDCWD, "./pin_sim.so", R_OK) = 0
faccessat(AT_FDCWD, "./pin_sim.so", X_OK) = 0
faccessat(AT_FDCWD, "/lib/ld-linux.so.2", X_OK) = -1 ENOENT (No such file or directory)
faccessat(AT_FDCWD, "/lib64/ld-linux-x86-64.so.2", X_OK) = 0
pipe([3, 4])                            = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
lseek(255, -52, SEEK_CUR)               = 5706
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x36aaed959d0) = 21099
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigaction(SIGCHLD, {0x798dd3a710, [], SA_RESTORER|SA_RESTART, 0x36aae35cc20}, {0x798dd3a710, [], SA_RESTORER|SA_RESTART, 0x36aae35cc20}, 8) = 0
close(4)                                = 0
read(3, Process 21099 attached
 <unfinished ...>
[pid 21099] close(255)                  = 0
[pid 21099] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 21099] rt_sigaction(SIGTSTP, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, {SIG_DFL, [], 0}, 8) = 0
[pid 21099] rt_sigaction(SIGTTIN, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, {SIG_DFL, [], 0}, 8) = 0
[pid 21099] rt_sigaction(SIGTTOU, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, {SIG_DFL, [], 0}, 8) = 0
[pid 21099] rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, 8) = 0
[pid 21099] rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, {SIG_IGN, [], SA_RESTORER, 0x36aae35cc20}, 8) = 0
[pid 21099] rt_sigaction(SIGCHLD, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x36aae35cc20}, {0x798dd3a710, [], SA_RESTORER|SA_RESTART, 0x36aae35cc20}, 8) = 0
[pid 21099] rt_sigaction(SIGCHLD, {0x798dd3a710, [], SA_RESTORER|SA_RESTART, 0x36aae35cc20}, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x36aae35cc20}, 8) = 0
[pid 21099] rt_sigaction(SIGINT, {0x798dd551a0, [], SA_RESTORER, 0x36aae35cc20}, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, 8) = 0
[pid 21099] dup2(4, 1)                  = 1
[pid 21099] close(4)                    = 0
[pid 21099] close(3)                    = 0
[pid 21099] rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
[pid 21099] rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, {0x798dd551a0, [], SA_RESTORER, 0x36aae35cc20}, 8) = 0
[pid 21099] rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, 8) = 0
[pid 21099] rt_sigaction(SIGCHLD, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x36aae35cc20}, {0x798dd3a710, [], SA_RESTORER|SA_RESTART, 0x36aae35cc20}, 8) = 0
[pid 21099] execve("/lib64/ld-linux-x86-64.so.2", ["/lib64/ld-linux-x86-64.so.2", "--verify", "./pin_sim.so"], [/* 51 vars */]) = 0
[pid 21099] brk(0)                      = 0x2b0b251b090
[pid 21099] open("./pin_sim.so", O_RDONLY|O_CLOEXEC) = 3
[pid 21099] read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240@\24\0\0\0\0\0"..., 832) = 832
[pid 21099] fstat(3, {st_mode=S_IFREG|0775, st_size=55707995, ...}) = 0
[pid 21099] getcwd("/path/to/sniper/lib", 128) = 52
[pid 21099] mmap(NULL, 12474024, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2b0af187000
[pid 21099] mprotect(0x2b0afa9b000, 2093056, PROT_NONE) = 0
[pid 21099] mmap(0x2b0afc9a000, 716800, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x913000) = 0x2b0afc9a000
[pid 21099] mmap(0x2b0afd49000, 145064, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2b0afd49000
[pid 21099] mprotect(0x3a0fbff6000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC|PROT_GROWSDOWN) = -1 EACCES (Permission denied)
[pid 21099] close(3)                    = 0
[pid 21099] exit_group(1)               = ?
[pid 21099] +++ exited with 1 +++
<... read resumed> "", 128)             = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=21099, si_uid=169971, si_status=1, si_utime=0, si_stime=1} ---
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], WNOHANG, NULL) = 21099
wait4(-1, 0x3de445c7ac0, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn()                          = 0
close(3)                                = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x798dd36e90, [], SA_RESTORER, 0x36aae35cc20}, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, 8) = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, {0x798dd36e90, [], SA_RESTORER, 0x36aae35cc20}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
faccessat(AT_FDCWD, "/libx32/ld-linux-x32.so.2", X_OK) = -1 ENOENT (No such file or directory)
fstat(1, {st_mode=S_IFCHR|0622, st_rdev=makedev(136, 0), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x36aaed99000
write(1, "\tnot a dynamic executable\n", 26 not a dynamic executable
) = 26
read(255, "\nexit $result\n# Local Variables:"..., 5758) = 52
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
exit_group(1)                           = ?
+++ exited with 1 +++

Edit 3

readelf -Wl pin_sim.so:

Elf file type is DYN (Shared object file)
Entry point 0x1440a0
There are 7 program headers, starting at offset 64

Program Headers:
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  LOAD           0x000000 0x0000000000000000 0x0000000000000000 0x913415 0x913415 R E 0x200000
  LOAD           0x913990 0x0000000000b13990 0x0000000000b13990 0x0ae648 0x0d1d18 RW  0x200000
  DYNAMIC        0x9bf0f8 0x0000000000bbf0f8 0x0000000000bbf0f8 0x0002a0 0x0002a0 RW  0x8
  GNU_EH_FRAME   0x7fbee0 0x00000000007fbee0 0x00000000007fbee0 0x036cfc 0x036cfc R   0x4
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RWE 0x10
  GNU_RELRO      0x913990 0x0000000000b13990 0x0000000000b13990 0x0ac670 0x0ac670 R   0x1
  PAX_FLAGS      0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000     0x8

 Section to Segment mapping:
  Segment Sections...
   00     .hash .dynsym .dynstr .gnu.version .gnu.version_d .gnu.version_r .rela.dyn .rela.plt .init .plt .text .fini .rodata .charmversion .eh_frame_hdr .eh_frame .gcc_except_table 
   01     .init_array .fini_array .jcr .data.rel.ro .dynamic .got .data .pinclientint .bss 
   02     .dynamic 
   03     .eh_frame_hdr 
   04     
   05     .init_array .fini_array .jcr .data.rel.ro .dynamic .got 
   06
  • Does ldd work when run against each of the NEEDED libraries on the cluster? – drewbenn Nov 13 '15 at 18:51
  • ...and does ld pin_sim.so warn about any missing symbols? – drewbenn Nov 13 '15 at 19:19
  • ldd seems to work on all NEEDED libraries; ld pin_sim.so says: "ld: warning: cannot find entry symbol _start; not setting start address" (which I think is normal, as it's a library, not an actual executable) – rpinheiro Nov 13 '15 at 20:34
  • 1
    Well, I was thinking strace ldd pin_sim.so and see where ldd is puking. (And, just a dumb double-check, but the SUSE machines are all 64-bit too, right?) – drewbenn Nov 13 '15 at 20:56
  • 1
    Compiled with an incompatible libc. I don't know why the libc would be incompatible though. – Gilles Nov 13 '15 at 23:25
2

In second strace part that tells about x32 binaries is strange, but unlikely:

faccessat(AT_FDCWD, "/libx32/ld-linux-x32.so.2", X_OK) = -1 ENOENT (No such file or directory)

Maybe cluster's /lib64/ld-linux-x86-64.so.2 dislikes your pin_sim.so somehow.

To further troubleshoot the issue, try running /lib64/ld-linux-x86-64.so.2 --list /path/to/your/pin_sim.so on your cluster.

Next time, don't forget to add -f to your strace arguments, it turns on full process tree tracing.

  • 1
    OK this can fall into several options. For now please attach output of readelf -Wl pin_sim.so for your failed pin_sim.so to your question (it should not be too lengthy). You probably hit several protection mechanisms your cluster admin has established, like PaX or complete grsecurity suite (and maybe others like SELinux enforce that too). – user140866 Nov 15 '15 at 0:16
  • 1
    And as you can see, ldd is just a script which goes through list of dynamic linkers and tries to call them, and even if one loads your appropriate library but dislikes it for some reason, the script just goes with error code other than 126/127 unnoticed. Pretty a glibc bug, again. – user140866 Nov 15 '15 at 0:22
  • 1
    Ah, as I expected. In column Type you see GNU_STACK, in column Flg in front of it you see RWE. This is exact your problem why library does not load. Your cluster admins had established some protection mechanisms that prevent library from loading with writable and executable stack at the same time. Now you have two options: 1. If your pin_sim.so requires executable writable stack (it's x86 simulator, so it can have such requirements), they you're probably out of luck, if your cluster admins will not change their mind about that. – user140866 Nov 16 '15 at 2:20
  • 1
    The reason it works for you just because your system does not enforce additional security restrictions (and most Linux don't) like your cluster admins do. Unfortunately if you're not an admin of cluster, you will not be able to fix it, if your program requires special environment. There is also no clear way to detect such restrictions other than to trigger them. – user140866 Nov 16 '15 at 2:27
  • 1
    I just downloaded sniper and checked it does not have execstack inside it's Makefiles, so here is second option which you can try: 2. Try to build your pin_sim.so with this LD_FLAGS: -Wl,-z -Wl,noexecstack (edit your pin/Makefile, find LD_FLAGS line and append this). I hope there will be either compilation errors, or crash at runtime, so you're really have little options there. – user140866 Nov 16 '15 at 2:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.