Shared library load mem address region is architecture dependent?

Question

I see that on x86, in a process's VM, shared libraries are loaded between heap and stack, i.e. the mmap area as noted in most of the online articles. But on a PowerPC linux box, I see that all libraries are loaded below where the program itself is loaded. "strace" shows the library load address is pre-determined (assuming by ld) before load/mmap the library.

I wonder whether this is an architecture dependent thing? Are there any online documents about this?

memory map(ppc):

> cat /proc/self/maps
00100000-00102000 r-xp 00000000 00:00 0          [vdso]
0fe40000-0ffab000 r-xp 00000000 08:02 147120     /lib/libc-2.11.1.so
0ffab000-0ffbb000 ---p 0016b000 08:02 147120     /lib/libc-2.11.1.so
0ffbb000-0ffbf000 r--p 0016b000 08:02 147120     /lib/libc-2.11.1.so
0ffbf000-0ffc0000 rw-p 0016f000 08:02 147120     /lib/libc-2.11.1.so
0ffc0000-0ffc3000 rw-p 00000000 00:00 0 
0ffd0000-0fff0000 r-xp 00000000 08:02 147113     /lib/ld-2.11.1.so
0fff0000-0fff1000 r--p 00020000 08:02 147113     /lib/ld-2.11.1.so
0fff1000-0fff2000 rw-p 00021000 08:02 147113     /lib/ld-2.11.1.so
10000000-10005000 r-xp 00000000 08:02 195850     /bin/cat
10014000-10015000 rw-p 00004000 08:02 195850     /bin/cat
10015000-10036000 rwxp 00000000 00:00 0          [heap]
24000000-24001000 rw-p 00000000 00:00 0 
24013000-24014000 rw-p 00000000 00:00 0 
bfade000-bfaff000 rw-p 00000000 00:00 0          [stack]

strace (on ppc):

> strace cat
execve("/bin/cat", ["cat"], [/* 26 vars */]) = 0
brk(0)                                  = 0x10015000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x24000000
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(0x3, 0xbfb84558)                = 0
mmap(NULL, 70203, PROT_READ, MAP_PRIVATE, 3, 0) = 0x24001000
close(3)                                = 0
open("/lib/libc.so.6", O_RDONLY)        = 3
read(3, "\177ELF\1\2\1\0\0\0\0\0\0\0\0\0\0\3\0\24\0\0\0\1\17\345\376\260\0\0\0004"..., 512) = 512
fstat64(0x3, 0xbfb84540)                = 0
mmap(0xfe40000, 1582324, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xfe40000
mprotect(0xffab000, 65536, PROT_NONE)   = 0
mmap(0xffbb000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16b000) = 0xffbb000
mmap(0xffc0000, 9460, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xffc0000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x24013000
mprotect(0xffbb000, 16384, PROT_READ)   = 0
mprotect(0xfff0000, 4096, PROT_READ)    = 0
munmap(0x24001000, 70203)               = 0
brk(0)                                  = 0x10015000
brk(0x10036000)                         = 0x10036000
fstat64(0x1, 0xbfb84d00)                = 0
fstat64(0, 0xbfb84d00)                  = 0

Answer 1

I will answer my own question after some digging. In short, this is a arch-dependent thing. PowerPC32 defines a preferred load address function, while most other arches, including x86, does not define this.

/* The idea here is that to conform to the ABI, we are supposed to try
   to load dynamic objects between 0x10000 (we actually use 0x40000 as
   the lower bound, to increase the chance of a memory reference from
   a null pointer giving a segfault) and the program's load address;
   this may allow us to use a branch instruction in the PLT rather
   than a computed jump.  The address is only used as a preference for
   mmap, so if we get it wrong the worst that happens is that it gets
   mapped somewhere else.  */

ElfW(Addr)
__elf_preferred_address (struct link_map *loader, size_t maplength,
             ElfW(Addr) mapstartpref)
{
  ElfW(Addr) low, high;
  struct link_map *l;
  Lmid_t nsid;

  /* If the object has a preference, load it there!  */
  if (mapstartpref != 0)
    return mapstartpref;

  /* Otherwise, quickly look for a suitable gap between 0x3FFFF and
     0x70000000.  0x3FFFF is so that references off NULL pointers will
     cause a segfault, 0x70000000 is just paranoia (it should always
     be superseded by the program's load address).  */
  low =  0x0003FFFF;
  high = 0x70000000;
  for (nsid = 0; nsid < DL_NNS; ++nsid)
    for (l = GL(dl_ns)[nsid]._ns_loaded; l; l = l->l_next)
      {
    ElfW(Addr) mapstart, mapend;
    mapstart = l->l_map_start & ~(GLRO(dl_pagesize) - 1);
    mapend = l->l_map_end | (GLRO(dl_pagesize) - 1);
    assert (mapend > mapstart);

    /* Prefer gaps below the main executable, note that l ==
       _dl_loaded does not work for static binaries loading
       e.g. libnss_*.so.  */
    if ((mapend >= high || l->l_type == lt_executable)
        && high >= mapstart)
      high = mapstart;
    else if (mapend >= low && low >= mapstart)
      low = mapend;
    else if (high >= mapend && mapstart >= low)
      {
        if (high - mapend >= mapstart - low)
          low = mapend;
        else
          high = mapstart;
      }
      }

  high -= 0x10000; /* Allow some room between objects.  */
  maplength = (maplength | (GLRO(dl_pagesize) - 1)) + 1;
  if (high <= low || high - low < maplength )
    return 0;
  return high - maplength;  /* Both high and maplength are page-aligned.  */
}

Answer 2

(Preferred) disposition of the run-time stack differs among hardware architectures. But as for the program proper and shared libraries, for a dynamically linked executable, memory location of mapped regions is determined by the linker. It’s not, generally, a kernel’s job, to decide where components of a userland program should be located. Nor is the order implied by CPU architecture. On the same hardware, and even within one running OS (i. e. kernel) we can imagine different linkers arranging it differently (exec Linux call extracts the name of the linker from the ELF file; see elf_interpreter variable in load_elf_binary() in fs/binfmt_elf.c).

In Linux, default dynamic linker ld-linux is a part of glibc. How it tries to map objects can be seen in the _dl_map_object_from_fd() function at elf/dl-load.c of its source code. Sometimes preferences in the executable are accounted for (that presumedly depend on compiler and linker created the executable), and in some cases disposition of a memory map is decided by the kernel.

There is some googleable information on dynamic linkers and their architecture dependence, such as:

• https://grugq.github.io/docs/subversiveld.pdf