non-reentrant libraries in shared memory?

Question

I found this Q&A saying shared libraries can be shared between processes using shared memory. It seems like it would be impossible, though, to share code between processes without some pretty severe restrictions on the type of code that can be shared. I'm thinking about libraries with non-reentrant C functions whose output depends on the values of global variables or static variables inside their definition body. Like this one.

int really_really_nonreentrant(void x)
{
    static int i = 0;
    i++;
    return i;
}

A library with a function like this will return separate increasing sequences for each process using it, so it definitely seems like the code isn't being shared between processes. Is really_really_nonreentrant() being separated from the reentrant functions, or is it being kept mostly with the other functions with just static int i being separated out? Or is the entire library being kept out of shared memory because this function is nonreentrant?

Ultimately, how much does one have to do to ensure one's library gets allocated to shared memory?

Answer 1

I believe that the really short answer is that Linux compilers arrange code into pieces, at least one of which is just pure code, and can therefore be memory mapped into more than one process' address space. Any globals get mapped such that each process gets its own copy.

You can see this using readelf, or objdump, but readelf gives a clearer picture, I think.

Here's a piece of output from readelf -e /usr/lib/libc.so.6. That's the C library, probably mapped into almost every process. The relevant part of readelf output (although all of it is interesting) is the Program Headers:

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  PHDR           0x000034 0x00000034 0x00000034 0x00140 0x00140 R E 0x4
  INTERP         0x164668 0x00164668 0x00164668 0x00017 0x00017 R   0x4
      [Requesting program interpreter: /usr/lib/ld-linux.so.2]
  LOAD           0x000000 0x00000000 0x00000000 0x1adfc4 0x1adfc4 R E 0x1000
  LOAD           0x1ae220 0x001af220 0x001af220 0x02c94 0x057c4 RW  0x1000
  DYNAMIC        0x1afd90 0x001b0d90 0x001b0d90 0x000f8 0x000f8 RW  0x4
  NOTE           0x000174 0x00000174 0x00000174 0x00044 0x00044 R   0x4
  TLS            0x1ae220 0x001af220 0x001af220 0x00008 0x00048 R   0x4
  GNU_EH_FRAME   0x164680 0x00164680 0x00164680 0x06124 0x06124 R   0x4
  GNU_STACK      0x000000 0x00000000 0x00000000 0x00000 0x00000 RW  0x10
  GNU_RELRO      0x1ae220 0x001af220 0x001af220 0x01de0 0x01de0 R   0x1

The two LOAD lines are the only pieces of the file that get mapped directly into memory. The first LOAD header maps a piece of /usr/lib/libc.so.6 into memory with R and E permissions: read and execute. That's the code. Hardware features keep a program from writing to that piece of memory, so all programs can share the same pages of real, physical memory. The kernel can set up the hardware to map the same physical memory into all processes.

The second LOAD header is marked RW - read and write. This is the part with global variables that the C library uses. Each process gets its own copy in physical memory, mapped into that process' address space with the hardware permissions set to allow reading and writing. That section is not shared.

You can see these memory mappings in a running process using the /proc file system. A good command to illustrate: cat /proc/self/maps. That lists all the memory mappings that the cat process has, and from what files the kernel got them.

As far as how much you have to do to ensure that your function is allocated to memory that gets mapped into different processes, it's pretty much all down to the flags you give to the compiler. Code intended for ".so" shared libraries is compiled "position independent". Position independent code does things like refer to memory locations of variables with offsets relative to the current instruction, and jumps or branches to locations relative to the current instruction, rather than loading from or writing to absolute addresses, and jumping to absolute addresses. That means the "RE" LOAD piece of /usr/lib/libc.so and the "RW" piece only have to have be loaded at addresses that are the same distance apart in each process. In your example code, the static variable will always be at least a multiple of a page size apart from the code that references it, and it will always get loaded that distance apart in a process' address space due to the way the LOAD ELF headers are given.

Just a note about the term "shared memory": There's a user-level shared memory system, associate with "System V interprocess communications system". That's a way for more than one process to very explicitly share a piece of memory. It's fairly complicated and obscure to set up and get correct. The shared memory that we're talking about here is more-or-less completely invisible to any user process. Your example code won't know the difference if it's running as position independent code shared between multiple processes, or if it's the only copy ever.