1

I've come across an issue where processes that load symlinked .so files crash if I update the .so while the processes are still running.

I have long-running processes that load shared libraries (.so) when first starting and may end up using those shared libraries much later on when the process is running. The shared libraries that the processes load are actually symlinks to the real .sos (to have all of the .sos in a structured directory pertaining to each process). Occasionally I need to apply some fix and recompile the .so file, but I've seen that when I do this, some of the long-running processes will crash when accessing symbols from that .so that I updated.

From my understanding, if the processes loaded the actual .so instead of a symlink, I should not see this behavior because when I recompile the .so (which removes and recreates the .so file with a new inode), the inode for that old .so should still stay in existence until all of the processes that have it opened close it. I think the problem I'm seeing is due to the fact that the processes are opening the symlink instead of the target .so, so the long-running processes only hold onto the symlinks inode instead of the target .sos inode. But I don't have enough information to confirm my theory is correct.

  1. Is it true that the inode for a file will remain in existence until all processors have closed it? And does this apply for processes that mmap the file but don't keep a file descriptor open for it?

  2. Does open/mmap of a symlink only keep track of the inode for the symlink? Or does it also prevent the O/S from destroying the target file's inode as well?

  • Have you tried any experiments (change code to access .so directly, to see what happens? – ctrl-alt-delor Apr 19 at 22:40
  • For ① Yes and yes. For ② symlinks should cause the target to be opened (try the experiments). – ctrl-alt-delor Apr 19 at 22:41
1
  1. Yes, on Linux (and all other POSIX compatible kernels) it is true that the inode for a file will remain in existence until all processors have closed it. This includes mmaped files.

    That is required by POSIX:

    The mmap() function adds an extra reference to the file associated with the file descriptor fildes which is not removed by a subsequent close() on that file descriptor. This reference is removed when there are no more mappings to the file.

  2. Open/mmap of a symlink only keeps track of the inode for the target file (except for the edge case of using O_PATH). All symlinks in the path are resolved when calling open() and the file descriptor only references the target file.

    You can test this:

    1. create a file /tmp/original
    2. create a symlink /tmp/symlink pointing to /tmp/original
    3. open /tmp/symlink in a program e.g. type in a Python shell f = open('/tmp/symlink', 'r')
    4. look at /proc/<pid of the program>/fd
    5. the file descriptor there will point to /tmp/original

Please check that when recompiling the .so file really is removed and recreated and not directly written to and that the long running process isn't using e.g. dlopen to dynamically reopen the .so file.

Another possibility I can think of is that lazy binding causes the loading of the .so file in the long running process to be deferred.

You might try starting the long running process with LD_BIND_NOW=1 in the environment and then watch if recompiling still crashes the process to check if the problem is caused by lazy binding.

  • I also realized that these files are on an NFS mount, and the long-running processes are on a different client than my process when I recompile them, and that the .so file is removed and recreated with a new inode# when recompiled. And I believe the long-running processes are using lazy binding, so when they attempt to load the .so file, the NFS server doesn't even have that inode anymore and should return an error for stale file handle. – Marshall Versteeg Apr 21 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.