14

I know that when a page cache page is modified, it is marked dirty and requires a writeback, but what happens when:

Scenario: The file /apps/EXE, which is an executable file, is paged in to the page cache completely (all of its pages are in cache/memory) and being executed by process P

Continuous release then replaces /apps/EXE with a brand new executable.

Assumption 1: I assume that process P (and anyone else with a file descriptor referencing the old executable) will continue to use the old, in memory /apps/EXE without an issue, and any new process which tries to exec that path will get the new executable.

Assumption 2: I assume that if not all pages of the file are mapped into memory, that things will be fine until there is a page fault requiring pages from the file that have been replaced, and probably a segfault will occur?

Question 1: If you mlock all of the pages of the file with something like vmtouch does that change the scenario at all?

Question 2: If /apps/EXE is on a remote NFS, would that make any difference? (I assume not)

Please correct or validate my 2 assumptions and answer my 2 questions.

Let's assume this is a CentOS 7.6 box with some kind of 3.10.0-957.el7 kernel

Update: Thinking about it further, I wonder if this scenario is no different than any other dirty page scenario..

I suppose the process that writes the new binary will do a read and get all cache pages since it’s all paged in, and then all those pages will be marked dirty. If they are mlocked, they will just be useless pages occupying core memory after the ref count goes to zero.

I suspect when the currently-executing programs end, anything else will use the new binary. Assuming that’s all correct, I guess it’s only interesting when only some of the file is paged in.

  • Just to make it explicite, replacing a file won’t be a big thing (depending on if it is reopened by the application and how the application reacts to modified content), but modifying mmaped files can severity crash applications (it is a common problem in the java world when a zip file which has a mmaped directory entry is changed). It does however depend on the platform, it is not guaranteed that the mmaped regions see the change or not. – eckes Sep 8 '19 at 13:04
12

Continuous release then replaces /apps/EXE with a brand new executable.

This is the important part.

The way a new file is released is by creating a new file (e.g. /apps/EXE.tmp.20190907080000), writing the contents, setting permissions and ownership and finally rename(2)ing it to the final name /apps/EXE, replacing the old file.

The result is that the new file has a new inode number (which means, in effect, it's a different file.)

And the old file had its own inode number, which is actually still around even though the file name is not pointing to it anymore (or there are no file names pointing to that inode anymore.)

So, the key here is that when we talk about "files" in Linux, we're most often really talking about "inodes" since once a file has been opened, the inode is the reference we keep to the file.

Assumption 1: I assume that process P (and anyone else with a file descriptor referencing the old executable) will continue to use the old, in memory /apps/EXE without an issue, and any new process which tries to exec that path will get the new executable.

Correct.

Assumption 2: I assume that if not all pages of the file are mapped into memory, that things will be fine until there is a page fault requiring pages from the file that have been replaced, and probably a segfault will occur?

Incorrect. The old inode is still around, so page faults from the process using the old binary will still be able to find those pages on disk.

You can see some effects of this by looking at the /proc/${pid}/exe symlink (or, equivalently, lsof output) for the process running the old binary, which will show /app/EXE (deleted) to indicate the name is no longer there but the inode is still around.

You can also see that the diskspace used by the binary will only be released after the process dies (assuming it's the only process with that inode open.) Check output of df before and after killing the process, you'll see it drop by the size of that old binary you thought wasn't around anymore.

BTW, this is not only with binaries, but with any open files. If you open a file in a process and remove the file, the file will be kept on disk until that process closes the file (or dies.) Similarly to how hardlinks keep a counter of how many names point to an inode in disk, the filesystem driver (in the Linux kernel) keeps a counter of how many references exist to that inode in memory, and will only release the inode from disk once all references from the running system have been released as well.

Question 1: If you mlock all of the pages of the file with something like vmtouch does that change the scenario

This question is based on the incorrect assumption 2 that not locking the pages will cause segfaults. It won't.

Question 2: If /apps/EXE is on a remote NFS, would that make any difference? (I assume not)

It's meant to work the same way and most of the time it does, but there are some "gotchas" with NFS.

Sometimes you can see the artifacts of deleting a file that's still open in NFS (shows up as a hidden file in that directory.)

You also have some way to assign device numbers to NFS exports, to make sure those won't get "reshuffled" when the NFS server reboots.

But the main idea is the same. NFS client driver still uses inodes and will try to keep files around (on the server) while the inode is still referenced.

  • 1
    Does rename(2) block until the ref count of the oldname file goes to zero? – Gregg Leventhal Sep 7 '19 at 17:23
  • 2
    No, rename(2) won't block. The old inode is kept around for potentially a very long time. – filbranden Sep 7 '19 at 17:34
  • 1
    See @mosvy's answer to why you can't write to a file being executed (you get ETXTBSY). Unlinking and creating new has the same effect of rename: you end up with a new inode. (Rename is better because then there's no moment in which the file name does not exist, it's an atomic operation replacing the name to point to the new inode.) – filbranden Sep 7 '19 at 22:04
  • 4
    @GreggLeventhal: "What assumption are you making about the continuous release process I am using that makes you certain it uses temporary files?" – Because for as long as Unix exists, that is and has been the only sane way to do it. rename is pretty much the only file and filesystem operation that is guaranteed to be atomic (assuming we don't cross filesystem or device boundaries), so "create temp file and then rename" is the standard pattern for updating files. It's also what every text editor on Unix uses, for example. – Jörg W Mittag Sep 8 '19 at 1:47
  • 1
    @grahamj42: rename is part of POSIX. Granted, it is included by reference to ISO C (section 7.21.4.2 in the current draft), but it's in there. – Jörg W Mittag Sep 8 '19 at 15:43
7

Assumption 2: I assume that if not all pages of the file are mapped into memory, that things will be fine until there is a page fault requiring pages from the file that have been replaced, and probably a segfault will occur?

No, that will not happen, because the kernel will not let you open for write an replace anything inside a file which is currently executed. Such an action will fail with ETXTBSY [1]:

cp /bin/sleep sleep; ./sleep 3600 & echo none > ./sleep
[9] 5332
bash: ./sleep: Text file busy

When dpkg, etc updates a binary, it doesn't overwrite it, but uses rename(2) which simply points the directory entry to a completely different file, and any processes which still have mappings or open handles to the old file will continue to use it without problems.

[1] such protection is not extended to other files which can also be considered "text" (live code / executable): shared libraries, java classes, etc; modifying such a file while mapped by another process will cause it to crash. On linux, the dynamic linker dutifully passes the MAP_DENYWRITE flag to mmap(2), but make no mistake -- it has no effect whatsoever.

  • 1
    In the dpkg scenario, at what point then does the rename complete such that the dentry for /apps/EXE will reference the inode of the new binary? When there are no more references to the old one? How does that work? – Gregg Leventhal Sep 7 '19 at 17:20
  • 2
    rename(2) is atomic; as soon as it has completed, the dir entry refers to the new file. The processes which were still using the old file at that point would only be able to access it via existing mappings, or via open handles to it (which may reference an orphan dentry, no longer accessible other than via /proc/PID/fd). – mosvy Sep 7 '19 at 17:28
  • 1
    I like your answer the best because your ETXTBSY mention led me to this utcc.utoronto.ca/~cks/space/blog/unix/WhyTextFileBusyError which answers all of my questions. – Gregg Leventhal Sep 7 '19 at 21:28
4

filbranden's answer is correct assuming the continuous release process does proper atomic replacement of files via rename. If it doesn't, but modifies the file in-place, things are different. However your mental model is still mistaken.

There is no possibility of things getting modified on disk and being inconsistent with the page cache, because the page cache is the canonical version and the one that's modified. Any writes to a file take place through the page cache. If it's already present there, the existing pages are modified. If it's not yet present, attempts to modify a partial page will cause the whole page to be cached, followed by modification as if it were already cached. Writes that span a whole page or more can (and almost surely do) optimize out the read step paging them in. In any case, there's only one canonical modifiable version of a file ever(*) in existence, the one in the page cache.

(*) I slightly lied. For NFS and other remote filesystems, there may be more than one, and they typically (depending on which one and what mount and server-side options are used) don't correctly implement atomicity and ordering semantics for writes. That's why a lot of us consider them fundamentally broken and refuse to use them for situations where there will be writes concurrent with use.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.