Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I wonder how killer applications such as Thunderbird or Firefox can be updated via the system's package manager while they are still running. What happens with the old code while they are being updated? What do I have to do when I want to write a program a.out that updates itself while it is running?

share|improve this question
    
    
@derobert Not exactly: that thread doesn't go into the peculiarities of executables. It provides relevant background but it isn't a duplicate. –  Gilles Jun 20 at 8:19
    
@Gilles Well, if you leave in the rest of the stuff it's too broad. There are two (at least) questions here. And the other question is pretty close, it's asking why replacing files to update works on Unix. –  derobert Jun 20 at 8:22
    
If you really want to do this with your own code, you might want to look at the programming language Erlang, where "hot" code updates are a key feature. learnyousomeerlang.com/relups –  mattdm Jun 20 at 10:47
    
@Gilles BTW: Having seen the essay you've added in response, I've retracted my close vote. You've turned this question into a good place to point anyone wanting to know how updates work. –  derobert Jun 20 at 16:55

3 Answers 3

Replacing files in general

First, there are several strategies to replace a file:

  1. Open the existing file for writing, truncate it to 0 length, and write the new content. (A less common variant is to open the existing file, overwrite the old content with the new content, truncate the file to the new length if it's shorter.) In shell terms:

    echo 'new content' >somefile
    
  2. Remove the old file, and create a new file by the same name. In shell terms:

    rm somefile
    echo 'new content' >somefile
    
  3. Write to a new file under a temporary name, then move the new file to the existing name. The move deletes the old file. In shell terms:

    echo 'new content' >somefile.new
    mv somefile.new somefile
    

I won't list all the differences between the strategies, I'll just mention some that are important here. With stategy 1, if any process is currently using the file, the process sees the new content as it's being updated. This can cause some confusion if the process expects the file content to remain the same. Note that this is only about processes that have the file open (as visible in lsof or in /proc/PID/fd/; interactive applications that have a document open (e.g. opening a file in an editor) usually do not keep the file open, they load the file content during the “open document” operation and they replace the file (using one of the strategies above) during the “save document” operation.

With strategies 2 and 3, if some process has the file somefile open, the old file remains open during the content upgrade. With strategy 2, the step of removing the file in fact only removes the file's entry in the directory. The file itself is only removed when it has no directory entry leading to it (on typical Unix filesystems, there can be more than one directory entry for the same file) and no process has it open. Here's a way to observe this — the file is only removed when the sleep process is killed (rm only removes its directory entry).

echo 'old content' >somefile
sleep 9999999 <somefile &
df .
rm somefile
df .
cat /proc/$!/fd/0
kill $!
df .

With strategy 3, the step of moving the new file to the existing name removes the directory entry leading to the old content and creates a directory entry leading to the new content. This is done in one atomic operation, so this strategy has a major advantage: if a process opens the file at any time, it will either see the old content or the new content — there's no risk of getting mixed content or of the file not existing.

Replacing executables

If you try strategy 1 with a running executable on Linux, you'll get an error.

cp /bin/sleep .
./sleep 999999 &
echo oops >|sleep
bash: sleep: Text file busy

A “text file” means a file containing executable code for obscure historical reasons. Linux, like many other unix variants, refuses to overwrite the code of a running program; a few unix variants allow this, leading to crashes unless the new code was a very well though-out modification of the old code.

On Linux, you can overwrite the code of a dynamically loaded library. It's likely to lead to a crash of the program that's using it. (You might not be able to observe this with sleep because it loads all the library code it needs when it starts. Try a more complex program that does something useful after sleeping, like perl -e 'sleep 9; print lc $ARGV[0]'.)

If an interpreter is running a script, the script file is opened in an ordinary way by the interpreter, so there is no protection against overwriting the script. Some interpreters read and parse the whole script before they start executing the first line, others read the script as needed. See What happens if you edit a script during execution? and How Does Linux deal with shell scripts? for more details.

Strategies 2 and 3 are safe for executables as well: although running executables (and dynamically loaded libraries) aren't open files in the sense of having a file descriptor, they behave in a very similar way. As long as some program is running the code, the file remains on disk even without a directory entry.

Upgrading an application

Most package managers use strategy 3 to replace files, because of the major advantage mentioned above — at any point in time, opening the file leads to a valid version of it.

Where application upgrades can break is that while upgrading one file is atomic, upgrading the application as a whole isn't if the application consists of multiple files (program, libraries, data, …). Consider the following sequence of events:

  1. An instance of the application is started.
  2. The application is upgraded.
  3. The running instance application opens one of its data files.

In step 3, the running instance of the old version of the application is opening a data file from the new version. Whether this works or not depends on the application, of which file it is and how much the file has been modified.

After an upgrade, you'll note that the old program is still running. If you want to run the new version, you'll have to exit the old program and run the new version. Package managers usually kill and restart daemons on an upgrade, but leave end-user applications alone.

A few daemons have special procedures to handle upgrades without having to kill the daemon and wait for the new instance to restart (which causes a service disruption). This is necessary in the case of init, which cannot be killed; init systems provide a way to request that the running instance call execve to replace itself with the new version.

share|improve this answer
    
"then move the new file to the existing name. The move deletes the old file." That's a little confusing, as its really just an unlink, as you cover later. Maybe "replaces the existing name", but that's still somewhat confusing. –  derobert Jun 20 at 16:13
    
@derobert I didn't want to get heavy onto “unlink” terminology. I'm using “delete” in reference to the directory entry, a subtlety which is explained later. Is it confusing at that stage? –  Gilles Jun 20 at 16:15
    
Probably not confusing enough to warrant an extra paragraph or two up top explaining unlink. I'm hoping for some wording that isn't confusing, but also is technically correct. Maybe just use "remove" again, which you already put a link to explain? –  derobert Jun 20 at 16:18

The upgrade can be run while the program runs, but the running program you see is actually the old version of it. The old binary remains on the disk until you close the program.

Explanation: on Linux systems, a file is just an inode, which can have several links to it. Eg. the /bin/bash you see is just a link to inode 3932163 on my system. You can find which inode does something link to by issuing ls --inode /path on the link. A file (inode) is only removed if there are zero links pointing to it and is not in use by any program. When the package manager upgrades eg. /usr/bin/firefox, it first unlinks (removes the hard link /usr/bin/firefox), then creates a new file called /usr/bin/firefox that is a hardlink to a different inode (the one which contains the new firefox version). The old inode is now marked as free and can be reused to store new data but remains on the disk (inodes are only created when you build your filesystem and are never deleted). On the next start of firefox, the new one will be used.

If you want to write a program that "upgrades" itself while running, the only possible solution I can think of is to periodically check the timestamp of its own binary file, and if it's newer than the program's start time, then reload itself.

share|improve this answer
1  
Actually, it's due to how deleting (unlinking) files works on Unix; see unix.stackexchange.com/questions/49299/… Also, at least on Linux, you can't actually write to a running binary, you'll get a "text file busy" error. –  derobert Jun 20 at 7:40
    
Strange... Then how does eg. Debian's apt upgrade work? I can upgrade any running program without problems including Iceweasel (Firefox). –  psimon Jun 20 at 7:43
2  
APT (or, rather, dpkg) doesn't overwrite files. It instead unlinks them and puts a new one under the same name. See the question & answer I linked to for an explanation. –  derobert Jun 20 at 7:46
2  
It's not just because it's still in RAM, it's still on disk. The file isn't actually deleted until the last instance of the program exits. –  derobert Jun 20 at 7:56
1  
The site won't let me suggest an edit (once your rep is high enough, you just make edits, you can no longer suggest them). So, as a comment: a file (inode) on a Unix system typically has one name. But it can have more if you add names with ln (hard links). You can remove names with rm (unlink). You can't actually directly delete a file, only remove its names. When a file has no names and additionally isn't open, the kernel will delete it. A running program has the file it's running from open, so even after removing all the names, the file is still around. –  derobert Jun 20 at 8:19

I wonder how killer applications such as Thunderbird or Firefox can be updated via the system's package manager while they are still running? Well, I can tell you that this doesn't really work well... I've had Firefox break on me quite horribly if I left it open while a package update was running. I had to kill it forcefully sometimes and restart it, because it was so broken I couldn't even close it properly.

What happens with the old code while they are being updated? Normally on Linux a program is loaded into memory, so the executable on disk isn't needed or used while the program is running. In fact you can even delete the executable and the program shouldn't care... However, some programs may need the executable, and certain OSs (like Windows) will lock the executable file, preventing deletion or even renames/moves, while the program is running. Firefox breaks, because its actually quite complex and uses a bunch of data files that tell it how to build its GUI (user interface). During a package update these files get overwritten (updated), so when an older Firefox executable (in memory) tries to use the new GUI files, weird things can happen...

What do I have to do when I want to write a program a.out that updates itself while it is running? There are lots of answers to your question already. Check this out: http://stackoverflow.com/questions/232347/how-should-i-implement-an-auto-updater By the way, questions about programming are better off on StackOverflow.

share|improve this answer
1  
Executables are actually demand-paged (swapped). They're not entirely loaded into memory, and may be dropped from memory whenever the system wants RAM for something else. The old version is actually remaining on disk; see unix.stackexchange.com/questions/49299/… . On Linux at least, you can't actually write to a running executable, you'll get the error "text file busy". Even root can't do it. (You're quite correct about Firefox, though). –  derobert Jun 20 at 7:43

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.