If process A copies files to some location loc and process B regularly copies the files from loc to some other location, can B read a file that is currently in the process of being copied to loc by A?

I'm using Ubuntu Linux 12.04 if that's important.

Background information: I want to continuously backup a PostgreSQL cluster. PostgreSQL provides WAL archiving for that. It works by having the database call a script that copies a completed WAL file to some backup location.

I want another process to regularly copy the backed up WAL files to another server. If a WAL file is currently being copied by the database, can the second process still read the file without running into some EOF condition before the file is copied as a whole?

In other words: Can I do the following with no synchronization between A and B?

A                                   B
cp pg_xlog/some_wal_file /backup/   scp /backup/* user@remote-machine:/backups/
  • Not to subvert the question, but why not transfer the file to a temporary location then issue a mv command from the remote server (via ssh) that moves it to the /backups directory? My concern in saying that is what might happen if whatever process is looking in there sees the WAL and tries to use it, but it's incomplete due to transferring. I don't have enough experience with postgresql to know whether that would error out or potentially cause a data coherency problem. – Bratchley Apr 7 '14 at 13:24
  • @JoelDavis: The postgres part is just background; it could be anything. The copying is completely under my control. In fact currently I do create a temp file first and rename it once copy is done. But if that is not necessary, I wouldn't mind throwing this bit of complexity out of my scripts. :) – musiKk Apr 7 '14 at 13:32

I think the best thing to is to ensure that process B only copies files which have been fully transferred by process A. One way to do this would be to use a combination of cp and mv in process A, since the mv process uses the rename system call (provided the files are on the same filesystem) which is atomic. This means that from the perspective of process B, the files appear in their fully formed state.

One way to implement this would be to have a partial directory inside your /backup directory which is ignored by process B. For process A you could do something like:

cp pg_xlog/"$file" /backup/partial
mv /backup/partial/"$file" /backup

And for process B (using bash):

shopt -s extglob
scp /backup/!(partial) user@remote-machine:/backups/

Although the program that you probably want to look into, both for process A and process B, is rsync. rsync creates partial files and atomically moves into place by default (although usually the partial files are hidden files rather than being in a specific directory). Rsync will also avoid transferring files that it doesn't need to and has a special delta algorithm for transferring only the relevant parts of files that need to be updated over the network (rsync must be installed in both locations, although transfers still go over ssh by default). Using rsync for process A:

rsync -a --partial-dir=/backup/partial pg_xlog/some_wal_file /backup/

For process B:

rsync -a --exclude=/partial/ /backup/ user@remote-machine:/backups/

In this scenario, the only guarantee you have is that B will either not copy the file, or copy a prefix of the file. B has no way to know that the file is being written to, so it will read to the (current) end of the file, then stop.

The common way to avoid this pitfall is to copy the file under a temporary name, then rename it afterwards:

dest=$(TMPDIR=/backup mktemp)
trap 'rm -f "$dest"' INT HUP ERR
cp -p pg_xlog/some_wal_file "$dest"
mv "$dest" "/backup/some_wal_file"

In the consumer, arrange to not copy temporary files. In your scenario, you can do that by making it a dot file — use dest=$(TMPDIR=/backup mktemp .XXXXXXXXXX) above. A simpler way is to call rsync instead of cp, as rsync uses this strategy by default:

rsync -a pg_xlog/some_wal_file /backup/

In step B, make sure to exclude these temporary files, for example:

rsync -a --exclude='/.*' /backup/ user@remote-machine:/backups/

If you don't want to rely on dot files, you can use a staging directory. Moving a file from a directory to another is atomic as long as the two directories are on the same filesystem.

mkdir -p /backup/incoming
cp -p pg_xlog/some_wal_file /backup/incoming/
mv /backup/incoming/some_wal_file /backup/
rsync -a --exclude=/staging  /backup/ user@remote-machine:/backups/
  • Surely staging should be incoming (or incoming should be staging). – Graeme Apr 7 '14 at 22:22
  • "B has no way to know" Yes, that's what I am/was unsure about. Sometimes there is magic. You can for example delete a file that's open; the actual file is removed once the last file descriptor is closed (at least that's what I read). Likewise the read() syscall could have been implemented to block instead of returning EOF if there is another FD in writing mode positioned at the end of the file or somesuch. Since there is no indication of that, I'll continue to use staging. – musiKk Apr 8 '14 at 8:47
  • I accepted the other answer simply because it was there first. The essence is the same. Thank you, too! – musiKk Apr 8 '14 at 11:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.