Copying large files in Linux

Question

What is the effect of copying a file say fileA.big (900mb) from location B to locationC. If during that cp operation, say 35% through the process, fileA.big is appended with new information and grows from 900mb to 930mb.

What is the result of the end copy (i.e. fileA.big at locationC)?

What if the copy is about 70% through, and the original file is updated but this time truncated to 400mb (i.e. the progress of the copy is beyond the truncation point), what is the result of the end copy?

Referring to a linux OS on an ext3/ext4 filesystem. No volume shadow magic etc.. Just plain old cp. Curiousity sparked by copying live couchdb files for backup, but more interested in general scenarios rather than specific use case.

Answer 1

If fileA.big is grown during the copy, the copy will include the data that was appended.

If the file is truncated shorter than where the copy is currently at, the copy will abort right where its at and the destination file will contain what was copied up to the time it aborted.

Answer 2

Patrick has it more or less correct, but here's why. The way you copy a file under UNIX works like this:

1. Try to read some (more) bytes from fileA.
2. If we failed to get bytes because we're at (or past) the end of the file, we're done; quit.
3. Otherwise, write the bytes to fileB and loop back to step 1.

Knowing that, and knowing it's as simple as that, lets us see some corner cases.

As soon as we find the end of the file, the copy is done. So let's say our file is growing during the copy, but is growing more slowly than we're copying it. The copy program will keep going past the original file size, because by the time it gets there, there is more to the file. But at some point, it catches up with the end of the file, and it knows it's at the end because it can't read any more bytes right now. So it quits right there, even if the file is about to grow further.

If the file is truncated, the copy program says "Whoa, I'm past the end of the file!" and quits.

And if pieces of the file are being updated at random by, say, a database program :-), then your copy is going to be some mix of old and new data, because the data is not all copied at the same time. The result will probably be a corrupt copy, which is why it's not generally a good idea to make copies of live databases.

(That said, I'm not familiar with CouchDB, and it's possible to design a database to be resistant to this sort of corruption. But best to be absolutely sure.)