Surprised by behavior of cp with hardlinks

Question

I understand the notion of hardlinks very well, and have read the man pages for basic tools like cp --- and even the recent POSIX specs --- a number of times. Still I was surprised to observe the following behavior:

$ echo john > john
$ cp -l john paul
$ echo george > george

At this point john and paul will have the same inode (and content), and george will differ in both respects. Now we do:

$ cp george paul

At this point I expected george and paul to have different inode numbers but the same content --- this expectation was fulfilled --- but I also expected paul to now have a different inode number from john, and for john to still have the content john. This is where I was surprised. It turns out that copying a file to the destination path paul also has the result of installing that same file (same inode) at all other destination paths that share paul's inode. I was thinking that cp creates a new file and moves it into the place formerly occupied by the old file paul. Instead what it seems to do is to open the existing file paul, truncating it, and write george's content into that existing file. Hence any "other" files with the same inode get "their" content updated at the same time.

Ok, this is a systematic behavior and now that I know to expect it I can figure out how to work around it, or take advantage of it, as appropriate. What puzzles me is where I was supposed to see this behavior documented? I'd be surprised if it's not documented somewhere in documents I've already looked at. But apparently I missed it, and can't now find a source that discusses this behavior.

Answer 1

First, why is it done this way? One reason is historical: that's how it was done in Unix First Edition.

Files are taken in pairs; the first is opened for reading,the second created mode 17. Then the first is copied into the second.

“Created” refers to the creat system call (the one that's famously missing an e), which truncates the existing file by the given name if there is one.

And here's the source code of cp in Unix Second Edition (I can't find the source code of First Edition). You can see the calls to open for the source file and creat for the second file; and, as an improvement to First Edition, if the second file is an existing directory, cp creates a file in that directory.

But, you may ask, why was it done that way at the time? The answer to “why did Unix originally do it that way” is almost always simplicity. cp opens its source for reading and creates its destination — and the system call to create a file overwrites an existing file by opening it for writing, because that allows the caller to impose the content of a file by the given name whether the file already existed or not.

Now, as to where it's documented: in the FreeBSD man page.

For each destination file that already exists, its contents are overwritten if permissions allow. Its mode, user ID, and group ID are unchanged unless the -p option was specified.

That wording was present at least as far back as 1990 (back when BSD was 4.3BSD). There is similar wording on Solaris 10:

If target_file exists, cp overwrites its contents, but the mode (and ACL if applicable), owner, and group associated with it are not changed.

Your case is even spelled out in the HP-UX 10 manual:

If new_file is a link to an existing file with other links, overwrites the existing file and retains all links.

POSIX puts it in standardese. Quoting from Single UNIX v2:

If dest_file exists, the following steps are taken: (…) A file descriptor for dest_file will be obtained by performing actions equivalent to the XSH specification open() function called using dest_file as the path argument, and the bitwise inclusive OR of O_WRONLY and O_TRUNC as the oflag argument.

The man pages and specification that I quoted further specifies that if the -f option is passed and the attempt to open/create the target file fails (typically due to not having permission to write the file), cp tries to remove the target and create a file again. This would break the hard link in your scenario.

You may want to report a documentation bug against the GNU coreutils manual, since it doesn't document this behavior. Even the description of --preserve=links, which in your scenario would lead to the paul link being removed and a new file being created, doesn't make it clear what happens without --preserve=links. The description of -f kind of implies what happens without it but doesn't spell it out (“When copying without this option and an existing destination file cannot be opened for writing, the copy fails. However, with --force, …”).

Answer 2

cp documents that it overwrites the destination file if the destination file is already present. You're right that it doesn't specify in detail what "overwrite" means, but it definitely says "overwrite", not "replace". If you want to be pedantic, you can argue that "overwrite" is exactly what cp does, and the behaviour you were expecting would be properly called "replace".

Also note that if cp were to "replace" pre-existing destination files, that might reasonable be considered surprising or incorrect, probably moreso than "overwriting". For example:

• If cp first deleted the old file and then created a new one then there would be an interval of time during which the file would be absent, which would be surprising.
• If cp first created a temporary file and then moved it in place then it should probably document this, owing to the fact that such temporary files with strange names would occasionally be noticed... but it doesn't.
• If cp could not create a new file in the same directory as the old file due to permissions then this would be unfortunate (especially if it had already deleted the old one).
• If the file was not owned by the user running cp and the user running cp was not root then it would be impossible to match the owner & permissions of the new file to those of the new file.
• If the file has fancy special attributes that cp does not know about, then these would be lost in the copy. Nowadays implementations of cp ought to reliably understand things like extended attributes, but it wasn't always so. And there are other things, like MacOS resource forks, or, for remote filesystems, basically anything.

So in conclusion: now you know what cp really does. You'll never be surprised by it again! Honestly, I think the same thing might have happened to me too, many years ago.

Answer 3

I see that the POSIX 2013 standard does specify the observed behavior. It says:

3. If source_file is of type regular file, the following steps shall be taken:

   a. ... if dest_file exists, the following steps shall be taken:

   i. If the -i option is in effect, the cp utility shall write a prompt to the standard error and read a line from the standard input. If the response is not affirmative, cp shall do nothing more with source_file and go on to any remaining files.

   ii. A file descriptor for dest_file shall be obtained by performing actions equivalent to the open() function defined in the System Interfaces volume of POSIX.1-2008 called using dest_file as the path argument, and the bitwise-inclusive OR of O_WRONLY and O_TRUNC as the oflag argument.

   iii. If the attempt to obtain a file descriptor fails and the -f option is in effect, cp shall attempt to remove the file by performing actions equivalent to the unlink() function defined in the System Interfaces volume of POSIX.1-2008 called using dest_file as the path argument. If this attempt succeeds, cp shall continue with step 3b.

   ...

   d. The contents of source_file shall be written to the file descriptor. Any write errors shall cause cp to write a diagnostic message to standard error and continue to step 3e.

   e. The file descriptor shall be closed.

Answer 4

If you can say, “copying a file to the destination path paul also copies the same file (same inode) to all other destination paths that share paul’s inode.”, I’m sorry to say that you don’t understand the notion of hard links very well.  If I give an apple to Sir McCartney, I have given an apple to Paul, and I have given an apple to John Lennon’s songwriting partner.  But I haven’t given out three apples; I’ve given an apple to a person who has multiple names/titles/descriptors.

Similarly, when you copy george to paul, you aren’t also copying it to john.  Rather, you are copying the george data to the file whose inode is pointed to by the paul directory entry.

Step by Step:  When you do

echo john > john

you have created a new file (assuming that there wasn’t already a file named john in that directory).  Or, to speak more strictly, this is assuming that there wasn’t already a directory entry with the name john in that directory (because, strictly speaking, there are no files in directories; only directory entries, which point to inodes).  After you do

cp -l john paul

or

ln john paul

you have not created a new file; rather, you have given your existing file a new name.  You now have a file with two names: john and paul.  And when you say

cp george paul

you are overwriting that file.  The fact that it has two names is irrelevant; it could have 42 names, possibly in places that you can’t even access, and this command would not be copying the george\n data to all of those names (paths); it is just copying the data to the one file that has multiple names.