58

Yesterday I read this SO comment which says that in the shell (at least bash) >&- "has the same result as" >/dev/null.

That comment actually refers to the ABS guide as the source of its information. But that source says that the >&- syntax "closes file descriptors".

It is not clear to me whether the two actions of closing a file descriptor and redirecting it to the null device are totally equivalent. So my question is: are they?

On the surface of it it seems that closing a descriptor is like closing a door but redirecting it to a null device is opening a door to limbo! The two don't seem exactly the same to me because if I see a closed door, I won't try to throw anything out of it, but if I see an open door I will assume I can.

In other words, I have always wondered if >/dev/null means that cat mybigfile >/dev/null would actually process every byte of the file and write it to /dev/null which forgets it. On the other hand, if the shell encounters a closed file descriptor I tend to think (but am not sure) that it will simply not write anything, though the question remains whether cat will still read every byte.

This comment says >&- and >/dev/null "should" be the same, but it is not so resounding answer to me. I'd like to have a more authoritative answer with some reference to standard or source core or not...

  • If I wanted to be charitable, I'd say that the comment didn't mean that they were implemented the same way, just that they both have the same end result of preventing you from seeing the program's output. – Barmar Oct 29 '14 at 18:57
73

No, you certainly don't want to close file descriptors 0, 1 and 2.

If you do so, the first time the application opens a file, it will become stdin/stdout/stderr...

For instance, if you do:

echo text | tee file >&-

When tee (at least some implementations, like busybox') opens the file for writing, it will be open on file descriptor 1 (stdout). So tee will write text twice into file:

$ echo text | strace tee file >&-
[...]
open("file", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 1
read(0, "text\n", 8193)                 = 5
write(1, "text\n", 5)                   = 5
write(1, "text\n", 5)                   = 5
read(0, "", 8193)                       = 0
exit_group(0)                           = ?

That has been known to cause security vulnerabilities. For instance:

chsh 2>&-

And chsh (a setuid application) may end up writing error messages in /etc/passwd.

Some tools and even some libraries try to guard against that. For instance GNU tee will move the file descriptor to one above 2 if the files it opens for writing are assigned 0, 1, 2 while busybox tee won't.

Most tools, if they can't write to stdout (because for instance it's not open), will report an error message on stderr (in the language of the user which means extra processing to open and parse localisation files...), so it will be significantly less efficient, and possibly cause the program to fail.

In any case, it won't be more efficient. The program will still do a write() system call. It can only be more efficient if the program gives up writing to stdout/stderr after the first failing write() system call, but programs generally don't do that. They generally either exit with an error or keep on trying.

  • 3
    I think this answer would be even better if the final paragraph was at the top (since it is what most directly answers the OP's question), and it then went on to discuss why it's a bad idea even if it did mostly work. But I'll take it, have an upvote. ;) – a CVn Oct 24 '14 at 9:12
  • @StéphaneChazelas: As Michael said, I'd have expected the last para on top, but thanks for clarifying it actually probably only creates problems. So I guess an ancillary question would be, when will closing an FD actually be useful? Or should I ask that as a separate question? – jamadagni Oct 24 '14 at 14:08
  • @jamadagni, see unix.stackexchange.com/search?q=user%3A22565+%223%3E%26-%22 for some examples – Stéphane Chazelas Oct 24 '14 at 14:13
  • 1
    @jamadagni If the link provided by Stéphane doesn't answer the question, I would say that sounds like the beginnings of a separate question as it isn't directly related to the relative efficiency of the two methods. – a CVn Oct 24 '14 at 14:21
  • 1
    I appreciate that Stephane starts with this vert important security warning, as it would be less visible if the last paragraph was on top. +1 from me. – Olivier Dulac Oct 25 '14 at 18:06
14

IOW I have always wondered if >/dev/null means that cat mybigfile >/dev/null would actually process every byte of the file and write it to /dev/null which forgets it.

It's not a full answer to your question, but yes, the above is how it works.

cat reads the named file(s), or standard input if no files are named, and outputs to its standard output the contents of those until it encounters an EOF on (including standard input) the last file named. That is its job.

By adding >/dev/null you are redirecting standard output to /dev/null. That is a special file (a device node) which throws away anything written in it (and immediately returns EOF on read). Note that I/O redirection is a feature provided by the shell, not by each individual application, and that there is nothing magical about the name /dev/null, only what happens to exist there on most Unix-like systems.

It's also important to note that the specific mechanics of device nodes vary from operating system to operating system, but cat (which, in a GNU system, means coreutils) is cross-platform (the same source code needs to run at least on Linux and Hurd) and hence cannot take dependencies to specific operating system kernels. Additionally, it still works if you create a /dev/null alias (on Linux, this means a device node with the same major/minor device number) with another name. And there's always the case of writing somewhere else that behaves effectively the same (say, /dev/zero).

It follows that cat is unaware of the special properties of /dev/null, and indeed likely is unaware of the redirection in the first place, but it still needs to perform exactly the same work: it reads the named files, and outputs the content of the/those file(s) to its standard output. That the standard output of cat happens to go into a void is not something cat itself is concerned with.

  • 2
    To extend your answer: Yes, cat mybigfile > /dev/null will cause cat to read every byte of bigfile into memory. And, for every n bytes, it will call write(1, buffer, n). Unbeknownst to the cat program, the write will do absolutely nothing (except maybe for some trivial bookkeeping). Writing to /dev/null does not require processing every byte. – G-Man Oct 24 '14 at 14:59
  • 2
    I remember I was blown away when I read the Linux kernel source of the /dev/null device. I was expecting there to be some elaborate system of free()ing buffers, etc., but, no, it's just basically a return(). – Brian Minton Oct 24 '14 at 18:22
  • 4
    @G-Man: I don't know that you can guarantee that to be true in all cases. I can't find evidence now, but I recall some implementation of either cat or cp that would work by mmaping big chunks of the source file into memory, then calling write() on the mapped region. If you were writing to /dev/null, the write() call would return at once without faulting in the pages of the source file, so it would never actually be read from the disk. – Nate Eldredge Oct 24 '14 at 19:17
  • 2
    Also, something like GNU cat does run on many platforms, but a casual glance at the source code will show lots of #ifdefs: it's not literally the same code that runs on all the platforms, and there are plenty of system-dependent sections. – Nate Eldredge Oct 24 '14 at 19:19
  • @NateEldredge: Interesting point, but I was just building on Michael's answer, so you aren't contradicting me so much as you are contradicting Michael. – G-Man Oct 24 '14 at 19:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.