24

I thought I had a good handle on bash file redirection, and generally I try to avoid "useless use of cat", but I experienced some unexpected behaviour with a script and I would like to understand why it occurs.

Within a bash script, I execute:

somecommand < file1 > file2

My expectation was that file1 is safe and opened in a read-only manner. In practice, I found that file1 can be overwritten. How/why does this happen, and is there a way to prevent it without resorting to a cat?

If it's working how I imagine (the process ends up with a direct rw file descriptor?), it seems like it should be considered dangerous to redirect files this way, yet I've never seen this behaviour mentioned before.

To add some specifics from my case: the command in question is sops, which in the background is doing some GPG stuff. The GPG password prompt is sometimes being written to the file used for input, overwriting it. The complete command I used is:

sops --input-type json --output-type json -d /dev/stdin < ./secrets/file.json > ./secrets/file-decrypted.json

I have since switched to cat file1 | sops.. > file2 and everything works as expected. I would have said this was a "useless use of cat" - but it doesn't seem so useless anymore!


It seems to be when gpg-agent is not running and prompts for the first time.
14
  • I don't see how cat would change anything here. Are you sure it actually solves the issue? Your file1 can still be overwritten just like before. You said it only happens "sometimes", so maybe it just hasn't happened yet with your cat version?
    – terdon
    Oct 5, 2020 at 15:23
  • Pretty sure. I can reliably reproduce it by killing my gpg agent. I don't know what funky things pinentry is doing behind the scenes, but it doesn't do it when I use the cat.
    – Lewis
    Oct 5, 2020 at 15:29
  • 1
    What Artem said. There's nothing in your command that would touch the input file, so something else must be doing it. Whether you open it with cat or < will make no difference.
    – terdon
    Oct 5, 2020 at 15:31
  • 2
  • 1
    @Lewis even if that happens when you are not using /dev/stdin, it still happens because that script/program is opening /dev/stdin explicitly at some point (or one of its "aliases": /dev/fd/0, /proc/self/fd/0, /proc/<pid>/fd/0, etc). A process is not able to write to file descriptor open in read-only mode, as fd 0 is in cmd < file.
    – user313992
    Oct 6, 2020 at 8:30

1 Answer 1

31

That's due to the way /dev/stdin (actually /proc/self/fd/0) is implemented on Linux (and Cygwin, but generally not other systems).

On Linux opening /dev/stdin is not like doing a dup(0), it just reopens the same file as open on fd 0 anew. It doesn't share the open file description that fd 0 refers to (with the readonly mode), but gets a completely unrelated new open file description, with the mode as specified in open().

So if sops -d /dev/stdin opens /dev/stdin in read+write mode and fd 0 was open in read-only on /some/file, /some/file will be open in read+write.

Effectively, cmd /dev/stdin < file there is the same as cmd file < file. You'll find that /dev/stdin is just a symlink¹ to file:

/tmp$ namei -l /dev/stdin < file
f: /dev/stdin
drwxr-xr-x root     root     /
drwxr-xr-x root     root     dev
lrwxrwxrwx root     root     stdin -> /proc/self/fd/0
drwxr-xr-x root     root       /
dr-xr-xr-x root     root       proc
lrwxrwxrwx root     root       self -> 73569
dr-xr-xr-x stephane stephane     73569
dr-x------ stephane stephane   fd
lr-x------ stephane stephane   0 -> /tmp/file
drwxr-xr-x root     root         /
drwxrwxrwt root     root         tmp
-rw-r--r-- stephane stephane     file

It can get worse. If it was opening with O_TRUNC, the file would be truncated. If fd 0 was pointing to the reading end of a pipe and /dev/stdin was open in write-only mode, you'd get the other end of the pipe.

But using:

cat file | cmd /dev/stdin

Would guard against cmd overwriting file as all cmd would see would be the pipe. And even if it did open in write-only mode, it couldn't get back to the file, it would just get to the writing end of the pipe and the only file descriptor on the reading end would be cmd's stdin.

Other OSes don't have the problem as opening /dev/stdin there is like doing a dup(0), so you get the same open file description and if you open with an incompatible mode, the open() system call just fails.


¹ technically, as noted by @user414777 in comments, /proc/<pid>/fd/<fd> are magic symlinks in that for instance they can reach into places that normal symlinks could not, but when it comes to opening them, past the path resolution stage, they act like normal symlinks in that you just open the target file

15
  • 2
    So in other words.. you need to trust a program not to use /dev/stdin? And so it is indeed potentially dangerous to be in the habit of redirecting a file for input this way?
    – Lewis
    Oct 5, 2020 at 15:56
  • 7
    @terdon, < file doesn't pass a stream or file, it just opens file with O_RDONLY on fd 0. The application can only read from that fd. The problem is just with those special /dev/stdin / /dev/fd, /proc/self/fd symlinks which on Linux are just symlinks to the corresponding files without any relation to how the corresponding fds were opened. Oct 5, 2020 at 15:58
  • 4
    @Lewis You already "trust" programs they don't read and publish your ~/.ssh/id_rsa. Oct 5, 2020 at 16:00
  • 3
    @KamilMaciorowski True, but somehow this feels different. I didn't realise my data was at risk when passed like this :)
    – Lewis
    Oct 5, 2020 at 16:02
  • 3
    Just to follow up, in my case it appears the embedded mozilla-services/gogpagent is writing to /proc/self/fd/0.
    – Lewis
    Oct 6, 2020 at 8:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .