I thought I had a good handle on bash file redirection, and generally I try to avoid "useless use of cat", but I experienced some unexpected behaviour with a script and I would like to understand why it occurs.

Within a bash script, I execute:

somecommand < file1 > file2

My expectation was that file1 is safe and opened in a read-only manner. In practice, I found that file1 can be overwritten. How/why does this happen, and is there a way to prevent it without resorting to a cat?

If it's working how I imagine (the process ends up with a direct rw file descriptor?), it seems like it should be considered dangerous to redirect files this way, yet I've never seen this behaviour mentioned before.

To add some specifics from my case: the command in question is sops, which in the background is doing some GPG stuff. The GPG password prompt is sometimes being written to the file used for input, overwriting it. The complete command I used is:

sops --input-type json --output-type json -d /dev/stdin < ./secrets/file.json > ./secrets/file-decrypted.json

I have since switched to cat file1 | sops.. > file2 and everything works as expected. I would have said this was a "useless use of cat" - but it doesn't seem so useless anymore!

It seems to be when gpg-agent is not running and prompts for the first time.
  • I don't see how cat would change anything here. Are you sure it actually solves the issue? Your file1 can still be overwritten just like before. You said it only happens "sometimes", so maybe it just hasn't happened yet with your cat version? – terdon Oct 5 '20 at 15:23
  • Pretty sure. I can reliably reproduce it by killing my gpg agent. I don't know what funky things pinentry is doing behind the scenes, but it doesn't do it when I use the cat. – Lewis Oct 5 '20 at 15:29
  • 1
    What Artem said. There's nothing in your command that would touch the input file, so something else must be doing it. Whether you open it with cat or < will make no difference. – terdon Oct 5 '20 at 15:31
  • 2
  • 1
    @Lewis even if that happens when you are not using /dev/stdin, it still happens because that script/program is opening /dev/stdin explicitly at some point (or one of its "aliases": /dev/fd/0, /proc/self/fd/0, /proc/<pid>/fd/0, etc). A process is not able to write to file descriptor open in read-only mode, as fd 0 is in cmd < file. – user414777 Oct 6 '20 at 8:30

That's due to the way /dev/stdin (actually /proc/self/fd/0) is implemented on Linux (and Cygwin, but generally not other systems).

On Linux opening /dev/stdin is not like doing a dup(0), it just reopens the same file as open on fd 0 anew. It doesn't share the open file description that fd 0 refers to (with the readonly mode), but gets a completely unrelated new open file description, with the mode as specified in open().

So if sops -d /dev/stdin opens /dev/stdin in read+write mode and fd 0 was open in read-only on /some/file, /some/file will be open in read+write.

Effectively, cmd /dev/stdin < file there is the same as cmd file < file. You'll find that /dev/stdin is just a symlink¹ to file:

/tmp$ namei -l /dev/stdin < file
f: /dev/stdin
drwxr-xr-x root     root     /
drwxr-xr-x root     root     dev
lrwxrwxrwx root     root     stdin -> /proc/self/fd/0
drwxr-xr-x root     root       /
dr-xr-xr-x root     root       proc
lrwxrwxrwx root     root       self -> 73569
dr-xr-xr-x stephane stephane     73569
dr-x------ stephane stephane   fd
lr-x------ stephane stephane   0 -> /tmp/file
drwxr-xr-x root     root         /
drwxrwxrwt root     root         tmp
-rw-r--r-- stephane stephane     file

It can get worse. If it was opening with O_TRUNCATE, the file would be truncated. If fd 0 was pointing to the reading end of a pipe and /dev/stdin was open in write-only mode, you'd get the other end of the pipe.

But using:

cat file | cmd /dev/stdin

Would guard against cmd overwriting file as all cmd would see would be the pipe. And even if it did open in write-only mode, it couldn't get back to the file, it would just get to the writing end of the pipe and the only file descriptor on the reading end would be cmd's stdin.

Other OSes don't have the problem as opening /dev/stdin there is like doing a dup(0), so you get the same open file description and if you open with an incompatible mode, the open() system call just fails.

¹ technically, as noted by @user414777 in comments, /proc/<pid>/fd/<fd> are magic symlinks in that for instance they can reach into places that normal symlinks could not, but when it comes to opening them, past the path resolution stage, they act like normal symlinks in that you just open the target file

  • 6
    @terdon, < file doesn't pass a stream or file, it just opens file with O_RDONLY on fd 0. The application can only read from that fd. The problem is just with those special /dev/stdin / /dev/fd, /proc/self/fd symlinks which on Linux are just symlinks to the corresponding files without any relation to how the corresponding fds were opened. – Stéphane Chazelas Oct 5 '20 at 15:58
  • 4
    @Lewis You already "trust" programs they don't read and publish your ~/.ssh/id_rsa. – Kamil Maciorowski Oct 5 '20 at 16:00
  • 2
    @KamilMaciorowski True, but somehow this feels different. I didn't realise my data was at risk when passed like this :) – Lewis Oct 5 '20 at 16:02
  • 2
    They're not "just symlinks". They're "magic" symlinks, which can be opened even when their target doesn't resolve or resolves to an inaccessible path. – user414777 Oct 6 '20 at 2:34
  • 3
    Just to follow up, in my case it appears the embedded mozilla-services/gogpagent is writing to /proc/self/fd/0. – Lewis Oct 6 '20 at 8:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.