1

Consider this bash script code:

#!/bin/bash

function bug_part() {
    cat $1 > sample.first
    cat $1 > sample.second #second time you open file $1, it contains no data
}

bug_part <(echo "TEST")
[ "$(cat sample.first)" != "$(cat sample.second)" ] && echo "THIS IS A BUG" 1>&2 && exit 1
rm sample.first sample.second

Do you agree with me that this problem is bash bug? Or Linux bug?

Is there anybody that knows exactly what's happening behind?

  • Side note: comparing files is diff's job. Using cat in such a way is the best way to mess stuff up with empty files, newlines and other metacharacters, should you make a mistake in quoting. if diff f1 f2; then ... – John WH Smith Aug 23 '15 at 12:22
  • @JohnWHSmith this was just a fast test, for two 4-character files. in real cases, I prefer to use a hash function (like md5sum) to check context-equality of general files – Mostafa Nazari Aug 23 '15 at 12:56
  • 2
    If you open /dev/urandom twice, you won't get the same data twice. (You also won't get EOF, but that's another problem.) If you read from the terminal twice, you won't get the same data twice unless the user is boringly repetitive. If you open a FIFO twice, you won't get the same data twice. If you open a socket twice, you won't get the same data twice. /dev/mem, /dev/kmem, non-rewind tape devices… There are lots of devices where you can't read the same data twice even if you open it twice and try reading from it again. – Jonathan Leffler Aug 23 '15 at 18:01
  • @JonathanLeffler totally agree with you, BUT, when stdout(or other ostream) of a process is redirected to another process by <( (it would pass the name of stream /dev/fd/... to the other process) , I think that re-opening of this filename is just have the meaning that provide the same data – Mostafa Nazari Aug 24 '15 at 8:17
  • 1
    @MostafaNazari: no, and I already gave examples where it doesn't apply (almost anything except a disk file). The device that is used with process redirection is not a disk file. You are not guaranteed to be able to read the same data twice even if you open it twice. – Jonathan Leffler Aug 24 '15 at 8:26
5

I don't think it's a bug. You can read from / write to a named pipe connected to commands prepared by process substitution only once.

  • Pipes work on a consumer/producer basis. Once data has been read by a process, it is removed from the pipe's queue. The consumer will be able to read again, if the producer writes to the pipe again :) – John WH Smith Aug 23 '15 at 12:22
  • If you test the bash implementation of operator <( you would use that it would execute the process inside parenthesis and send the name of stdout file descriptor /dev/fd/... as an argument value to outside process. now, in general purpose, if the inside application decide to open-and-read the file twice, this would cause problem in second reading. but if you call outside process with a real persisted file, this would not happened – Mostafa Nazari Aug 23 '15 at 13:17
  • @MostafaNazari: yes, exactly. If you want a real file, use a here-string, like cmd <<<"$text". You might need some extra redirections or cat /dev/stdin to open the file twice, instead of having the 2nd cat try to read from the same fd that's still at EOF from the first cat's use. – Peter Cordes Aug 23 '15 at 14:29
4

It's a bug in your script. Use tee if you want to duplicate data that can only be read once. As the other answer explains, <(cmd) makes a pipe, and puts /dev/fd/62 or similar on the command line:

echo <(true)
  /dev/fd/63

Another alternative to tee is a here-string:

cmd <<<"$text"

If you want bash to make a seekable tmp file and redirect input from that. (I'm not sure exactly how you'd rewind stdin to the beginning of the file inside your function, though. I think cat /dev/stdin might just get the same file position.)

  • <<< doesn't give you a regular file either, it uses a pipe. If you want a regular, seekable file, you have to create one. Or avoid trying to read the file twice. – Gordon Davisson Aug 23 '15 at 16:55
  • 1
    @GordonDavisson: The bash docs don't guarantee it, but the current implementation (bash4.3) writes the string to a file like /tmp/sh-thd-1045505329, opens it, redirects stdin from it, then unlinks it. strace -f -efile,write,read,dup2 bash -c 'cat <<< foo' to see it in action. – Peter Cordes Aug 24 '15 at 1:23
  • Wow, you're right; I though I'd tested that before commenting, but apparently not... – Gordon Davisson Aug 24 '15 at 5:25
  • @PeterCordes It is obvious that I write this simple code to make the problem clear. but consider you don't know what's going on in function bug_part, and if it opens the file twice, all thing go wrong without any error or alert! using <<< is not a solution for me (I test it and the same error still exists). my problem is why bash did not consider multiple opening of redirected streams by <( or even another operator – Mostafa Nazari Aug 24 '15 at 5:31
3

Expanding on @Peter Cordes' suggestion of using tee instead of trying to read the file (/pipe) twice, here's a possible rewrite of the function:

bugless_part() {
    tee sample.first >sample.second <"$1"
}

When run as bugless_part <(echo "TEST"), it puts "TEST" in both files.

  • I really like your Idea, You totally understand the problem, and the proper solution as I did. but The question is, would you agree with me about that this is a bash bug? because the usage of a sub-process in this way, depends on the implementation, and I think bash should handle this requirement, not me. At least it provides another operator to handle this – Mostafa Nazari Aug 24 '15 at 5:41
  • 3
    @MostafaNazari It's not a bug (and actually doesn't have anything to do with bash), it's just the way pipes work. When you read from a pipe (e.g. with the first cat), the data you read is removed the from the pipe -- there's just no way to go back and read it again, because the data isn't there anymore. That's why the second cat didn't get anything. – Gordon Davisson Aug 24 '15 at 16:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.