25

I created a test file named 'test' that contains the following:

xxx
yyy
zzz

I ran the command:

(sed '/y/ q'; echo aaa; cat) < test

and I got:

xxx
yyy
aaa
zzz

Then I ran:

cat test | (sed '/y/ q'; echo aaa; cat)

and got:

xxx
yyy
aaa

Question

sed reads and prints until it encounters a line with 'y', then stops. In the first case, but not the second, cat reads and prints the rest.

Can someone explain what phenomenon is behind this difference in behavior?

I also noticed it works this way in Ubuntu 16.04 and Centos 6 but in Centos 7 neither command prints 'zzz'.

migrated from serverfault.com Aug 22 '16 at 10:24

This question came from our site for system and network administrators.

  • My guess is that cat (in the sub shell) can re-use the file descriptor in the first case, because stdin is bound to a real file. In the second case, stdin is from a pipe and not a real file. Note that also (sed '/y/ q'; echo aaa; cat) < <(cat test) does not print zzz. – Martin Nyolt Aug 22 '16 at 10:54
  • 1
    A simpler example: (head -n1; head -n1) < test and cat test | (head -n1; head -n1) – Martin Nyolt Aug 22 '16 at 11:02
22

When input file is seekable (like reading from regular file) or un-seekable (like reading from a pipe), sed (and other standard utilities) will behave differently (Read INPUT FILES section in this link).

Quote from the doc:

When a standard utility reads a seekable input file and terminates without an error before it reaches end-of-file, the utility shall ensure that the file offset in the open file description is properly positioned just past the last byte processed by the utility.

So in:

(sed '/y/ q'; echo aaa; cat) < test

sed performed quit command before reaching EOF, so it left file offset at beginning of zzz line, so cat can continue printing the remain lines (GNU sed is not POSIX compliant in some condition, see below).

And continuing from the doc:

For files that are not seekable, the state of the file offset in the open file description for that file is unspecified

In this case, the behavior is unspecified. Most standard tools, include sed will consume the input as much as possible . It read pass the yyy line, and quit without restoring the file offset, so nothing is left for cat.


GNU sed is not compliant to the standard, depends on system's stdio implementation and glibc version:

$ (gsed '/y/ q'; echo aaa; cat) < test
xxx
yyy
aaa

Here, the result was got from Mac OSX 10.11.6, virtual machines Centos 7.2 - glibc 2.17, Ubuntu 14.04 - glibc 2.19, which are run on Openstack with CEPH backend.

On those systems, you can use -u option to achieve the standard behavior:

(gsed -u '/y/ q'; echo aaa; cat) </tmp/test

and for pipe:

$ cat test | (gsed -u '/y/ q'; echo aaa; cat)
xxx
yyy
aaa
zzz

which leads to terribly inefficient performance, because sed has to read one byte at a time. A partial output from strace:

$ strace -fe read sh -c '{ sed -u "/y/q"; echo aaa; cat; } <test'
...
[pid  5248] read(3, "", 4096)           = 0
[pid  5248] read(0, "x", 1)             = 1
[pid  5248] read(0, "x", 1)             = 1
[pid  5248] read(0, "x", 1)             = 1
[pid  5248] read(0, "\n", 1)            = 1
xxx
[pid  5248] read(0, "y", 1)             = 1
[pid  5248] read(0, "y", 1)             = 1
[pid  5248] read(0, "y", 1)             = 1
[pid  5248] read(0, "\n", 1)            = 1
yyy
...
  • 1
    For GNU sed, that depends on the system's stdio implementation. On GNU systems (with the GNU libc), GNU sed will be compliant as exit() will lseek back for files managed by stdio. – Stéphane Chazelas Aug 22 '16 at 13:32
  • @StéphaneChazelas: How to verify it? With my Centos 7.2, Ubuntu 14.04 VM, sed isn't compliant, my manjaro laptop does, all have the same sed version 4.2.2 – cuonglm Aug 22 '16 at 13:55
  • @StéphaneChazelas: Sounds like something happened under the hood. On my virtual machines, strace -f sh -c '{ sed "/y/q"; echo aaa; cat; } <test' show that no lseek() was performed, while in my manjaro a lseek() was called before exit_group(). – cuonglm Aug 22 '16 at 14:13
  • I suppose that's down to the version of the GNU libc. You can test with a main() { char buf[999]; gets(buf); }' program. – Stéphane Chazelas Aug 22 '16 at 14:29
  • 1
    @StéphaneChazelas: Confirmed. Both my VMs have 2.17 and 2.19, while the one in my manjaro is 2.23. Is this consider a glibc bug? Do you have any information about the change between glibc versions – cuonglm Aug 22 '16 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.