17

The following bash commands go into an infinte loop:

$ echo hi > x
$ cat x >> x

I can guess that cat continues to read from x after it has started writing to stdout. What is confusing, however, is that my own test implementation of cat exhibits different behavior:

// mycat.c
#include <stdio.h>

int main(int argc, char **argv) {
  FILE *f = fopen(argv[1], "rb");
  char buf[4096];
  int num_read;
  while ((num_read = fread(buf, 1, 4096, f))) {
    fwrite(buf, 1, num_read, stdout);
    fflush(stdout);
  }

  return 0;
}

If I run:

$ make mycat
$ echo hi > x
$ ./mycat x >> x

It does not loop. Given the behavior of cat and the fact that I'm flushing to stdout before fread is called again, I would expect this C code to continue reading and writing in a cycle.

How are these two behaviors consistent? What mechanism explains why cat loops while the above code does not?

  • It does loop for me. Have you tried running it under strace/truss? What system are you on? – Stéphane Chazelas Sep 10 '14 at 20:23
  • It seems BSD cat has this behaviour and GNU cat reports an error when we try something like this. This answer discusses the same and I believe you are using BSD cat since I have GNU cat and when tested got the error. – Ramesh Sep 10 '14 at 20:28
  • I'm using Darwin. I like the idea that cat x >> x causes an error; however, this command is suggested in Kernighan and Pike's Unix book as an exercise. – Tyler Sep 10 '14 at 20:32
  • 3
    cat most likely uses system calls instead of stdio. With stdio, your program may be caching EOFness. If you start out with a file larger than 4096 bytes, do you get an infinite loop? – Mark Plotnick Sep 10 '14 at 20:33
  • @MarkPlotnick, yes! The C code loops when the file is over 4k. Thanks, perhaps that is the entire difference right there. – Tyler Sep 10 '14 at 20:35
12

On an older RHEL system I've got, /bin/cat does not loop for cat x >> x. cat gives the error message "cat: x: input file is output file". I can fool /bin/cat by doing this: cat < x >> x. When I try your code above, I get the "looping" you describe. I also wrote a system call based "cat":

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
int
main(int ac, char **av)
{
        char buf[4906];
        int fd, cc;
        fd = open(av[1], O_RDONLY);
        while ((cc = read(fd, buf, sizeof(buf))) > 0)
                if (cc > 0) write(1, buf, cc);
        close(fd);
        return 0;
}

This loops, too. The only buffering here (unlike for stdio-based "mycat") is what goes on in the kernel.

I think what's happening is that file descriptor 3 (the result of open(av[1])) has an offset into the file of 0. Filed descriptor 1 (stdout) has an offset of 3, because the ">>" causes the invoking shell to do an lseek() on the file descriptor before handing it off to the cat child process.

Doing a read() of any sort, whether into a stdio buffer, or a plain char buf[] advances the position of file descriptor 3. Doing a write() advances the position of file descriptor 1. Those two offsets are different numbers. Because of the ">>", file descriptor 1 always has an offset greater than or equal to the offset of file descriptor 3. So any "cat-like" program will loop, unless it does some internal buffering. It's possible, maybe even likely, that a stdio implementation of a FILE * (which is the type of the symbols stdout and f in your code) that includes its own buffer. fread() may actually do a system call read() to fill the internal buffer fo f. This may or may not change anything in the insides of stdout. Calling fwrite() on stdout may or may not change anything inside of f. So a stdio-based "cat" might not loop. Or it might. Hard to say without reading through a lot of ugly, ugly libc code.

I did an strace on the RHEL cat - it just does a succession of read() and write() system calls. But a cat doesn't have to work this way. It would be possible to mmap() the input file, then do write(1, mapped_address, input_file_size). The kernel would do all the work. Or you could do a sendfile() system call between the input and output file descriptors on Linux systems. Old SunOS 4.x systems were rumored to do the memory mapping trick, but I don't know if any one has ever done a sendfile-based cat. In either case the "looping" wouldn't happen, as both write() and sendfile() require a length-to-transfer parameter.

  • Thanks. On Darwin, it looks like the fread call cached a EOF flag as Mark Plotnick suggested. Evidence: [1] Darwin cat uses read, not fread; and [2] Darwin's fread calls __srefill which sets fp->_flags |= __SEOF; in some cases. [1] src.gnu-darwin.org/src/bin/cat/cat.c [2] opensource.apple.com/source/Libc/Libc-167/stdio.subproj/… – Tyler Sep 11 '14 at 16:21
  • 1
    This is awesome - I was the first to upvote it yesterday. It might be worth mentioning that the only POSIX-defined switch for cat is cat -u - u for unbuffered. – mikeserv Sep 11 '14 at 18:00
  • Actually, >> ought to be implemented by calling open() with the O_APPEND flag, which causes every write operation to (atomically) write to the current end of the file no matter what the file descriptor's position was prior to the read. This behavior is necessary for foo >> logfile & bar >> logfile to work correctly, for example -- you can't afford to assume that the position after the end of your own last write is still the end of the file. – Henning Makholm Apr 5 '17 at 20:52
1

A modern cat implementation (sunos-4.0 1988) uses mmap() to map the whole file and then calls 1x write() for this space. Such an implementation will not loop as long as the virtual memory allows to map the whole file.

For other implementations it depends on whether the file is larger than the I/O buffer.

  • Many cat implementations don't buffer their output (-u implied). Those will always loop. – Stéphane Chazelas Aug 25 '15 at 20:39
  • Solaris 11 (SunOS-5.11) doesn't appear to be using mmap() for small files (seems to resort to it only for files 32769 bytes large or above). – Stéphane Chazelas Aug 25 '15 at 20:45
  • Correct -u is usually the default. This does not imply a loop as an implementation can read the whole filesize and do just one write with that buf. – schily Aug 25 '15 at 21:23
  • Solaris cat only loops if the filesize is > max mapsize or if the initial fileoffset is != 0. – schily Aug 25 '15 at 21:33
  • What I observe with Solaris 11. It does a read() loop if the initial offset is != 0 or if the filesise is betwen 0 and 32768. Above that, it mmaps() 8MiB large regions of the file at a time and never seem to revert to read() loops even for PiB files (tested on sparse files). – Stéphane Chazelas Aug 25 '15 at 21:47
0

As written in Bash pitfalls, you cannot read from a file and write to it in the same pipeline.

Depending on what your pipeline does, the file may be clobbered (to 0 bytes, or possibly to a number of bytes equal to the size of your operating system's pipeline buffer), or it may grow until it fills the available disk space, or reaches your operating system's file size limitation, or your quota, etc.

The solution is to either use text editor, or temporary variable.

-1

You have some kind of race condition between both x. Some implementations of cat (e.g. coreutils 8.23) forbid that:

$ cat x >> x
cat: x: input file is output file

If this is not detected, the behavior will obviously depend on the implementation (buffer size, etc.).

In your code, you could try to add a clearerr(f); after the fflush, in case the next fread would return an error if the end-of-file indicator is set.

  • It seems that a good OS will have deterministic behavior for a single process with a single thread running the same read/write commands. In any case, the behavior is deterministic for me, and I'm mainly asking about the discrepancy. – Tyler Sep 10 '14 at 20:33
  • @Tyler IMHO, without clear specification on this case, the above command makes no sense, and determinism isn't really important (except an error like here, which is the best behavior). This is a bit like C's i = i++; undefined behavior, hence the discrepancy. – vinc17 Sep 10 '14 at 20:41
  • 1
    No, there is no race condition here, the behavior is well-defined. It is however implementation-defined, depending on the relative size of the file and the buffer used by cat. – Gilles Sep 11 '14 at 8:45
  • @Gilles Where do you see that the behavior is well-defined/implementation-defined? Can you give some reference? POSIX cat specification just says: "It is implementation-defined whether the cat utility buffers output if the -u option is not specified." However, when a buffer is used, the implementation doesn't have to define how it is used; it may be non-deterministic, e.g. with a buffer flushed at random time. – vinc17 Sep 11 '14 at 12:22
  • @vinc17 Please insert “in practice” in my previous comment. Yes, that's theoretically possible and POSIX-compliant, but nobody does it. – Gilles Sep 11 '14 at 15:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.