How does a program know what to read in a pipeline? [duplicate]

Question

How does the unix command line program know what file to read?

For example:

 cat someFile | foo

How does the program foo know which file to read, and which process is responsible for opening and reading the file from disk?

Answer 1

foo does not know what file to read, it (presumably) just reads from its standard input stream. This data stream is connected by the shell to the standard output stream of the cat command. This "administrational plumbing" is done by the shell as it starts the two processes (they would run concurrently).

In the example in the question, it is cat that will open the file for reading, read it and pass the data on its standard output.

You might as well ask "How does cat know where to write the result?" The answer is that it writes to its standard output stream, connected by the shell to the standard input stream of the foo command. Likewise, the standard output stream of the foo command is connected to the terminal by the shell since there are no further pipes or redirections for this command.

---

The pipeline that you show,

cat someFile | foo

is functionally identical to

foo <somefile

Here I've deleted cat since it's not really needed. The shell will connect the standard input stream of foo to the given file, so the effect will be the same (foo would be able to read the contents of somefile from its standard input stream).

In this last command, foo still does not know that it reads data coming from a file called somefile. It also does not know it's no longer reading from the output of cat. It just reads its standard input stream as before.

It is now the shell that will open the file somefile for reading, but the shell will not read anything from the file, just hook the standard input stream up with the opened file for foo to read from.

---

Note that we don't know what the foo command does, whether it actually does anything with its standard input stream, or whether it expects a filename to read from on its command line. This sort of information would be available in the manual of the foo program.

In the case where foo actually needs to be told to read from a specific file by giving it a pathname on the command line, you may use

foo somefile

Now foo would be responsible for both opening and reading from the file.

If the file has to be processed in some way (assuming "cat" is a more complicated process that actually modifies the data read from somefile):

cat somefile >newfile
foo newfile
rm newfile

I.e., process the file and save the result in a new temporary file, then pass the name of the temporary file to foo. Then remove the temporary file.

Or, with a shell that understands process substitutions (like bash):

foo <( cat somefile )

Here, the shell would arrange for the output of cat somefile to be written to a temporary file (or named pipe, it does not really matter), and the pathname of that data would be inserted in place of the process substitution <( ... ). foo would then open that as its file to read from.

In this final example, cat would open the original file and read from it while foo would open whatever pathname the shell gives it (where the output of cat is found) and read from it.

Answer 2

In cat someFile | foo there are three elements:

1. cat someFile
2. the pipe |
3. foo

And now what's going on:

1. cat knows it should read someFile, because that's how it was called, and so the sell tells it the details. Namely, someFile is a parameter here and all parameters are passed on to the called app.

2. The pipe is what makes shell fork subshells and start processes in them, as well as arrange the subshells and processes, setting inter alia their input and output. That's how foo gets the input from cat.

3. foo is called the same as cat before. But after step two (which in fact is step 1 in the shell) the input to foo comes from cat. That's how the pipeline was called.

Answer 3

You wrote two commands: cat someFile and foo. First command has to read someFile and writes it to standard output which is redirected to pipe. The next command reads the data from its standard input to which is redirected to the output from the pipe.

Standard input, standard output and standard error output are the basic linux terms and redirection and pipe are the most powerfull principle to flow the data between the programs. Pleas, look for these terms to understand the basics linux command line.