3

I have an application which requires a producer to send filenames to a consumer, and have producer indicate to the consumer when the last filename has been sent and the end of file has been reached.

For simplicity, in the following example producer is demonstrated with echo and printf, while the consumer is demonstrated with cat. I have tried to extrapolate the "here file" method without success, using <<EOF to indicate to the producer-wrapper (if such a thing exists) what to look for as an indication of end of file. If it worked cat should filter EOF from the output.

Ex 1)

input

{
echo "Hello World!" 
printf '\x04' 
echo "EOF"
} <<EOF |\
cat

output

bash: warning: here-document at line 146 delimited by end-of-file (wanted `EOF')
Hello World!
EOF

Ex 2)

input

{ 
echo "Hello World!" 
printf '\x04' 
echo "EOF"
} |\
cat <<EOF

output

bash: warning: here-document at line 153 delimited by end-of-file (wanted `EOF')

Is it correct that the "here files" method for indicating delimiter only works for static text, and not dynamically created text?

-- the actual application --

inotifywait -m --format '%w%f' /Dir |  <consumer>

The consumer is waiting for files to be written to directory /Dir. It would be nice if when a file "/Dir/EOF" was written the consumer would detect logical end-of-file condition simply by writing shell script as follows:

inotifywait -m --format '%w%f' /Dir |<</Dir/EOF  <consumer>

-- In response to Giles answer --

Is it theoretically possible to implement

cat <<EOF
hello
world
EOF

as

SpecialSymbol="EOF"
{
    echo hello
    echo world
    echo $SpecialSymbol
} |\
while read Line; do 
  if [[ $Line == $SpecialSymbol ]]
    break
  else 
    echo $Line
  fi
done |\
cat

By theoretically possible I mean "would it support existing usage patterns and only enable extra usage patterns which had previously been illegal syntax?" - meaning no existing legal code would be broken.

  • 3
    Why not have the sender close the pipe after it is done sending files? – thrig Aug 30 '17 at 23:33
  • 1
    @CraigHicks: shell statements in 'round brackets' (US parentheses) run in a subshell but do not by themselves involve any kind of pipes. You may be thinking of process substitution using 'angle' characters and parentheses >( command ) and <( command )which in some implementations sometimes use pipes, but if those pipes do have names using the name would only allow you to open them redundantly NOT close them. – dave_thompson_085 Sep 1 '17 at 15:19
  • You don't need escape newline after | – Hauke Laging Sep 2 '17 at 0:04
6

For a pipe, the end of file is seen by the consumer(s) once all the producers have closed their file descriptor to the pipe and the consumer has read all the data.

So, in:

{
  echo foo
  echo bar
} | cat

cat will see end-of-file as soon as the second echo terminates and cat has read both foo\n and bar\n. There's nothing more for you to do.

Things to bear in mind though is that if some of the commands on the left side of the pipe starts some background process, that background process will inherit a fd to the pipe (its stdout), so cat will not see eof until that process also dies or closes its stdout. As in:

{
  echo foo
  sleep 10 &
  echo bar
} | cat

You see cat not returning before 10 seconds have passed.

Here, you may want to redirect sleep's stdout to something else like /dev/null if you don't want its (non)output to be fed to cat:

{
  echo foo
  sleep 10 > /dev/null &
  echo bar
} | cat

If you want the writing end of the pipe to be closed before the last command in the subshell left of the | is run, you can close stdout or redirecting to that subshell in the middle of the subshell with exec, like:

{
  echo foo
  exec > /dev/null
  sleep 10
} | (cat; echo "cat is now gone")

However note that most shells will still wait for that subshell in addition to the cat command. So while you'll see cat is now gone straight away (after foo is read), you'll still have to wait 10 seconds for the whole pipeline to finish. Of course, in that example above, it would make more sense to write it:

echo foo | cat
sleep 10

<<ANYTHING...content...ANYTHING is a here-document, it's to make the stdin of command a file that contains the content. It wouldn't be useful there. \4 is byte that when read from a terminal makes data held by a terminal device be flushed to the application reading from it (and when there's no data, read() returns 0 which means end-of-file). Again, not of any use here.

  • - It answers exactly how the un-named pipe is closed in two way: (1) the normal way e.g. ending the process of the pipe writer (2) using the trick of "exec >/dev/null". Therefore I select this as the answer. – Craig Hicks Sep 1 '17 at 18:19
  • I have modified my question to make it clearer what I was trying to achieve by suggesting a new and backwards-compat use for "<< <special symbol>". – Craig Hicks Sep 1 '17 at 21:35
4

The problem with your attempt is that what you call “the here-file method” simply doesn't exist. Here documents are a feature of the syntax of the shell programming language. They don't get interpreted by applications such as cat.

Consider a shell script with a here document:

cat <<EOF
hello
world
EOF

The way it works is that as part of parsing the script, the shell interpreter sees the here document syntactic construct. When it runs these lines of code, the interpreter gathers the data and passes it as input to cat. Depending on the shell, this is essentially equivalent to either

echo 'hello
world
' >/tmp/heredoc.tmp
cat </tmp/heredoc.tmp
rm /tmp/heredoc.tmp

(but with saner management of the temporary file) or to

echo 'hello
world
' | cat

If you have echo EOF and you want this to indicate the end of a here document, then the output of the echo command must be a shell.

{
  echo 'cat <<EOF'
  echo hello
  echo world
  echo EOF
} | sh

This works but it's rarely useful.

None of this helps you to do what you set out to do. There's nothing magical about a here document that makes it end “harder” than any other input. End-of-file is not a marker that gets transmitted over the input channel, it's a condition of the input channel. When the input is a regular file, the end-of-file condition is triggered when the application tries to read past the last byte of the file. When the input is a pipe (named or not), the end-of-file condition is triggered when the application tries to read but the producer has closed the pipe.

Usually it's enough to write

producer | consumer

When producer exits, the write end of the pipe will be closed since no process has it open any longer. If you want the pipe to be closed earlier, arrange for the producer to close its standard output (exec >&- in sh, close(1) in C).

Note that to reach end-of-file on a pipe, all the processes that had the write end of the pipe open must have closed it. If the producer runs background processes, you may want to set the close-on-exec flag on the pipe (fcntl(1, F_SETFD, fcntl(1, F_GETFC, 0) | FD_CLOEXEC)) before forking them.

  • Terrific answer and you include the stdout switch "exec >&-". Steph also had the output redirect in his answer and it was an earlier response , so I selected that as the answer. No hard feelings I hope. Apart from that, I have modified my question in response to your answer about underlying implementation - I would be grateful if you have time to read it. – Craig Hicks Sep 1 '17 at 21:33
  • @CraigHicks Regarding your addition: that doesn't make sense. You've written a program that prints hello world in a very complicated way, why would you want to do this? – Gilles Sep 1 '17 at 21:35
  • if you look at my modified question in the new section "-- the actual application --" you will see that it is not about another way to write "hello world", but about a way to pass locally-specified special text symbols as a method to close pipes. The explicit example uses "inotifywait" – Craig Hicks Sep 1 '17 at 21:39
  • @CraigHicks If you want to wait for one event with inotifywait, don't pass the -m flag! And once again, you do not close a pipe by sending data to it. You close a pipe… by closing the pipe! – Gilles Sep 1 '17 at 21:52
  • If you don't specify the -m flag, then inotifywait will return at the first event. If inotifywait is called in a loop, then between returning from one wait and entering the next, events could be missed. (Unless of course inotify doesn't stop waiting for events, but there is nothing in the documentation for the version of inotifywait I am using (Ubuntu 16.04.3) that that is the case. (And if it were how would the waiting stop? No handle is returned from inotifywait). – Craig Hicks Sep 1 '17 at 22:37
3

If you want to use a FIFO then you can do so, of course:

script 1

FIFO_PATH="/path/to/fifo"
exec 3<"$FIFO_PATH" # stalls until FIFO is opened for writing, too
while read line; do
    : whatever
done <&3
exec 3<&-
: continue script

script 2

FIFO_PATH="/path/to/fifo"
exec 3>"$FIFO_PATH" # stalls until FIFO is opened for reading, too
echo foo >&4
exec 4>&-
: continue script
  • Very compact. Thanks for taking the time to explain about named pipes. – Craig Hicks Aug 31 '17 at 14:26
  • I have modified my question to make it clearer what I was trying to achieve by suggesting a new and backwards-compat use for "<< <special symbol>". – Craig Hicks Sep 1 '17 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.