9

Consider this script:

tmpfile=$(mktemp)

cat <<EOS > "$tmpfile"
line 1
line 2
line 3
EOS

cat <(tail -1 "$tmpfile") "$tmpfile"

This works and outputs:

line 3
line 1
line 2
line 3

Let's say that our input source, rather than being an actual file, was instead stdin:

cat <<EOS | # what goes here now?
line 1
line 2
line 3
EOS

How do we modify the command:

cat <(tail -1 "$tmpfile") "$tmpfile"

So that it still produces the same output, in this different context?

NOTE: The specific Heredoc I'm catting, as well as the use of a Heredoc itself, is merely illustrative. Any acceptable answer should assume that it is receiving arbitrary data via stdin.

3
  • 1
    stdin is always an "actual file" (a fifo/socket/etc is a file too; not all files are seekable). The answer to your question is either a trivial "use a temporary file" or some horror which will load the whole file in the memory. "How can I retrieve old data from a stream without having stored it anywhere?" cannot have a good answer.
    – user313992
    Mar 31, 2019 at 19:29
  • 1
    @mosvy That's a perfectly acceptable answer if you'd like to add it.
    – Jonah
    Mar 31, 2019 at 19:34
  • 2
    @mosvy As Jonah has said, answers should be posted in the answer box. I know it's tricky to read any of the website at the moment, but please ignore the red that's slowly dripping over your vision and use the lower textarea.
    – wizzwizz4
    Mar 31, 2019 at 20:35

6 Answers 6

7

Try:

awk '{x=x $0 ORS}; END{printf "%s", $0 ORS x}'

Example

Define a variable with our input:

$ input="line 1
> line 2
> line 3"

Run our command:

$ echo "$input" | awk '{x=x $0 ORS}; END{printf "%s", $0 ORS x}'
line 3
line 1
line 2
line 3

Alternatively, of course, we could use a here-doc:

$ cat <<EOS | awk '{x=x $0 ORS}; END{printf "%s", $0 ORS x}'
line 1
line 2
line 3
EOS
line 3
line 1
line 2
line 3

How it works

  • x=x $0 ORS

    This appends each line of input to the variable x.

    In awk, ORS is the output record separator. By default, it is a newline character.

  • END{printf "%s", $0 ORS x}

    After the we have read in the whole file, this prints the last line, $0, followed by the contents of the whole file, x.

Since this reads the whole input into memory, it would not be appropriate for large (e.g. gigabyte) inputs.

1
  • Thanks John. So is it not possible to do this in a way analogous to my named file example in the OP? I was imagining the stdin being duplicated somehow... sort of the way tee does, but of a stdin and a file, we'd be piping the same stdin into two different process substitutions. or anything that would be roughly equivalent to that?
    – Jonah
    Mar 30, 2019 at 19:34
5

If stdin points to a seekable file (like in the case of bash's (but not all other shell's) here documents which are implemented with temp files), you can get the tail and then seek back before reading the full contents:

seek operators are available in the zsh or ksh93 shells, or scripting languages like tcl/perl/python, but not in bash. But you can always call those more advanced interpreters from bash if you have to use bash.

ksh93 -c 'tail -n1; cat <#((0))' <<...

Or

zsh -c 'zmodload zsh/system; tail -n1; sysseek 0; cat' <<...

Now, that won't work when stdin points to a non-seekable files like a pipe or socket. Then, the only option is to read and store (in memory or in a temporary file...) the whole input.

Some solutions for storing in memory have already been given.

With a tempfile, with zsh, you could do it with:

seq 10 | zsh -c '{ cat =(sed \$w/dev/fd/3); } 3>&1'

If on Linux, with bash or zsh or any shell that uses temp files for here-documents, you could actually use the temp file created by a here-document to store the output:

seq 10 | {
  chmod u+w /dev/fd/3 # only needed in bash5+
  cat > /dev/fd/3
  tail -n1 /dev/fd/3
  cat <&3
} 3<<EOF
EOF
4
cat <<EOS | sed -ne '1{h;d;}' -e 'H;${G;p;}'
line 1
line 2
line 3
EOS

The issue with translating this to something that uses tail is that tail needs to read the whole file to find the end of it. To use that in your pipeline, you need to

  1. Provide the full contents of the document to tail.
  2. Provide it again to cat.
  3. In that order.

The tricky bit is not to duplicate the document's content (tee does that) but to get the output of tail to happen before the rest of the document is outputted, without using an intermediate temporary file.

Using sed (or awk, as John1024 does) gets rid of the double parsing of the data and the ordering issue by storing the data in memory.

The sed solution that I propose is to

  1. 1{h;d;}, store the first line in the hold space, as-is, and skip to the next line.
  2. H, append each other line to the hold space with an embedded newline.
  3. ${G;p;}, append the hold space to the last line with an embedded newline and print the resulting data.

This is quite a literal translation of John1024's solution into sed, with the caveat that the POSIX standard only guarantees that the hold space is at lest 8192 bytes (8 KiB; but it recommends that this buffer is dynamically allocated and expanded as needed, which both GNU sed and BSD sed is doing).


If you allow yourself to use a named pipe:

mkfifo mypipe
cat <<EOS | tee mypipe | cat <( tail -n 1 mypipe ) -
line 1
line 2
line 3
EOS
rm -f mypipe

This uses tee to send the data down mypipe and at the same time to cat. The cat utility will first read the output from tail (which reads from mypipe, which tee is writing to), and then append the copy of the document coming directly from tee.

There's a serious flaw in this though, in that if the document is too large (larger than the pipe's buffer size), tee's writing to mypipe and cat would block while waiting for the (unnamed) pipe to empty. It would not be emptied until cat read from it. cat would not read from it until tail had finished. And tail would not finish until tee had finished. This is a classic deadlock situation.

The variation

tee >( tail -n 1 >mypipe ) | cat mypipe -

has the same issue.

2
  • 3
    The sed one doesn't work if the input has only one line (maybe sed '1h;1!H;$!d;G'). Also note that several sed implementations have a low limit on the size of their pattern and hold space. Mar 30, 2019 at 20:30
  • The named pipe solution is the kind of thing i was looking for. The limitation is a shame. I understood your explanation except for “And tail would not finish until tee had finished” — could you elaborate on why that’s the case?
    – Jonah
    Mar 31, 2019 at 12:25
2

There is a tool named pee in a collection of command-line utilities usually packaged with the name "moreutils” (or otherwise retrievable from its home website).

If you can have it on your system then the equivalent for your example would be like:

cat <<EOS | pee 'tail -1' cat 
line 1
line 2
line 3
EOS

Ordering of the commands run through pee is important because they get executed in the sequence provided.

0

If you don't care about the order. Then this will work cat lines | tee >(tail -1). As others have said. You need to read file twice, or buffer the whole file, to do it in the order you asked for.

-1

Try:

cat <<EOS # | what goes here now? Nothing!
line 3
line 1
line 2
line 3
EOS

Since the whole thing is literal data (a "here-is document"), and the difference between it and the desired output is trivial, just massage that literal data right there to match the output.

Now suppose line 3 comes from somewhere and is stored in a variable called lastline:

cat <<EOS # | what goes here now? Nothing!
$lastline
line 1
line 2
$lastline
EOS

In a here doc, we can generate text by substituting variables. Not only that but we can calculate text using command substitution:

cat <<EOS
this is template text
here we have a hex conversion: $(printf "%x" 42)
EOS

We can interpolate multiple lines:

cat <<EOS
multi line
preamble
$(for x in 3 1 2 3; do echo line $x ; done)
epilog
EOS

In general, avoid text processing the here doc template; try to generate it using interpolated code.

10
  • 1
    I honestly can't tell if this is a joke or not. The cat <<EOS... in the OP was just an example standin for "catting an arbitrary file," to make the post specific and the question clear. Was that really not obvious to you, or did you just think it would be clever to interpret the question literally?
    – Jonah
    Mar 31, 2019 at 17:55
  • @Jonah The question clearly says "[l]et's say that our input source, rather than being an actual file, was instead stdin:". Nothing about "arbitrary files"; it's about here docs. A here doc is not arbitrary. It's not an input to your program, but a piece of its syntax that the programmer chooses.
    – Kaz
    Mar 31, 2019 at 19:07
  • 1
    I think the context and existing answers made it clear that was the case, if only because for your interpretation to be correct you literally had to assume that neither I nor any of the other posters who replied realized that it was possible to copy and paste a line of code. Nevertheless, I will edit the question to make it explicit.
    – Jonah
    Mar 31, 2019 at 19:13
  • 1
    Kaz, thank you for the reply, but note even with your edit, you are missing the intention of the question. You are receiving arbitrary multiline input via a pipe. You have no idea what it will be. Your task is to output the last line of input, followed by the entire input.
    – Jonah
    Mar 31, 2019 at 19:19
  • 1
    Kaz, the input is there only as an example. Most people, including myself, find it helpful to have an example of real input and expected output, rather than only the abstract question. You are the only one who was confused by this.
    – Jonah
    Mar 31, 2019 at 19:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .