13

Aside from using a temporary file to help, is there a way/program could buffer input from stdin but does not output the contents until getting the EOF. I don't want to use a shell variable either(e.g. buffer=$(cat)).

This program should behave as below(assume the program name is buffered-cat):

$ buffered-cat
line 1
line 2
line 3
^D # Ctr-D here(End of Line)

Now that the program received ^D, the buffered-cat outputs the contents

line 1
line 2
line 3
1
  • 1
    I normally use sort for this purpose. Obviously it has side-effects that may be undesirable. – Adrian Pronk Jan 19 '17 at 8:32
20

A poor man's sponge using awk:

awk '{a[NR] = $0} END {for (i = 1; i <= NR; i++) print a[i]}'

If you have tac, you can misuse it too:

... | tac | tac
4
  • 1
    tac is simple and elegant, I think. – Ekeyme Mo Jan 13 '17 at 7:11
  • 6
    @EkeymeMo | tac | tac is simple, but bear in mind that it is highly inefficient for the task and you'll suffer a performance hit, especially for larger files. – Digital Trauma Jan 13 '17 at 17:22
  • @DigitalTrauma yeah, I know the drawback of that. I test a file that with 100 thousands line. It's ok for me. I voted up your comment as it is true. – Ekeyme Mo Jan 14 '17 at 0:22
  • Depending on the version of awk, it may or may not deal well with NUL bytes in the input. (As far as I tested, GNU awk dealt with them fine, mawk and Busybox awk corrupted the input in different ways.) The awk version adds a trailing newline if it's missing from the input, and tac also corrupts the input if the trailing newline is missing. (printf 'foo\nbar' | tac | tac outputs barfoo<newline>) – ilkkachu Mar 22 at 19:02
30

You can do this with sponge from moreutils. sponge will "soak up standard input and write to a file". With no arguments, that file is standard output. Input given to this command is stored in memory until EOF, and then written out all at once.

For writing to a normal file, you can just give the filename:

cmd | sponge filename

The main purpose of sponge is to allow reading and writing from the same file within a pipeline, but it does what you want as well.

0
6

As long as your input is ASCII text (contains no NUL 0x0 bytes until the end), then sed -z does what you want:

$ sed -z ''
Line 1
Line 2
Line 3
^D
Line 1
Line 2
Line 3
$ 

The -z causes sed to treat the NUL byte as a line delimiter instead of the usual newline. So as long as your input is regular text with no NUL bytes, then sed will continue reading the whole input into its pattern buffer until EOF is reached. sed then does no processing on the buffer and outputs it.


If NUL bytes are present in your input, then you can do this instead:

sed ':l;N;bl'
1
  • Similarly to awk, this may depend on the version of sed. E.g. Busybox's sed doesn't appear to deal with NULs nicely, while GNU sed looks to be ok. – ilkkachu Mar 22 at 19:10
4

This sed solution is a little longer than DigitalTrauma's, but also works with NUL bytes.

sed -n 'H;${x;s/^\n//;p}'
2
  • 2
    sed ':l;N;bl' should work too. – Digital Trauma Jan 14 '17 at 0:50
  • Similarly to awk, this may depend on the version of sed. E.g. Busybox's sed doesn't appear to deal with NULs nicely, while GNU sed looks to be ok. – ilkkachu Mar 22 at 19:10
1

Same idea as muru's awk example, except in Python. Use CtrlD to stop reading in lines

$ python -c 'import sys;print("".join(sys.stdin.readlines()))'                                                           
line1
line2
line3 # press Enter and Ctrl+D at this point
line1
line2
line3
2
  • 4
    Or perl -e 'print <>' (!) – dave_thompson_085 Jan 15 '17 at 15:54
  • This looks to add an extra newline at the end, though you could fix that by adding end="" to print(). Or using sys.stdout.write() instead. – ilkkachu Mar 22 at 19:13
1

You can use the following:

#!/bin/bash
stdin="$(cat /dev/stdin ; echo . )"
# To preserve newlines at the end we add a "." above and remove it below. See: https://unix.stackexchange.com/a/383411/62628
stdin=${stdin%.}
echo -n "$stdin"
$ buffered-cat
line 1
line 2
line 3
^D # Ctr-D here(End of Line)
line 1
line 2
line 3
$ buffered-cat

But since the command is called bufferred-cat you might want to be able to pass all options of cat. In this case:

#!/bin/bash
stdin="$(cat /dev/stdin ; echo . )"
stdin=${stdin%.}
cat "$@" <(echo -n "$stdin")
$ buffered-cat -n
line 1
line 2
line 3
^D # Ctr-D here(End of Line)
    1 line 1
    2 line 2
    3 line 3
$ buffered-cat
0
0

The behavior you are asking for, isn't it the default behavior of simple cat?

gv@debian:$ cat << EOF #or cat <<EOF >file or cat <<EOF >/dev/stdout
> Line 1
> Line 2
> Line 3
> EOF
Line 1
Line 2
Line 3
gv@debian:$
2
  • 3
    No, that's the behavior of here documents.  Try typing cat(Enter) Line 1(Enter) and see what happens. – G-Man Says 'Reinstate Monica' Jan 14 '17 at 4:11
  • Yep, that's here-doc structure. Simple and gets pretty close to what OP wants. Doesn't require installation either – Sergiy Kolodyazhnyy Jan 14 '17 at 5:44
0

A trivial one-line sponge-replacement in Perl:

perl -e 'print <>'

(from @dave_thompson_085, in a comment)

Perl should be able to deal with NUL bytes too.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.