How to write a script that can take input from stdout

Question

I want to be able to write a script that can take stdout as an argument, if anything is piped into it (ultimately, I would like it to be polymorphic) -

The trouble is, I have searched and searched for how to do this with no avail - lots of alternative suggestions about how to do other things that are not - this:

cat /var/log/some.log | grep something | awk '{print $1 $6 $8}' | myscript

Why do that, instead of? : myscript $(!!) At this point, solely to prove that it is possible...

I know that you can 'read variable' in a script, but say I don't care about the lines - let's say, I want to accept the whole of it as a blob of text and do something with it in the script -

Do I really have to :

while read x; do
stdin=$stdin" "$x;
done;

solely in order to read from STDIN ?

There must be a better way ...

Answer 1

You can read from stdout by redirecting input from file descriptor 1. Stdout is file descriptor 1 by definition. The fact that file descriptor 1 is used for output is a matter of convention, not a technical obligation. However it's a bizarre thing to do which is bound to confuse the people who use your script.

read line <&1

If you want to read a whole file, use cat. To stuff the content into a variable, use a command substitution.

whole_input=$(cat)
whole_input_from_stdout=$(cat <&1)

Some shells let you write $(</dev/stdin) as a slightly more efficient shortcut for $(cat).

Note that this strips trailing newlines. This behavior is built into command substitution. To retain trailing newlines, hide them behind another character and remove that character.

whole_input=$(cat; echo .); whole_input=${whole_input%?}

The shell variable will only contain the data up to the first null byte, if there is a null byte in the input. With some shells, you'll get all the data with the null bytes stripped. Shells can't deal with binary data. Zsh is an exception, it retains the null bytes.

Answer 2

If you want to read all of stdin into a shell script, usually you just capture it into a temp file:

TMPFILE=$(mktemp /tmp/$0.$$.XXXXXX)
cat > "$TMPFILE"
# Script works with $TMPFILE and its contents,
# ultimately writing everything to stdout.
rm -f "$TMPFILE"

Even system utilities do things very much like this. sort has to have all of stdin before it can print anything to stdout, for example.

Answer 3

You don't have to get the whole lump if you don't want to - you can chunk out stdin and work it as a stream if you like. I did some googling, though, and I don't think I can offer you any advice on how to change the sex of your script, or whatever it is you meant by polymorphic.

In any case, I typically find that putting the whole of input aside in some storage is rarely what I'm after. I usually want to handle it in delimited lumps of some kind. Here's an example of how you can get input of any kind split out into 4k chunks per while loop iteration:

splitin(){
    dd obs=4k | { j=$1 f=$2;shift 2 && 
    while dd bs=4k count=1 of="$f"  &&
          [ -s "$f" ] 
    do    "$j" "$@" < "$f"
    done
};  }

...which is a function you might call from your shell script with the name of a some job you want to perform on 4k input intervals and the name of a temp file it can use to store each latest chunk. Like:

splitin handle_chunk /tmp/work/chunk \
        other args to pass on as appropriate