Bash - Redirect output to a variable or file descriptor, then read from the variable or file descriptor

Question

Here's an example of a bash script that redirects all output to a file (and shows output on the screen too):

# writing to it
exec > >(tee --ignore-interrupts ./log)
exec 2>&1

echo "here is a message"

# reading from it again
cat ./log

Now I don't want to create the file ./log. I want to keep all the stdout in memory, and be able to read from it again later in the script. Also I'm in a situation where there may be no root filesystem mounted.

I tried to do this with process substitution instead of ./log, but I can't seem to make sense of how to pass the file descriptor created by process substitution for a subsequent command in order to read what I just wrote.

Here's another example:

# can I make ./log a temporary file descriptor, an in-memory buffer?
exec 3>./log 4<./log
# writes
echo >&3 "hello"
# reads
cat <&4

Answer 1

If you want to globally redirect everything that happens to be written (like you do now), it's tricky, but can be hacked together.

I strongly recommend that, if it's possible, just do it by normal piping. just wrap everything you do in a subshell. In this case

(
 echo "this is the message"
 other stuff
) | cat

or just write everything into a variable with "$()" syntax.

The next way is to use what you did, but write to a tmpfs or /dev/shm if they are available. That's pretty straight forward, but you have to know what ram-based filesystems are in place (and set them up if possible).

Another way is to create a fifo with mkfifo. In both cases, you need to clean up after yourself.

EDIT:

I have a very ugly hack, but I bet someone can improve it.

#!/bin/bash
exec 3>&1
exec > >( tee >( ( tac && echo _ ) | tac | (read && cat > ./log) ) )

echo "lol"
sleep 5
echo "lol"

echo "finished writing"

exec >&-
exec >&3
exec 3>&-
echo "stdout now reopen"
sleep 1 #wait if the file is still being written asynchronously
cat ./log

How it works: first, you have a tee so you can see what's going on. This in turn outputs to another process substitution. There, you have the trick tac|tac which (because tac needs entire input to start outputting) waits for the entire stream to finish before going on. The last piece is in a subshell that actually outputs this into a file. Of course, the final shell would, immeadiately upon instantiation, create the output file in the filesystem if that was the only line. So something that also waits for the input to finally come, has to be done first, to delay file creation. I do this by outputting a dummy line first with echo, and then reading and discarding it. The read blocks until you close the file descriptor, signalling to tac its time has come. Hence, the closing of the stdout file descriptor at the end. I also saved the original stdout before opening the process substitution, in order to restore it at the end (to use cat once more). There's a sleep 5 in there, so I could check with ls if the file really wasn't created too early. The final sleep is trickier... The subshell is asynchronous and if there is a lot of output, you are waiting for both tacs to do their thing before the file is really there. So reasonably, you'll probably need to do something else to check if the thing really is finished. For instance, && touch sentinel at the end of the last subshell, and then while [ ! -f sentinel ]; do sleep 1; done && rm sentinel before you finally use the file.

All in all, two process substitutions and in the inner one another two subshells and 2 pipes. It's one of the ugliest things I've ever written... but it should create the file only when you close the stdout, which means it's well controlled and can be done when your filesystems are ready.

Answer 2

simple case:

output="$(echo "here is a message")"
# [...]
echo "$output"

Not so simple if you have to read from a redirection and cannot use a pipeline like

echo "$output" | while IFS= read -r line; do :; done

You can use a FIFO and a background process:

mkfifo fifo
echo "$output" >fifo &
while IFS= read -r line; do :; done <fifo

tee version

mkfifo fifo
( output="$(cat fifo)"; echo "$output" >fifo ) &
exec 3>&1
exec >fifo
...
exec 1>&3
cat fifo