Store file outputs of command in bash variables

Question

Suppose I have a utility foo that writes to n different files simultaneously. Different data is written to every file. What is written to the files is not known in advance. For example, the utility might be invoked like:

foo file_1 file_2 ... file_n

I want to store what is written to the files in bash variables. Of course, I could let the utility write to actual files on the file system and then read from them:

foo file_1 ... file_n

output_1="$(cat file_1)"
...
output_n="$(cat file_n)"

rm file_1 ... file_n

However, I think it might be more efficient to skip the file system and avoid creating temporary files. How can this be done? Can named pipes maybe be used for this?

Answer 1

I think I found an answer. The problem with @terdon's solution is, as @ilkkachu mentioned, that it assumes the files to be written in sequential order. It first reads from file1 until completion, and then continues to file2. However, foo might write to file1 and file2 at the same time, and because no reader is attached to file2, it blocks.

This could be fixed by opening every fifo for reading right after invoking foo, and then reading from them in sequential order. However, this only works as long as foo does not write more than the fifo buffer size.

Every file/fifo thus needs to be read concurrently. For each fifo we can create a subshell that runs in the background and reads from the fifo until completion. The subshell then prints the content to stdout. It thereby serves as a buffer for the data that is written to the fifo.

# assume files don't exist yet
mkfifo "$@"

# opens fifos as writing
foo "$@" &

# for each fifo, create background subshell that reads from it
i=0
for file in "$@"; do
    exec {fd}< <(echo "$(cat $file)")
    fds[$i]=$fd
    (( i++ ))
done

# fifos are now fully connected
# consume content buffered in subshells
i=0
for file in "$@"; do
    file_contents[i]="$(cat <&${fds[$i]})"
    (( i++ ))
done

for (( i=0; i<${#file_contents[@]}; i++)); do
  printf "The contents of file number %d are: %s\n" "$(( $i+1 ))" "${file_contents[$i]}"
done

Answer 2

Yes, this is exactly what named pipes (also known as FIFOs) are for. Here's a toy example:

#!/bin/bash
i=0
for file in "$@"; do
  mkfifo "$file"
  printf "This is file '%s'\n" "$file" > "$file" &
  file_contents[i]=$(cat < "$file")
  rm "$file"
  (( ++i ))
done

for (( i=0; i<${#file_contents[@]}; i++)); do
  printf "The contents of file number %d are: %s\n" "$i" "${file_contents[i]}"
done

This looks like you're creating and deleting files, and formally you are; but they're named pipes and not regular files and therefore no data are actually written to disk.

However, this still uses the filesystem, strictly speaking, it just uses it minimally. The file has no contents, so nothing is actually written to disk, but there will be a filesystem entry created for it. From man fifo:

DESCRIPTION
       A  FIFO  special file (a named pipe) is similar to a pipe, except that
       it is accessed as part of the filesystem.  It can be opened by  multi‐
       ple  processes  for reading or writing.  When processes are exchanging
       data via the FIFO, the kernel passes all data internally without writ‐
       ing it to the filesystem.  Thus, the FIFO special file has no contents
       on the filesystem; the filesystem entry merely serves as  a  reference
       point  so  that  processes  can  access  the  pipe using a name in the
       filesystem.

In your case, you could do something like this:

#!/bin/bash

for file in "$@"; do
  ## CAREFUL: this will delete any files with the same name if they exist
  if [ -e "$file" ]; then
    rm -- "$file"
  fi
  mkfifo -- "$file"
done

## This is your utility
foo "$@" &

## And now read the variables
i=0
for file in "$@"; do
  file_contents[i]=$(cat < "$file")
  rm -- "$file"
  (( ++i ))
done

## And here you can use the file_contents array to do whatever you need
for (( i=0; i<${#file_contents[@]}; i++)); do
  printf "The contents of file number %d are: %s\n" "$i" "${file_contents[i]}"
done

To run foo file1 file2 file3 you would just run that script with file1 file2 file3 as its arguments:

my_script.sh file1 file2 file3