I would like to read in a number of files and pipe their output to subsequent programs while still maintaining them as individual pipelines of data.

program1 *.txt | program2 | program3 folder

I know what the above syntax can accomplish for single streams of data, but I am looking at keeping the files separate throughout the entire operation. The above would translate to the following:

  1. program1 reads text files and pipes to program2
  2. program2 processes data individually and pipes to program3
  3. program3 writes data to files in folder with the same file names as the original

This kind of operation is currently the domain for build tools like Gulp, but I am trying to see if a shell can fully replace them. Since programs are written to handle only one stdin, it doesn't seem feasible.

The reading and writing of multiple files isn't an issue as I will just handle that within the programs themselves.

I have looked into the following, but they don't seem like the correct solution:

  • the tee command
  • file descriptors
  • substitutions

One possible way is to create a process for each individual file, and maintain a list of file names somewhere, but I am hoping for something more elegant.

  • Do you hear about loop? for file in *.txt ; do mkdir -p /path_to/"$file" ; program1 "$file" | program2 | program3 /path_to/"$file" ; done
    – Costas
    Jul 26 '15 at 5:36

A pipe, just like any file, is a stream of text (more precisely, a stream of bytes). The basic building blocks of Unix tend to be simple. Interactions between processes are mostly based on unstructured data. The operating system doesn't provide a communication channel with multiple streams labeled by a file name. If programs need this, they need to arrange their own — and separate pipes, one for each stream, would be the most natural implementation.

If program2 and program3 act independently on each stream, run one copy of them for each of the files. To run them sequentially, use a shell loop. Like the pipe, the loop is one of the shell's features to tie programs together. To tell program3 where to put the output, the usual interface is for program3 to write to its standard output, and use a redirection construct in the shell to direct the output to a file. The shell provides some basic string manipulation constructs to build file names; here it's just concatenation.

for x in *.txt; do
  program1 "$x" | program2 | program3 >"folder/$x"

If the programs are light on IO but CPU-intensive and you have multiple CPUs, you may want to run them in parallel. With recent enough GNU tools, you can use xargs to run programs in parallel. Pass the number of CPU on your system as the argument to -P. Since the command that xargs needs to execute is a pipeline, you need to make it invoke a shell.

find -maxdepth 1 -name '*.txt' -print0 |
xargs -0 -n 1 -P 4 sh -c 'program1 "$1" | program2 | program3 >"$0/$1"' "folder"

You can use GNU parallel instead of xargs to let it determine the number of CPUs on your system automatically.

parallel sh -c 'program1 "$1" | program2 | program3 >"$0/$1"' "folder" ::: *.txt

If you need a single instance of program2 and program3 to process multiple files, you'll need to design these programs with a custom interface to receive multiple pipes as input. There's no standard way to do that. One method is to let them invoke the program that provides their input. It would be similar to the way xargs and parallel are told what program to invoke to process their output.


Are you talking about

program1 file1.txt   | program2 | program3 > folder/file1.txt
program1 file2.txt   | program2 | program3 > folder/file2.txt
program1 file42.txt  | program2 | program3 > folder/file42.txt
program1 green.txt   | program2 | program3 > folder/green.txt
program1 indigo.txt  | program2 | program3 > folder/indigo.txt
program1 leopard.txt | program2 | program3 > folder/leopard.txt
program1 lion.txt    | program2 | program3 > folder/lion.txt
   ⋮        ⋮            ⋮          ⋮                 ⋮ 


You can do that with

for f in file1.txt file2.txt file42.txt green.txt indigo.txt leopard.txt lion.txt ...
    program1 "$f" | program2 | program3 > folder/"$f"

If you want to do this to all the text files in the current directory, just use the wildcard (a.k.a. “glob”):

for f in *.txt
    program1 "$f" | program2 | program3 > folder/"$f"

This site is temporarily in read only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .