I would like to read in a number of files and pipe their output to subsequent programs while still maintaining them as individual pipelines of data.
program1 *.txt | program2 | program3 folder
I know what the above syntax can accomplish for single streams of data, but I am looking at keeping the files separate throughout the entire operation. The above would translate to the following:
- program1 reads text files and pipes to program2
- program2 processes data individually and pipes to program3
- program3 writes data to files in folder with the same file names as the original
This kind of operation is currently the domain for build tools like Gulp, but I am trying to see if a shell can fully replace them. Since programs are written to handle only one stdin
, it doesn't seem feasible.
The reading and writing of multiple files isn't an issue as I will just handle that within the programs themselves.
I have looked into the following, but they don't seem like the correct solution:
- the
tee
command - file descriptors
- substitutions
One possible way is to create a process for each individual file, and maintain a list of file names somewhere, but I am hoping for something more elegant.
loop
?for file in *.txt ; do mkdir -p /path_to/"$file" ; program1 "$file" | program2 | program3 /path_to/"$file" ; done