Quite often, when working on the command line, I find myself specifying the same operation on a bunch of different instances specified by some input stream, and then wanting to recombine their outputs in some specific way.
Two use cases from yesterday:
I wanted to see the Subjects of a bunch of SSL certificates bundled in a PEM file.
cat mypemfile.crt | (openssl x509 -noout -text | fgrep Subject:)
only shows me the first one. I need to split out the certificates and run the same command on each of them, then concatenate the results.
csplit
can split them, but to files only. This is a hassle.Why can't I just say
cat mypemfile.crt | csplit-tee '/BEGIN/ .. /END/' | on-each ( openssl x509 -noout -text | fgrep Subject: ) | merge --cat
?
We run a JupyterHub instance that splits off notebook servers as Docker containers. I want to watch their timestamped logs. It's easy enough for one container:
sudo docker logs -t -f $container_id
(The
-t
adds a timestamp, and the-f
keeps the pipe open, liketail -f
.)It's easy enough to list the logs of all containers, sorted by timestamps:
sudo docker ps | awk '{print $1}' | while read container_id do sudo docker logs -t $container_id done | sort
or
sudo docker ps | awk '{print $1}' | xargs -n1 sudo docker logs -t | sort
or
sudo docker ps | awk '{print $1}' | parallel sudo docker logs -t {} | sort
but none of that will let me use the
-f
option to watch the logs as they come in.Why can't I just use
sudo docker ps | awk '{print $1}' | csplit-tee /./ | on-each (xargs echo | sudo docker logs -t -f) | merge --line-by-line --lexicographically
or
sudo docker ps | awk '{print $1}' | parallel --multipipe sudo docker logs -t -f {} | merge --line-by-line --lexicographically
?
Clearly, this requires
- Specific shell support. Maybe we'd need a distinct "multi-pipe" symbol.
- New tools (
csplit-tee
,on-each
andmerge
) that split and merge pipelines. - A fixed convention for how to specify, within tools, arbitrarily many input and output file descriptors such that this shell will treat them as parallel pipelines.
Has this been done? Or something equivalent that I can apply to my use cases?
Is it feasible without specific kernel support? I know the kernel typically has a fixed maximum number of open file descriptors, but an implementation can work around that by not blindly trying to open them all at once.
Is it feasible, knowing that GNU parallel is feasible?
csplit-tee
,oneach
andmerge
are supposed to do, but youropenssl
example seems to be doingawk -v cmd="openssl x509 -noout -text | fgrep Subject:" '/BEGIN/ { p = 1 } p { print | cmd } /END/{close (cmd)}' mypemfile.crt
stdout
(handle 1) and other programs read fromstdin
(handle 0). A pipe connectsstdout
from process tostdin
by a different one. Handling that is usually done by the shell (parent process). BTWlsof
can help visualise this.tee
outputting to temporary files, and after all the parallel writers have finished, merge the temporary files. For the examples you're giving, that's more than enough.