Do any shells generalize the notion of pipes to multiple parallel pipelines? Would supporting it require changes to the OS kernel?

Question

Quite often, when working on the command line, I find myself specifying the same operation on a bunch of different instances specified by some input stream, and then wanting to recombine their outputs in some specific way.

Two use cases from yesterday:

1. I wanted to see the Subjects of a bunch of SSL certificates bundled in a PEM file.

    cat mypemfile.crt |
    (openssl x509 -noout -text | fgrep Subject:)
   

   only shows me the first one. I need to split out the certificates and run the same command on each of them, then concatenate the results. csplit can split them, but to files only. This is a hassle.

   Why can't I just say

    cat mypemfile.crt |
    csplit-tee '/BEGIN/ .. /END/' |
    on-each ( openssl x509 -noout -text | fgrep Subject: ) |
    merge --cat
   

   ?

2. We run a JupyterHub instance that splits off notebook servers as Docker containers. I want to watch their timestamped logs. It's easy enough for one container:

    sudo docker logs -t -f $container_id
   

   (The -t adds a timestamp, and the -f keeps the pipe open, like tail -f.)

   It's easy enough to list the logs of all containers, sorted by timestamps:

    sudo docker ps | awk '{print $1}' |
    while read container_id
    do
      sudo docker logs -t $container_id
    done |
    sort
   

   or

    sudo docker ps | awk '{print $1}' |
    xargs -n1 sudo docker logs -t |
    sort
   

   or

    sudo docker ps | awk '{print $1}' |
    parallel sudo docker logs -t {} |
    sort
   

   but none of that will let me use the -f option to watch the logs as they come in.

   Why can't I just use

    sudo docker ps | awk '{print $1}' |
    csplit-tee /./ |
    on-each (xargs echo | sudo docker logs -t -f) |
    merge --line-by-line --lexicographically
   

   or

    sudo docker ps | awk '{print $1}' |
    parallel --multipipe sudo docker logs -t -f {} |
    merge --line-by-line --lexicographically
   

   ?

Clearly, this requires

1. Specific shell support. Maybe we'd need a distinct "multi-pipe" symbol.
2. New tools (csplit-tee, on-each and merge) that split and merge pipelines.
3. A fixed convention for how to specify, within tools, arbitrarily many input and output file descriptors such that this shell will treat them as parallel pipelines.

Has this been done? Or something equivalent that I can apply to my use cases?

Is it feasible without specific kernel support? I know the kernel typically has a fixed maximum number of open file descriptors, but an implementation can work around that by not blindly trying to open them all at once.

Is it feasible, knowing that GNU parallel is feasible?

Answer 1

With GNU Parallel:

cat mypemfile.crt |
  parallel --pipe -N1 --recstart '-----BEGIN' 'openssl x509 -noout -text | fgrep Subject:'

Untested:

sudo docker ps | awk '{print $1}' |
  sudo parallel -j0 --lb docker logs -t -f {}

sudo docker ps | awk '{print $1}' |
  sudo parallel -j0 --tag --lb docker logs -t -f {}

Answer 2

No the shell can't do that. A pipe is just a stream from a source to a destination, usually in different processes.

Let's look at your first example:

cat mypemfile.crt |
(openssl x509 -noout -text | fgrep Subject:)

You want the input file to be split along lines containing BEGIN and END, but cat doesn't care about the content, and pipes don't have out of bound indication of record separators anyway.

You could achieve this with specific conventions in the source and destination of a pipe, but it would remove the main advantage of a pipe, to be able to combine programs as building blocks for tasks that were not anticipated by their authors.

In this specific example, it would be much easier to modify openssl to process multiple certificates in one stream than it would be to modify openssl to process multiple stream AND modify cat to provide these multiple streams.