pdsh input from file possible?

Question

I can pipe data from one command to another, example:

$ echo test | cat
test

Unsure of what to call the operation I can get a similar effect using:

$ cat < <(echo test)
test

Where <(echo test) is a bashism for creating a file on the fly. Using regular files it looks like:

$ cat file
test
$ cat < file
test

This works equally well over ssh:

$ ssh server cat < <(echo test)
test

Using the ssh-example as a base one might think you could do something like:

$ pdsh -a cat < <(echo test)

But no data is sent to cat on the connected machines and the command never terminates.

tee seem quite able to send what it receives on stdin to more than one place:

$ tee >(cat) >(cat) < <(echo test) 
test
test
test

Is it possible to achieve the same over pdsh?

Answer 1

As pdsh doesn't support stdin forwarding/broadcasting, you could consider using clush from the ClusterShell project as drop-in replacement for pdsh, which has support for stdin broadcasting:

$ clush -a cat < <(echo test)
n1c01: test
n2c01: test
n1c02: test
n2c02: test

In addition, clush has nice output gathering features (no need external pipe to dshback for example), like shown here:

$ clush -ab cat < <(echo test)
---------------
n[1-2]c[01-02] (4)
---------------
test

See clush's standard input bindings.

Answer 2

You need to quote the remote command, so that the entire command runs on the remote host(s). e.g.

pdsh -a 'cat < <(echo test)'

Without the quotes, only the cat runs on the remote host. The redirection (<) and the process substitution <(echo test) run on the local host.

When there's only one remote host consuming the ouput of echo test (e.g. with the ssh command), this works...but when there are multiple remote hosts wanting to consume that output (e.g. because you used pdsh instead of ssh), it won't work.

Answer 3

I was able to get in contact with one of the pdsh developers and learned the following:

What you want is "stdin broadcast" and unfortunately support for this was never added to pdsh. It would be a nice feature, but there was not historically much need for it, so it wasn't ever done.

Which seem to confirm what has already been established in this post.

However, it was followed by:

BTW, it isn't that it is impossible to do the stdin broadcast. Parallel launchers that are part of HPC schedulers can do it, like srun(1) and the like. The mechanism is that stdin is read in once, then copied to a buffer for each remote process, i.e. the duplication is done inside of the parallel launcher.

The reason for the follow-up is that there are some misleading answers on that stackexchange post.

Another way to get around the serial for-loop problem would be to run ssh from GNU parallel or pdsh -R exec. Example with pdsh -R exec:

$ pdsh -R exec -w host[0-10] bash -c 'ssh %h cat < <(echo test)'

Of course the drawback here is that you are creating the temporary file for redirection N times. Might be better to put your output into a local file and then just cat that file to each ssh command.

The benefit of pdsh/parallel over a for loop is that you get the parallelism.

In my own tests I had some trouble getting that exact example working:

root@master# pdsh -R exec -w host1 bash -c 'ssh %h cat < <(echo test)'
host1: bash: -c: line 0: syntax error near unexpected token `<'
host1: bash: -c: line 0: `ssh n1 cat < <(echo test)'
pdsh@master: host1: bash exited with exit code 1

One small tweak gets it alive, and that is using a "normal" file:

root@master# cat data.txt
test from file
root@master# pdsh -R exec -w host1 bash -c 'ssh %h cat < data.txt'
host1: test from file

Conclusion: pdsh could grow a feature to handle what I was asking for better, but even as it is now there are ways to achieve what I asked for.

Answer 4

I assume the reason why you want the data to be sent to multiple computers is that you cannot easily put it into a temporary file and transfer the file to the remote machines. Maybe the output is generated continuously, maybe it is too big to fit on the disk.

GNU Parallel's --tee can help you:

seq 1000 |
  parallel --tee --pipe ssh {} wc ::: server1 server2

Replace seq 1000 and wc with your commands.

On an unmodified GNU/Linux system this should be able to run on 250 servers. If you need more, you will run out of file handles, and will have to change ulimit -n or nofile in /etc/security/limits.conf or /proc/sys/fs/file-max.

Behind the scenes the following is started (It is a little more complicated because GNU Parallel collects stdout and stderr from the commands but the principle is the same):

... | tee >(ssh server1 wc) >(ssh server2 wc) >/dev/null

So performance is comparable to tee. But it also means that there will be an ssh to all the servers in parallel - not just a few at a time.