scp does not display output when used with gnu parallel

Question

I have a short script that uses scp to copy files to a number of remote hosts (yes, I know about rdist and rsync; they both fail to work for a few of the hosts - that's not the point here; I'm only copying a few non-critical files anyway).

The meat of the script looks something like this:

for h in $HOSTS; do
    echo $h
    echo '----------------------------------------'
    scp -r $FILES ${h}:
    echo ''
done

Here is partial output from running this script:

protector
----------------------------------------
.bash_profile                                                                                                                100%  555     0.5KB/s   00:00    
.bashrc                                                                                                                      100% 2124     2.1KB/s   00:00    
.zshenv                                                                                                                      100%  561     0.6KB/s   00:00    
.zshrc                                                                                                                       100% 2354     2.3KB/s   00:00    
.shrc                                                                                                                        100% 1887     1.8KB/s   00:00    
.bash_logout                                                                                                                 100%   17     0.0KB/s   00:00    
.logout                                                                                                                      100%   64     0.1KB/s   00:00    
.zlogout                                                                                                                     100%   17     0.0KB/s   00:00    
.vimrc                                                                                                                       100%  717     0.7KB/s   00:00    

pup
----------------------------------------
.bash_profile                                                                                                                100%  555     0.5KB/s   00:00    
.bashrc                                                                                                                      100% 2124     2.1KB/s   00:00    
.zshenv                                                                                                                      100%  561     0.6KB/s   00:00    
.zshrc                                                                                                                       100% 2354     2.3KB/s   00:00    
.shrc                                                                                                                        100% 1887     1.8KB/s   00:00    
.bash_logout                                                                                                                 100%   17     0.0KB/s   00:00    
.logout                                                                                                                      100%   64     0.1KB/s   00:00    
.zlogout                                                                                                                     100%   17     0.0KB/s   00:00      
.vimrc                                                                                                                       100%  717     0.7KB/s   00:00    

Since this script copies to hundreds of hosts, it takes a little while, so I decided to try to use GNU Parallel to speed it up. Here is the revised version of the script utilizing parallel (please don't comment on the echo hack at the beginning, it's not relevant to the problem):

(for host in $(echo $HOSTS); do echo $host; done) | parallel "echo {}; echo '----------------------------------------' ; scp -r $FILES {}: ; echo ''"

The problem is that the output from using scp in this way looks like this:

pup
----------------------------------------

soils
----------------------------------------

Because the commands are running in parallel, the ordering is different, and parallel seems to complete much faster than a for loop of scps, but as you can see, there is no output to stdout from scp. I can't figure out why - output to stdout from echo is not being suppressed, for instance. When I try scp -v inside parallel, I get partial verbose output, but I don't get any of the usual size/time information.

Is there anybody who knows why scp output is getting suppressed?

Answer 1

To trick scp into thinking it has a tty connected you can use script:

script -c 'scp foo server1:/bar/'

So to run that in with parallel you need something like:

parallel -j30 "echo {}; echo '----------------------------------------' ; script -c 'scp -r $FILES {}:' ; echo ''"   ::: $HOSTS

To learn more check out the intro videos (https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1), the EXAMPLEs (n1_argument_appending">http://www.gnu.org/software/parallel/man.html#example__working_as_xargs_n1_argument_appending), and the tutorial (http://www.gnu.org/software/parallel/parallel_tutorial.html).

Answer 2

This seems to work for me:

#!/bin/sh

for h in $(cat /tmp/servers);
do
    for f in $(cat /tmp/files);
    do
        echo $h
        echo '----------------------'
        parallel -vv "scp root@$h:$f /tmp"
    done
done

Take the -vv out to remove the verbose output. And of course modify it so that A) it uses your $HOSTS and $FILES variables, and B) it copies to the remote server rather than from.