Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 4 down vote favorite
1

Can scp be used to send a single file to multiple remote servers at the same time? If so, how? If not, what's the alternative?

share|improve this question
1  
Why do you need that "at the same time"? What is your scenario? – Nils Sep 15 '12 at 22:06
at the same time->parallely – MiNdFrEaK Sep 19 '12 at 2:33
I know what at the same time is. The question is WHY? It is one thing asking a question - but I am interested in your use-case, too. I never needed that up until now... – Nils Sep 19 '12 at 13:35
well, I needed to do some experiments, where the root site needs to send copies of the same thing to multiple site at the same time, and multiple sites do some processes then again send the result to root site. – MiNdFrEaK Sep 19 '12 at 19:44

2 Answers

up vote 10 down vote accepted

pdcp from the pdsh package is one option. pdsh was written to help with management of HPC clusters - I've used it for that, and I've also used it for management of multiple non-clustered machines.

pdsh and pdcp use genders to define hosts and groups of hosts (a "group" is any arbitrary tag you choose to assign to a host, and hosts can have as many tags as you want.)

For example, if you had a group called 'webservers' in /etc/genders that included hostA, hostB, hostC, then pdcp -g webservers myscript.sh /usr/local/bin would copy myscript.sh into /usr/local/bin/ on all three hosts.

Similarly, pdsh -g all uname -r would run uname -r on every host tagged with "all" in /etc/genders, with the output from each host prefixed with the host's name.

$ pdsh -g all uname -r
indra: 3.2.0-3-amd64
kali: 3.2.0-3-amd64
ganesh: 3.2.0-3-amd64
hanuman: 3.2.0-2-686-pae

pdsh commands and pdcp copies are executed in parallel (with limits and timeouts to prevent overloading of the originating system).

When the command being run produces multi-line output, it can get quote confusing to read. Another program in the pdsh package called dshbak can group the output by hostname for easier reading.

after seeing all your comments, it's possible that pdsh & pdcp may be overkill for your needs...it's really designed to be a system admin's tool rather than a normal non-root user's tool.

It may be that writing a simple shell script wrapper around scp may be good enough for you. e.g. here's an extremely simple, minimalist version of such a wrapper script.

#! /bin/bash 

# a better version would use a command line arg (e.g. -h) to get a
# comma-separated list of hostnames, but hard-coding it here illustrates
# the concept well enough.
HOSTS="mindfreak@asper.comgrid.ac mindfreak@looper.comgrid.ac"

# last argument is the target directory on the remote hosts
target_dir="$BASH_ARGV"

# all but the last arg are the files to copy
files=${@:1:$((${#@} - 1))}

for h in $HOSTS; do
    scp $files "$h:$target_dir"
done
share|improve this answer
Can you shed some light on how can I use it in an environment where I have 3 remote servers each in different locations, how to install and get it working, any man page? – MiNdFrEaK Sep 15 '12 at 22:55
It uses ssh. if the commands you want to run need root privs, that means you'll need to enable root ssh. it doesn't matter where the remote servers are, as long as you can ssh to them. install pdsh, genders and optionally rdist (for pdcp) onto your client machine that you'll be running pdsh/pdcp from, and on the remote servers - these are all available packaged for debian, ubuntu, and other distros. add your hosts to /etc/genders. man pages are included with the software. – Craig Sanders Sep 16 '12 at 0:02
Is there any other way round than using gender database? I have 1 root site(from where I use to send files to all the 3 client sites).from the man page I found I can use pdcp -w user@[host1,host2...hostn] command, please correct me if its wrong, also here how can I mention the destination hosts path is like pdcp -w user@[host1, host2, host3] /home/user/Test_Data(let this is the common destination path for each hosts). – MiNdFrEaK Sep 17 '12 at 7:04
yes, you can specify hostnames directly with -w, even without an /etc/genders file. your 'root site' will still need to have your three client hosts in ~/.ssh/known_hosts, and your 3 remote hosts will need to be configured to allow password-less public_key access from your root site. using ssh-copy-id should achieve both requirements. i'm puzzled by what your second question could mean because the obvious interpretation is answered both in my answer above and in your question itself...as with cp, the destination is the final argument. – Craig Sanders Sep 17 '12 at 7:20
I asked that because I was puzzled too from the man page as it was written that the path cann't be specified the way we specify it in scp,but with that they meant the use of :and then file path name, I didn't get it then,now its clear. – MiNdFrEaK Sep 17 '12 at 7:58
show 10 more comments
up vote 1 down vote

Instead of sending a file to multiple targets at the same time you could read it multiple times at the same time.

Transfer your file to NFS, mount that filesystem on your targets and copy it from NFS to your local destination.

To do so manually and concurrent, you could use cluster-ssh (cssh) on the targets.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.