scp files from multiple directories on local to multiple directories on remote in one command

Question

I need to copy some files from multiple local directories to multiple remote directories.

The command:

scp -v /file/source1/* username@host_server:/file/destination1
scp -v /file/source2/* username@host_server:/file/destination2
scp -v /file/source3/* username@host_server:/file/destination3

It asks for password again and again.

The command:

scp file1 file2 ... fileN user@host:/destination/directory/

Will put all the files into one destination directory.

But my destination are different for all the files.

Answer 1

You can't have multiple destinations in one scp command. If you want to make a single SSH connection, you'll need to use some other tool.

The simplest solution is to mount the remote filesystem over SSHFS and then use the cp command. This requires SFTP access.

mkdir host_server
sshfs username@host_server:/file host_server
cp /file/source1/* host_server/destination1
cp /file/source2/* host_server/destination2
cp /file/source3/* host_server/destination3
fusermount -u host_server
rmdir host_server

Another solution is to first organize the files locally, then copy the hierarchy. This requires rsync.

mkdir destination1 destination2 destination3
ln -s /file/source1/* destination1
ln -s /file/source2/* destination2
ln -s /file/source3/* destination3
rsync -a --copy-unsafe-links destination1 destination2 destination3 username@host_server:/file
rm -r destination1 destination2 destination3

Another solution is to keep using scp, but first open a master connection to the server. This is explained in Using an already established SSH channel

Alternatively, just bear with it and make three scp connections. But don't use your password to log in; instead, create a key pair and load the private key into your key agent (ssh-add ~/.ssh/id_rsa), then you won't have to type anything on each connection.

Answer 2

1. to avoid the repeated password prompts, use key-based authentication rather than password-based.

   e.g. run ssh-keygen on your local machine and then run ssh-copy-id user@remotehost for each remote host. That'll be the last time you need to enter the password when you ssh or scp from that machine.

   Use a good pass-phrase on your key and run ssh-agent or similar. That way you'll only have to enter your private key's pass-phrase the first time you use it in a login session.

2. Use pdcp from the pdsh package. For example:

   pdcp -w host1,host2,host3 /path/to/source/file /path/on/destination/

   or if your hostnames match a pattern:

   pdcp -w host[1-3] /path/to/source/file /path/on/destination/

   You can't have multiple destinations, but you can have multiple sources. e.g. pdcp -w h1,h2,h3 file1 file2 file3 /remote-path/ will copy to the /remote-path/ directory on each machine (any target machines where that directory is missing will output an error message)

   You can use a for loop wrapper if you need to copy multiple source files to different directories on multiple hosts. you could do the same with plain old scp but you'd need an outer loop for the hosts as well as an inner loop to do the scps. Also, pdcp copies to multiple machines in parallel, not one at a time.

   With pdsh (and pdcp) you can configure arbitrary names for groups of machines, e.g. all compute nodes are in the compute-nodes group, all web servers in the web group, database servers in db, all hosts in group all etc.

   Then you can do things like:

   pdsh -g web 'uname -a ; uptime' or pdcp -g web,db /etc/motd /etc/

Answer 3

scp uses ssh for data transfer, and uses the same authentication and provides the same security as ssh. Unlike rcp, scp will ask for passwords or passphrases if they are needed for authentication. So you can:
1. Settle ssh for connection via public key by editing remote sshd_config like

# Should we allow Identity (SSH version 1) authentication?
RSAAuthentication yes       
# Should we allow Pubkey (SSH version 2) authentication?
PubkeyAuthentication yes
# Where do we look for authorized public keys?
# If it doesn't start with a slash, then it is
# relative to the user's home directory
AuthorizedKeysFile .ssh/authorized_keys

Generate key-pairs

localhost$ ssh-keygen -t rsa

And send it to remote host

localhost$ scp ~/.ssh/id_rsa.pub [email protected]
localhost$ ssh [email protected]
remote.server.host$ [ -d ~/.ssh ] || (mkdir ~/.ssh; chmod 700 ~/.ssh)
remote.server.host$ cat ~/id_rsa.pub >> ~/.ssh/authorized_keys
remote.server.host$ chmod 600 ~/.ssh/authorized_keys

Or
2. [Unsecure] Set passwordless remote host user
Or
3. Use scripts with expect

#!/usr/bin/expect
stty -echo
send_user -- "Enter Password: "
expect_user -re "(.*)\n"
send_user "\n"
stty echo
set PASS $expect_out(1,string)
…
spawn scp -v /file/source1/* username@host_server:/file/destination1
expect "assword:"
send $PASS"\r"
expect eof

Answer 4

With a very slight modification in your command, this can be achieved as shown below.

Assuming you have source1, source2 .. folders inside your "file" folder, you can run as below to copy all these folders source1, source2 .. into the destination server.

scp -r /source/file username@host_server:/file/destination1

This will create the folders source1, source2 and copy their contents in the respective folders present in your destination path /file/destination1 on the target server.