3

I want to scp files from machineA into my machineB and this is how I am doing it. I am copying bunch of files one by one from primary folder of machineA to primary folder of machineB and secondary folder of machineA to secondary folder of machineB.

trinity@machineB:~$ scp trinity@machineA:/data01/primary/* /data01/primary/
trinity@machineB:~$ scp trinity@machineA:/data02/secondary/* /data02/secondary/

Is there any way by which I can copy multiple files in parallel? Like five files at a time from a folder? So instead of copying one files at a time, I want to copy five files from primary or secondary folders respectively?

Basically I want to copy whatever is there in primary and secondary folders of machineA into machineB parallely.

I also have GNU Parallel installed on my box if I can use that. I tried below command but it doesn't work. I was expecting that it should copy 5 files in parallel at a time until everything gets copied from that folder.

parallel -j 5 scp trinity@machineA:/data01/primary/* /data01/primary/

Anything wrong with my parallel syntax? What is the best way by which I can copy five files in parallel from a remote folder until everything gets copied from it?

  • Do the files have specific names? No shame in creating a ssh key, using a key agent to keep it unlocked, and just calling scp multiple times sending each process to the background. Using some other method like rsync may be better too, since it will only copy what is needed – ivanivan May 25 '17 at 1:13
  • Yeah I can write a script for that but I was thinking if there is any other way instead of doing that. I just need to copy everything whatever is there in that remote folders. – david May 25 '17 at 1:42
  • Create a symlink of secondary in primary on both machineA and machineB. Then do the scp of primary, all files of primary along with secondary would be copied. – sai sasanka May 25 '17 at 6:09
1

Why do you think the parallel connection would make it faster? SCP is very simple tool to transfer few small files, nothing for throughput and performance. Running it from more parallel processes could make it little bit faster, but not significantly. What you can do on the other way is:

  • Use sftp to get better throughput and more clever copying (should be enough) for example using -r
  • Use ControlMaster to get rid of the overhead of parallel TCP connections
  • Use correct parameters to the parallel

I would start with the sftp:

sftp -r trinity@machineA:/data01/primary/ /data01/primary/
1

You need the * expansion to happen on the remote side:

ssh machineA 'parallel -j 5 scp {} machineB:/data01/primary/ ::: /data01/primary/*'
  • When I run above command on machineB, I am see this error message on the console - Host key verification failed. lost connection multilple times and eventually nothing gets copied. Any thoughts why? I can ssh perfectly fine from machineB to machineA. – david Jun 1 '17 at 20:08
  • Are you running as the same user? Because it sounds as if you have not accepted the host key. It might help to do scp -o StrictHostKeyChecking=no .... – Ole Tange Jun 2 '17 at 6:53
  • I tried and I am still getting this error Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,password). lost connection. And I can ssh just fine from machineB to machineA. – david Jun 2 '17 at 18:16
  • Do you have to enter a passphrase when you ssh? In that case, please use ssh-agent, so you do not need to enter a passphrase. GNU Parallel assume you do not need to enter a passphrase. – Ole Tange Jun 2 '17 at 23:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.