0

To be clear, I am running this exact command(variables changed) in a gnu parallel setting:

/usr/bin/parallel -q -j20 
    rsync --recursive -av -e ssh -oCompression=no -x -T 
    -oUserKnownHostsFile=/dev/null -oConnectTimeout=15 
    -i ssh_key -l some_username --chmod=Dugo+rwx,Fugo+rw 
    --files-from {} /some/base/dir somehostame:/some/other/dir

This consumes about 50 files(containing list of file/directory names for rsync), across 20 workers through gnu parallel. I ran the above yesterday night, and I came back to work to see that there is still 1 rsync job, hanging:

rsync --recursive -av -e ssh -oCompression=no -x -T -
    oUserKnownHostsFile=/dev/null -oConnectTimeout=15 
    -i ssh-beta-secdata-20180515t213639 -l diskmaker 
    --chmod=Dugo+rwx,Fugo+rw 
    --files-from ./some_filelist /some/base/dir somehostname:/some/other/dir

So to be clear, some_filelist is a list of directories I want to do rsync in.

I'll take the files from the base(as if chroot'd) from /some/base/dir to a remote directory somehostname:/some/other/dir.

So there are like, 500 jobs that parallel pulls from. I analyzed each file list, and there's nothing special about this ./some_filelist; standard 200 files from an rsync dryrun.

Does anyone have diagnosis for this problem? I am completely lost to how this could happen. I can post more snippets if required.

4
  • can you list what files the hanging process has open (lsof -p pid), and what syscall it is hanging on (strace -p pid).
    – meuh
    May 16 '18 at 19:38
  • Does it work, if you run the hanging command on the command line? Can you try splitting ./some_filelist and see if some parts work find? Is the the same part that is hanging if you use parallel --shuf?
    – Ole Tange
    May 17 '18 at 11:49
  • Hope you make sure different rsync processes do not read/write to the same dirs. May 17 '18 at 21:45
  • It looks as if rsync would have some issues sorting out what options are meant for ssh and what options are not meant for ssh (since the option-argument to rsync -e is not a single string).
    – Kusalananda
    Oct 14 '19 at 9:26
0

As Ole Tange suggested, you may see that rsync asks you for the ssh password, when you try rsyncing with only one argument(a directory name in your case). When I encountered the same problem in my experiments, I figured out that parallel doesn't prompt you for the ssh password, but in the background it needs one.

Applying one of the answers on this SuperUser question solved my problem. It says in particular;

1) Create a public key;

ssh-keygen

2) Copy that key to "somehostname"

ssh-copy-id -i ~/.ssh/id_rsa.pub user@somehostname OR somehostname's IP address

That's all. After these steps, ssh connections between those two machines doesn't require a ssh password to be provided. In my case, the command stopped hanging using this configuration.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.