I have a bash script that kicks off multiple rsync processes in parallel. I'd like to limit the number of simultaneous child processes but not serialize the whole script.

Here's a simplified version of the current script:

  flock -n 200 || exit 0
    for f in $(ssh $1 ls /var/images/)
      rsync … &
  } &> /var/log/my.log
) 200>/var/lock/my.lockfile

I believe xargs with --max-procs might be a solution but I can't figure out how to integrate that with my script. Can it be done like this, or is there a better way?

  • 1
    Why don't you let rsync do its job? You can use its recursive option...
    – Spack
    Commented Sep 30, 2013 at 9:07
  • @Spack because it's not parallel? I don't need the recursive option, the 30-odd files I'm rsync-ing are all in the same directory, and they range in size from 10GB to 800Gb.
    – user12810
    Commented Sep 30, 2013 at 9:16

1 Answer 1

ssh "$1" "find /var/images -type f -print0" |
  xargs --null --replace --max-procs=X rsync "${1}:{}" /my/destination

Should do the trick.

  • thanks, what does "host:{}" mean btw?
    – user12810
    Commented Sep 30, 2013 at 9:39
  • 1
    using xargs, the replace option will replace any {} by the actual argument, here the path to your file. Since you are using rsync, you should provide the host to connect to. In your script, you can replace host by $1.
    – Spack
    Commented Sep 30, 2013 at 9:59
  • man xargs says: "Use the -n option with -P; otherwise chances are that only one exec will be done". Should I be using -n1 here?
    – user12810
    Commented Oct 6, 2013 at 23:07
  • @JackDouglas if you use the replace option,-i , the -n is useless.
    – Spack
    Commented Oct 8, 2013 at 10:18

You must log in to answer this question.