1

I'm using gnu parallel to generate backups for about ~1500 web sites on pantheon.io, using their terminus CLI. The terminus backup:create command does not finish until a response is received that it has completed on the remote end. I'm wondering if there is any way to better speed this up with parallel so that more sites can be backing up while waiting on previous ones to complete or just run more overall if not. If it makes any difference, this is being run from a Jenkins CI job. Thank you.

#!/bin/bash +x

backup_sites() {
  BACKUP=$(terminus backup:create "$*".live)
  echo "$*": "$BACKUP"
}

SITE_LIST=$(terminus site:list --field=name)

export -f backup_sites
echo "$SITE_LIST" | parallel backup_sites
3
  • Consider switching to ZFS. Then your local system's snapshot is atomic, it can be transfered to your backup machine at maximum wire speed of your disk and network, and the creation of the snapshot on the backup machine is also atomic.
    – Jim L.
    Apr 23, 2021 at 20:48
  • I'm calling this command to run on a remote SaaS provider so I don't have control over this.
    – Vincent
    Apr 23, 2021 at 22:45
  • Assuming that you're the one paying the bill, you do have control. You tell your provider you require ZFS, or you're going to switch to a different provider. :)
    – Jim L.
    Apr 24, 2021 at 6:08

1 Answer 1

Reset to default
1

Not specifying how many jobs to run in parallel will make it default to number of cpus.

From the manual:

-j

Number of jobslots on each machine. Run up to N jobs in parallel. 0 means as many as possible. Default is 100% which will run one job per CPU on each machine.

In general this is a safe bet, but you are waiting on network, not computation. So you can easily boost up the number. I would try -j 200. It should work quite well. You can tweak this parameter to get the speed you need.

So echo "$SITE_LIST" | parallel -j 200 backup_sites instead of echo "$SITE_LIST" | parallel backup_sites

2
  • Thank you, I will try this out. Is there a way to determine how many jobs the machine can handle at once, depending on what you are trying to run?
    – Vincent
    Apr 25, 2021 at 19:47
  • Typically you will get no more benefit after a certain upper bound, as either your network chokes or RAM maxes out. This depends entirely on your host machine and network. The best way to do it is to keep increasing jobs by 100 till you see no more speed up. You can test it on small subset of your target to get measurements.
    – Capstone
    Apr 25, 2021 at 21:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .