I'm planning to backup the /home folder of my Kubuntu box to an external drive.

In order to speed up the process I consider zipping large files (individually) so that writing them is faster. I also want to do the zipping/copying in multiple threads so that the resources of the box are better used. I thought of something like this:

find /home/ -exec sh -c 'bzip2 "$1" "/path/to/target/$1.bz2"&' find-sh {} \;

On the other hand I want to avoid to spawn thousands of processes, what this script most likely will do.

What is the best way to combine find and bzip2 so that the latter runs in parallel with a limited number of threads?

  • Consider the use of rsync and parallel together. – steve Nov 12 '16 at 19:16
  • @steve do you mean something like this: find /home/ -exec sh -c 'parallel rsync --compress ... "$1" /path/to/target/$1.gz' find-sh {} \;? – Timothy Truckle Nov 12 '16 at 20:11
  • I was thinking of the answers shown at unix.stackexchange.com/questions/189878/… – steve Nov 12 '16 at 21:04
  • 1
    xz has a better compression/time tradeoff than bzip2. Consider zipping files together, you'll get better compression. And I strongly recommend using a robust tool for backups rather than cobbling up your own. Duplicity is pretty simple to use and does compressed incremental backups, optionally encrypted. – Gilles Nov 13 '16 at 22:04
  • @Gilles does Duplicity also use more than one core of my cpu? and can I backut to alocal folder like /mnt/removableDisk/backup? – Timothy Truckle Nov 13 '16 at 22:23

I finally ended up with starting a few tar jobs by hand...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.