6

I want to tar four directories which contain large number of small files using a shell script. Because of this script takes too long to execute so I want to make these 4 tar commands run in parallel using shell script hoping I can make better use of resources available.

Commands that I am currently using:

tar cf - /ebs/uat/uatappl | gzip -c > /ebs/backup/uatappl.tar.gz
tar cf - /ebs/uat/uatcomn | gzip -c > /ebs/backup/uatcomn.tar.gz
tar cf - /ebs/uat/uatora | gzip -c > /ebs/backup/uatora.tar.gz
tar cf - /ebs/uat/uatdata | gzip -c > /ebs/backup/uatdata.tar.gz
  • What hardware are you running this on? Some Solaris servers, such as the older SPARC T* series, have pretty poor single-threaded CPU performance - they were designed to run many threads where each thread didn't have much performance requirements - such as a web server serving lots of users. On that hardware, your bottleneck could be gzip. – Andrew Henle May 4 '17 at 10:37
7

You can put all the tars in background like this:

tar cf - /ebs/uat/uatappl | gzip -c > /ebs/backup/uatappl.tar.gz &
tar cf - /ebs/uat/uatcomn | gzip -c > /ebs/backup/uatcomn.tar.gz &
tar cf - /ebs/uat/uatora | gzip -c > /ebs/backup/uatora.tar.gz &
tar cf - /ebs/uat/uatdata | gzip -c > /ebs/backup/uatdata.tar.gz &

But be aware you must have enough processor power and fast disk, otherwise the concurrency will make total execution longer than consecutive one

  • 1
    And even a fast disk will slow down some under such concurrent load - it'll still be "fast", but it will be slower than it would be otherwise. For such concurrent I/O to scale when done in parallel, the disk entire system needs to be designed to support such concurrent loads - both software and hardware. Not many file systems are. And using a single physical disk as the source of the reads or target of the writes (or both) won't scale no matter what the file system is. – Andrew Henle May 3 '17 at 9:49
  • 3
    @AndrewHenle Usually it's simple to see where the bottleneck is. As long as all the gzip processes are able to consume 100% CPU usage, then the storage is fast enough. On the other hand if the storage is too slow then you could use a better compression (gzip -9 or xz) for free, without needing more absolute time for the whole job. – rudimeier May 3 '17 at 11:03
  • 2
    +1 for it would be slower than consecutive ones. Such is often the case for consumer hardware grade; contemporary server grade hardware may be another story. – Rui F Ribeiro May 3 '17 at 13:49
  • Since you added a Solaris tag, which version are you using? As well as what filesystem(s) are you using? – sleepyweasel May 3 '17 at 21:24
4

what's wrong with adding a & at the end of command line ?

tar cf - /ebs/uat/uatappl | gzip -c > /ebs/backup/uatappl.tar.gz &
tar cf - /ebs/uat/uatcomn | gzip -c > /ebs/backup/uatcomn.tar.gz &
tar cf - /ebs/uat/uatora | gzip -c > /ebs/backup/uatora.tar.gz &
tar cf - /ebs/uat/uatdata | gzip -c > /ebs/backup/uatdata.tar.gz &

wait

wait will wait for job to finish.

  • If one of the tar commands fail for some reason, will the exit status after wait be 0 or some other value? This is important because the script should whether tar commands executed successfully – Girish Sunkara May 3 '17 at 8:01
  • wait will return 0 in ubuntu and solaris as well, you'll need extra coding. – Archemar May 3 '17 at 8:10
  • writing a wrapper script may do, so it can write a log file for each job or print output to console or whatever you want to do. call the wrapper, passing teh source and destination as arguments, and put that entire process and children in the background. after the 4 calls to the wrapper script, put your wait as Archemar suggests in this answer – ivanivan May 3 '17 at 22:03
3

You can use GNU parallel, which is a tool for executing jobs in parallel. I've tested the below, and I believe this one-liner would work:

ls -1 /ebs/uat | parallel 'tar cf - /ebs/uat/{} | gzip -c > /ebs/backup/{}.tar.gz'

The webpage is here: https://www.gnu.org/software/parallel/, and a quick google appears to suggest there are solaris packages available, so you may not even have to manually install it.

  • 2
    Building is extremely easy: It is a single filed perl-script only dependent on Perl 5.8. – Ole Tange May 3 '17 at 22:37
  • 1
    thanks for the new tool. It looks interesting and I'm going to check it out to see it can replace some custom scripts as well as using xarg. As a fyi, I found an older release in the Solaris 11 repo: Name: shell/parallel Version: 0.2012.11.22 Build Release: 5.11 Packaging Date: August 21, 2015 05:24:17 PM – sleepyweasel May 4 '17 at 18:02
1

Since you added a Solaris tag, which version are you using? As well as what filesystem(s) are you using? If UFS, can you move to ZFS which handles lots of smaller files better than UFS. It may have been Roche who benchmarked this.

ie: if you're on ZFS, you could create a dataset for each main dir and replicate it to a different filesystem or system where a higher level of compression is set. Including possibly dedup.

Based on the dir structure, it appears all the files live on the same FS that you're backing up to. So, a parallel option might not help any as I suspect IO will be the limiting factor, not CPU resources.

You could try throwing the jobs in the background as @Romeo Ninov suggests. Or even create a script for each directory and run them at the same time. In the end, only by playing and testing various options will you know for sure what may or may not help to find what best meets your needs. Including different archivers and compression algorithms.

For instance if speed is more important try a lower level of compression in gzip or use compress or zip.

Also, if you don't think CPU is an issue, and have multiple, you could try using a pigz (parallel implementation of gzip) to see if that helps. That pkg is in the support repo for v11, but you'd have to obtain the source and compile it for earlier versions of Solaris.

Reco also a timex in front of each command, as it might only be one of the dirs. giving you the slowness. If so, concentrate on optimizing that one.

  • 1
    What hardware, too. See my comment on the question itself. – Andrew Henle May 4 '17 at 10:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.