Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I never really used tarballs/archived much, except for uploads (with compression). Now I've accumulated a huge list of software projects coding experiments -- basically directories with lots and lots of little files (mainly source-code files and git objects), and it seems those seem to slow things down when I'm backing up my home or syncing with another device (I mostly sync over an USB cable, with rsync).

I wonder, is this a documented phenomenon (benchmarks?) and would tarballing long untouched project directories speed things up? Would that be an advisable thing to do?

I'm using ext4 filesystems.

share|improve this question
One option is to reduce the frequency of backups for dormant projects, or omits new backups entirely. I'm not sure why tarballing would help. A perhaps better option is to use distributed version control for text-only projects, and then pushing commits to remote makes other backups more or less redundant. This is what I do. Of course, you still have to think about local changes and uncommitted files, so if you want to be super-careful you can back them up as well. – Faheem Mitha Mar 6 '15 at 8:34
up vote 1 down vote accepted

Archiving old directories that you rarely access as tarballs can definitely improve the performance of a file-based backup system.

I wonder, is this a documented phenomenon (benchmarks?)

It's not really a "documented phenomenom" so much as a natural consequence of having to scan the filesystem and examine each file one by one to determine whether it needs to be backed up.

You could reduce the frequency of backups as Faheem Mitha suggests, but you may find it troublesome to maintain multiple backups at different frequencies (for oft-updated stuff and old archived stuff) or to maintain file exclusions lists and such. If you really don't plan to be needing access to these directories anytime soon, I think it's a perfectly fine idea to tar them up. I've done this many times for exactly the same reason.

share|improve this answer

Rsync needs to check all those files and folders each time. That takes up time, performance and network load. If you put each project in a tarball that means one file check instead of thousands checks. It saves space as well.

share|improve this answer

I ran a small benchmark on a directory of cloned repos -- lots of little files.

Here are parameters:

17002 files
46 root directories 
tar command: tar cf (no compression)
rsync command: rsync -aH --delete --stats 

And the results:

Local rsync to an empty directory (unpacked files):

real    5m36.447s
user    0m34.692s
sys     0m56.390s

Second local rsync (unpacked files):
real    0m6.810s
user    0m2.257s
sys     0m3.363s

Tarring time:

real    1m14.648s
user    0m14.278s
sys     0m2.175s

Local rsync to an empty directory (unpacked files):

real    2m6.355s
user    0m20.799s
sys     0m21.122s

Local rsync to an empty directory (packed files):

real    0m0.125s
user    0m0.005s
sys     0m0.011s

So it seems that tarring quite significantly improves performance. What's quite surprising is that the tarring + the second local rsync take together less time than the first local rsync.

Tarring also quite significantly increases the speed of no-op rsync runs.

I also tried to do tarring with compression. Tarring with gzip took about 10 minutes, lzop didn't do much better (I stopped it at about 7 mins). According to the nice graph at http://www.linuxjournal.com/article/8051?page=0,2 , compression won't improve my bandwidth anyway if the slowest link I'm going to be using is a USB cable (approx. 20MBps).

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.