Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Currently I'm running tar czf to combine backup files. The files are in a specific directory.

But the number of files is growing. Using tzr czf takes too much time (more than 20 minutes and counting).

I need to combine the files more quickly and in a scalable fashion.

I've found genisoimage, readom and mkisofs. But I don't know which is fastest and what the limitations are for each of them.

share|improve this question
    
I doubt that tar introduces any significant overhead, reading the files is the expensive operation here. You should either modify the way your files are stored, or use a radically different approach (copy the filesystem as a whole). We can't help you much without knowing how your files are organized. –  Gilles Aug 21 '11 at 18:12
2  
Mount your FS with the "noatime" option, maybe speedup the IO operations. –  J.F. Aug 21 '11 at 20:36
    
+1 for noatime, it really does make a significant difference. Especially for regular hard drives, and also just for reducing extraneous writes. –  TechZilla Nov 5 '13 at 0:05

2 Answers 2

up vote 10 down vote accepted

You should check if most of your time are being spent on CPU or in I/O. Either way, there are ways to improve it:

A: don't compress

You didn't mention "compression" in your list of requirements so try dropping the "z" from your arguments list: tar cf. This might be speed up things a bit.

There are other techniques to speed-up the process, like using "-N " to skip files you already backed up before.

B: backup the whole partition with dd

Alternatively, if you're backing up an entire partition, take a copy of the whole disk image instead. This would save processing and a lot of disk head seek time. tar and any other program working at a higher level have a overhead of having to read and process directory entries and inodes to find where the file content is and to do more head disk seeks, reading each file from a different place from the disk.

To backup the underlying data much faster, use:

dd bs=16M if=/dev/sda1 of=/another/filesystem

(This assumes you're not using RAID, which may change things a bit)

share|improve this answer
    
don't compress: or use pigz if exist in the system more than one processor. –  J.F. Aug 21 '11 at 20:39

To repeat what others have said: we need to know more about the files that are being backed up. I'll go with some assumptions here.

Append to the tar file

If files are only being added to the directories (that is, no file is being deleted), make sure you are appending to the existing tar file rather than re-creating it every time. You can do this by specifying the existing archive filename in your tar command instead of a new one (or deleting the old one).

Write to a different disk

Reading from the same disk you are writing to may be killing performance. Try writing to a different disk to spread the I/O load. If the archive file needs to be on the same disk as the original files, move it afterwards.

Don't compress

Just repeating what @Yves said. If your backup files are already compressed, there's not much need to compress again. You'll just be wasting CPU cycles.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.