I have a folder containing approximate 50GB of images and video files (~35k files) that I'd like to compress into a single file for storage along side a database. Are there any compression options (or utilities with options) that will actually take advantage of the reproduced data across those many small files and provide some reasonable compression? Standard "tar czf", etc don't seem to consider large enough chunks of data and end up returning a file nearly the size of the uncompressed directory.

(And yes, I know you generally can't compress one large file more than it already is, I'm looking for something that'll detect patterns across the files.)

(Note: Eventual goal is to add this functionality to an application I'm writing for photo organization, this is just a test case. Replying with "just use tar cjf" isn't terribly helpful. :P)

  • AFAIK, there is no Linux compressor that will dedup parts of images.
    – fpmurphy
    Jul 17, 2017 at 9:59

2 Answers 2


Try this,

gzip=-9 tar -cvzf videos_backup.tgz </path/to/videos>

This might take some time to compress the file in the directory.


I would use this:

tar cvf images.tar file1 file2 ...
gzip -9 images.tar  

You roll all of your files into one, and then compress it; he copmmon parts are all then in one file. How well this works will depend on your compression program of choice (and the options you feed it); experiment with different options and programs (e.g. gzip, xz and even perhaps bzip2, although I would probably expect xz to win). Note that some programs/options might be quite slow, and if you have limited memory that might be a problem.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .