Best way to compress a folder of small pre-compressed files (images)?

Question

I have a folder containing approximate 50GB of images and video files (~35k files) that I'd like to compress into a single file for storage along side a database. Are there any compression options (or utilities with options) that will actually take advantage of the reproduced data across those many small files and provide some reasonable compression? Standard "tar czf", etc don't seem to consider large enough chunks of data and end up returning a file nearly the size of the uncompressed directory.

(And yes, I know you generally can't compress one large file more than it already is, I'm looking for something that'll detect patterns across the files.)

(Note: Eventual goal is to add this functionality to an application I'm writing for photo organization, this is just a test case. Replying with "just use tar cjf" isn't terribly helpful. :P)

Answer 1

Try this,

gzip=-9 tar -cvzf videos_backup.tgz </path/to/videos>

This might take some time to compress the file in the directory.

Answer 2

I would use this:

tar cvf images.tar file1 file2 ...
gzip -9 images.tar  

You roll all of your files into one, and then compress it; he copmmon parts are all then in one file. How well this works will depend on your compression program of choice (and the options you feed it); experiment with different options and programs (e.g. gzip, xz and even perhaps bzip2, although I would probably expect xz to win). Note that some programs/options might be quite slow, and if you have limited memory that might be a problem.