I have a folder containing approximate 50GB of images and video files (~35k files) that I'd like to compress into a single file for storage along side a database. Are there any compression options (or utilities with options) that will actually take advantage of the reproduced data across those many small files and provide some reasonable compression? Standard "tar czf", etc don't seem to consider large enough chunks of data and end up returning a file nearly the size of the uncompressed directory.
(And yes, I know you generally can't compress one large file more than it already is, I'm looking for something that'll detect patterns across the files.)
(Note: Eventual goal is to add this functionality to an application I'm writing for photo organization, this is just a test case. Replying with "just use tar cjf" isn't terribly helpful. :P)