Updating a large compressed file without creating temp file

Question

I'm trying to figure out a way that I can update a compressed file (currently using zip, but am open to tar/gz/bz derivatives too) on a linux server without creating a temp file for the file to be compressed.

I'm compressing an entire domain's directory (about 36Gb +- at any given time) and I have limited drive space on the webserver. The issue being that, as zip builds the new compressed file, it creates a temp file which presumably overwrites the existing zip file when it's complete, but in the process, the 36Gb of the source directory + the 32Gb of the existing zip file + the 30 some Gb of the temp file come very close to maxing out my drive space and at some point in the future, it will exceed the drives available space.

Currently, the directory is backed up using a cronjob command, like so...
0 0 * * * zip -r -u -q /home/user/SiteBackups/support.zip /home/user/public_html/support/

I don't want to delete the zip file each time, firstly because the directory is zipped every 4 hours and also because the directory is so large, it is rather resource intensive to re-zip the entire directory as opposed to just updating it - at least I believe that to be true. Perhaps I'm wrong?

Additionally, breaking it out into different commands for different directories will not work because a large portion of the data (30 ish Gb out of the total 36Gb) is all in one directory and the file names are GUIDs, so there's no way to target files in a predictable way.

Thanks in advance to the sysadmins with some terminal jujitsu!

Answer 1

This is almost certainly not going to work (update: see also this answer)

The Zip archive (but things change little with other archives) is built like a file system:

Suppose we were to update File#1 without moving File#2, and with File#1 potentially larger once compressed. This would require:

• remove Central Header
• add File#1 Data (2nd copy) after File#2
• add Central Header back again, with updated offset for File#1

creating a "dead zone" at the beginning of the Zip file. It would be possible to use that area for further storage of another file. Basically you'd need to zip the incoming file into a temporary file, thus getting its final size; armed with that, you would scan the zip file and look for "holes". If a suitable "hole" exists, copy the temporary file inside the zip file, possibly leaving a smaller "hole"; otherwise, add it by replacing the central header.

While possible, managing the slack space inside the Zip archive as well as the coalescing of adjacent "holes" requires care, and to my knowledge nobody ever did that (I could for example write a compression-agnostic utility to replace a file inside a Zip file, using the main zip utility to generate the new compressed stream and replacing the old file name with a recognizable sequence to mark it as free space; it would be horribly slow).

The closest you can get to what you want would be to use a completely different format - you'd create, say, a btrfs file system on a loop device, setting it to the maximum compression available (I believe that would be LZO). Then you mount the loop device and use rsync to update it. Unmount the loop device, and the host file is a compressed archive... of sorts. Depending on the file nature, you might even be able to leverage btrfs's deduplication capability.

The compression ratio of compressed file systems is lower than Zip, but several files (PDF, ZIP obviously, most image formats like JPEG, PNG and GIF, modern (Libre)Office formats...) cannot be compressed, so this is not a problem. Since you say that the uncompressed files are 36Gb and the Zip is 32Gb, you're probably in this situation, and would probably benefit from a non-compressed format).