I'm trying to figure out a way that I can update a compressed file (currently using zip, but am open to tar/gz/bz derivatives too) on a linux server without creating a temp file for the file to be compressed.
I'm compressing an entire domain's directory (about 36Gb +- at any given time) and I have limited drive space on the webserver. The issue being that, as zip builds the new compressed file, it creates a temp file which presumably overwrites the existing zip file when it's complete, but in the process, the 36Gb of the source directory + the 32Gb of the existing zip file + the 30 some Gb of the temp file come very close to maxing out my drive space and at some point in the future, it will exceed the drives available space.
Currently, the directory is backed up using a cronjob command, like so...
0 0 * * * zip -r -u -q /home/user/SiteBackups/support.zip /home/user/public_html/support/
I don't want to delete the zip file each time, firstly because the directory is zipped every 4 hours and also because the directory is so large, it is rather resource intensive to re-zip the entire directory as opposed to just updating it - at least I believe that to be true. Perhaps I'm wrong?
Additionally, breaking it out into different commands for different directories will not work because a large portion of the data (30 ish Gb out of the total 36Gb) is all in one directory and the file names are GUIDs, so there's no way to target files in a predictable way.
Thanks in advance to the sysadmins with some terminal jujitsu!