12

Is gzip atomic?

What happens if I stop the gzip process while it's in the middle of gzipping a file?

If it's not atomic, and if I already pressed Ctrl+C on a gzip *.txt process, how do I safely resume?

(I am not just curious about how to resume, but also about whether gzip specifically is atomic.)

Improve this question
  • Possible duplicate of How do I resume a tar command which was killed – Anthony Geoghegan Aug 23 '19 at 10:05
  • 4
    " how do I safely resume? "_... Use CTRL+Z instead of CTRL+C, then kill or resume the interrupted job (it answers with a number n [--[n]+ Stopped-- gzip ... ] then you can resume with %n or with fg, or with bg... in the same way you can kill it with kill %n). – Hastur Aug 23 '19 at 17:32
  • Compress a large file, Ctrl-C while compressing, and see what happens. – RonJohn Aug 24 '19 at 3:57
  • No. Only mv is atomic, except on ext4… sarcasm dripping, but at least they fixed the default mount options some time ago. – mirabilos Aug 25 '19 at 2:32
30

Is gzip atomic?

No. It creates a compressed file and then removes the uncompressed original.

Specifically, it does not compress a file in situ and there is a period of time while the file is being compressed where,

  • the compressed target is incomplete
  • the partially compressed file and its source both exist in the filesystem.

What happens if I stop the gzip process while it's in the middle of gzipping a file?

If you stop the gzip process with a catchable signal (SIGINT from Ctrl C, for example) it will cleanup partially created files. Otherwise, depending on the point at which it's stopped, you may end up with a partially compressed file alongside the untouched original.

If it's not atomic, if I already pressed Ctrl+C on a gzip *.txt process, how do i safely resume?

You delete the partially compressed version (if it still exists) and restart the gzip.

Improve this answer
  • 5
    the 2nd happens when the process is terminated, not when it's stopped, and only happens for non-handled signals (not for ^C -> SIGINT or SIGTERM for which gzip installs signal handlers which remove the output file). – mosvy Aug 23 '19 at 9:11
  • 1
    @mosvy so it does. I've never seen that before. Thank you – roaima Aug 23 '19 at 9:21
  • 1
    You take extremely great care to assure you don't delete any gzipped files for thich the original has been deleted. When gzip is killed irregularly, that is usually one file, usually the last one. – Harper - Reinstate Monica Aug 24 '19 at 17:00
  • @Harper yes. If you stop gzip mid flow there's always a tiny race condition there. Alternatively you can tell gzip always to overwrite target files, which sidesteps most of the cleanup issues. – roaima Aug 24 '19 at 17:04
15

It's not atomic (the Unix filesystem API doesn't really provide any way to perform atomic operations that affect multiple files), but it is fail-safe. The compressed file is a new file, it doesn't overwrite the original, and it doesn't delete the original file until it has completed creating the compressed file (this can actually cause an issue if you don't have enough disk space for both files).

If it gets an error or you interrupt the compression, the original file will remain unchanged. The partial compressed file will usually be removed.

There's no way to resume it in the middle, you just start it over from the beginning.

Improve this answer
  • This make me thinking about how possibly could atomic multifile operations be implemented. Something like SQL transactions? – val says Reinstate Monica Aug 24 '19 at 13:56
  • 1
    @val About 30 years ago I was on a team that was designing a new OS as a Multics/GCOS followon, and a database-like filesystem was part of the idea. The project never got very far, though. – Barmar Aug 24 '19 at 20:40
  • They removed NTFS transactions, seems to be not worth the complication. Rename is the most atomic operation (as long as you are on the same file system and it has posix semantics), so having a rename (after close/fsync) from temp to final name would ensure the uncompressed file is at least complete. You can work around those problems with using pipes (which have their own partial failure modes) – eckes Aug 24 '19 at 23:03
  • @eckes As long as it deletes the original after it closes the compressed file, you don't need the atomic rename. If the original is gone, you can be sure that the compressed file is complete. You need atomic rename for operations that replace the original file (e.g. sed -i). – Barmar Aug 25 '19 at 17:24
  • @Barmar if you only want to Trigger by the existence of the target file (which many directory polling workflows do) you better be sure the file is complete. If you do not trigger on that or can detect incomplete files by checking for source existence, then you are fine without the final rename. – eckes Aug 25 '19 at 19:08
4

You don't need to worry about that because gzip creates a new .gz file, populates it with the compressed content, then deletes the original file. So if you stop the process in the middle, it won't affect your original file.

Improve this answer
3

.txt files already successfully processed by gzip will have been replaced with .txt.gz compressed files, so you can safely run gzip *.txt again - only the files that haven't been processed yet will be compressed.

The file that was being processed by gzip at the time you pressed Ctrl-C will be unmodified - gzip won't replace it until after successfully compressing it.

Improve this answer
0

No, it's very unatomic. This can get you in big trouble if you gzip a file that's being occasionally appended to, like a Web log.

Gzip reads, creates the .gz file (with current timestamp), copies the original file's timestamp, then deletes the original.

Certain interruptions may leave a stray, unfinished .txt.gz file right next to the .txt file. This then creates a data integrity issue: Which is the real file? Is this

  • a gzip which failed, leaving an incomplete / corrupted .txt.gz? Or
  • a gunzip which failed, leaving an incomplete / truncated .txt file? Or
  • A file successfully gzipped into txt.gz, and a newly created .txt file?

(This last happens when you go into your HTTP log directory and go gzip *).

I generally find it's prudent to sort this out by hand, unless you know exactly what happened because you just did it.

Fortunately gzip usually operates serially so you should only have this problem with one file. Paralleling gzip isn't a good idea - even though it'll use CPU more fully, it'll thrash the disk forcing it to read several files at once, greatly slowing down all gzip's. SSD or RAMdisk, on the other hand...

Improve this answer
  • 1
    @roaima. We do indeed, I was relying on a slang meaning we used to use a long time ago at one place I worked. Correcting to the common definition. – Harper - Reinstate Monica Aug 25 '19 at 2:46
  • 1
    If you are going to downvote please leave a comment explaining why. – JBentley Aug 25 '19 at 12:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.