I'm in the process of compressing a very large file (several TBs) with the following command, and it's been running for several days.

tar -I pigz -cf /path/to/compressed.tgz /dataset/to/compress

I'm slightly worried that I'll lose these days of progress over the smallest glitch.

For future reference, is there such a compression scheme/command available that allows one to resume where they've left-off? If not, is there an algorithmic reason for not being able to resume the compression where you've left off?

  • 1
    I don't know about existent commands, but this is possible. If you are using Huffman coding, one pass would write out symbol weights and form the tree, then a second pass would traverse the input and write the output. So long as you keep track of how far you are, this is trivial to stop and resume. I implemented this compression approach once, but didn't bother to make it resumable since my input files were always tiny – Fox Feb 27 '19 at 5:06
  • @Fox fascinating, I appreciate your input/experience here. I'll look into that – bd1251252 Feb 27 '19 at 6:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.