11

Why is this not possible?

pv ${dest_file} | gzip -1

pv is a progress bar

error

gzip: compressed data not written to a terminal. Use -f to force compression.
For help, type: gzip -h
   0 B 0:00:00 [   0 B/s] [>                                   ]  0%

This works

pv ${file_in} | tar -Jxf - -C /outdir
1
  • 1
    what are you trying to achieve? Did you read do docs of pv and gzip command? – Jakuje Dec 26 '15 at 21:52
25

What are you trying to achieve is to see the progress bar of the compression process. But it is not possible using pv. It shows only transfer progress, which you can achieve by something like this (anyway, it is the first link in the google):

pv input_file | gzip > compressed_file

The progress bar will run fast, and then it will wait for compression, which is not observable anymore using pv.

But you can do that other way round and watch the output stream, bot here you will not be able to see the actual progress, because pv does not know the actual size of the compressed file:

gzip input_file | pv > compressed_file

The best I found so far is the one from commandlinefu even with rate limiting and compression of directories:

$D=directory
tar pcf - $D | pv -s $(du -sb $D | awk '{print $1}') --rate-limit 500k | gzip > target.tar.gz
6
  • the progress bar is not working with gzip input_file | pv > compressed_file – clarkk Dec 26 '15 at 22:13
  • it is, but it does not know the whole size of the compressed file so it just scans from the side to the side. The first one is probably better. – Jakuje Dec 26 '15 at 22:18
  • The progress bar doesn't run fast for me. – mikezter Apr 3 '18 at 19:03
  • 1
    gzip -c is required to send output to stdout to allow it to be passed to pv, otherwise the progress bar won't work as there's no data going to pv :) – David Gardner Apr 5 '18 at 11:28
  • Thanks for the tip! You can also achieve this with something like mysqldump ... | gzip | pv > output.sql.gz. – jevon May 14 '18 at 5:27
5

Yes, it is kinda possible using the lines of the file, not the bytes. You get an accurate enough progress bar:

cat input_file | pv -ls $( wc -l input_file ) | gzip -c -- > compressed_file
1

You can use 7z instead to get a percentage progress:

$ 7z a spaCy.tar.gz spaCy.tar 

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz (906E9),ASM,AES-NI)

Open archive: spaCy.tar.gz
--
Path = spaCy.tar.gz
Type = gzip
Headers Size = 20

Scanning the drive:
1 file, 326451200 bytes (312 MiB)

Updating archive: spaCy.tar.gz

Items to compress: 1

 32% U spaCy.tar   

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.