3

I am executing following pipeline: tar -c directory | pv -T -c -B 2G | gzip -c9 | pv -T -c -B 2G | split -b 1G - /mnt/usbStick/f.tar.gz_

The idea is to gzip a >4GB directory onto a large FAT32-formatted (reliably supports only <2GB files) USB stick. split just cuts a .gz into 1GB pieces.

pv is used as pipe buffer (2GB each), mainly to prevent gzip throttling when it stumbles upon poorly compressible data and outputs compressed data faster than USB stick can write it.

The problem:

When yet another 1GB part is finished, split flushes and closes the file. It does not start writing next part until flush is complete, so it stops accepting input.

I'd expect the second 2GB buffer to start filling up at this point, but instead everything stops. System is far from dead overall, but gzip stops using CPU and even pv stops updating its output. From the latter part I conclude that all IO is stalled, even pipes. (Prove me wrong if I am.) (I was wrong, just that specific pipeline stalled.)

So the question is, why it works this way and how to fix it.

Edit
It really was pv's fault, buffer works fine. Build included in Ubuntu 16.04 can use only up to 2048 blocks up to 512KB each (buffer -m 1024m -s 512k), but they can be daisy-chained to form larger buffer.

3

Since pv is just a monitoring tool, I guess it's a single-threaded program which uses synchronous IO. Given input and output running at different rates, it will fill and empty its buffer to prevent throttling, but if one of the pipes is completely stalled (in your case it's split not accepting input), the second pv instance will stall in the write call just as well. Then gzip will try to output more data, overflow the pipe buffer and also stall, and so on until the whole pipeline is stalled.

Try using buffer instead of pv and see if that helps. buffer spawns two processes (one for input and one for output), and should keep the pipeline running even if the output is completely stalled.

  • 1
    Thanks. buffer works as expected. Actually, I tried it before pv, but it complained about too many blocks, too large buffer, etc., so I concluded that it was too ancient and so I gave up on it. – EvgEnZh Oct 7 '16 at 6:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.