I am executing following pipeline:
tar -c directory | pv -T -c -B 2G | gzip -c9 | pv -T -c -B 2G | split -b 1G - /mnt/usbStick/f.tar.gz_
The idea is to gzip a >4GB directory onto a large FAT32-formatted (reliably supports only <2GB files) USB stick.
split just cuts a
.gz into 1GB pieces.
pv is used as pipe buffer (2GB each), mainly to prevent
gzip throttling when it stumbles upon poorly compressible data and outputs compressed data faster than USB stick can write it.
When yet another 1GB part is finished,
split flushes and closes the file. It does not start writing next part until flush is complete, so it stops accepting input.
I'd expect the second 2GB buffer to start filling up at this point, but instead everything stops. System is far from dead overall, but
gzip stops using CPU and even
pv stops updating its output.
From the latter part I conclude that all IO is stalled, even pipes. (Prove me wrong if I am.) (I was wrong, just that specific pipeline stalled.)
So the question is, why it works this way and how to fix it.
It really was
buffer works fine. Build included in Ubuntu 16.04 can use only up to 2048 blocks up to 512KB each (
buffer -m 1024m -s 512k), but they can be daisy-chained to form larger buffer.