I have the following process I use for backups:

btrfs send snapshots/home-2016-06-04 |
zstd --verbose -T4 - |
gpg --batch --passphrase-file /tmp/secret --compress-algo none --symmetric - >

But I find that while it runs very quick at 100% CPU for a while, writes to the usb drive get buffered and then the whole pipeline suddenly stalls while it syncs to the drive. It then alternates between these two states.

What I would like to achieve is for zstd --adapt -T1 to work properly. According to the man-page

zstd will dynamically adapt compression level to perceived I/O conditions.

This should give the optimal throughput so that the write is at the maximum (40 MB/s - tested with dd) to the thumb drive, and so that zstd will not fill up it's buffer.

I suspect I can achieve this by adding dd as the last stage of the pipeline; I've tried this with direct fsyncdata and sync options to oflag, to no avail.

Is there some way to achieve this? Am I right in assuming the problem is with the write caching?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.