I have the following process I use for backups:
btrfs send snapshots/home-2016-06-04 | zstd --verbose -T4 - | gpg --batch --passphrase-file /tmp/secret --compress-algo none --symmetric - > /mnt/usbdrive/packup.zst.gpg
But I find that while it runs very quick at 100% CPU for a while, writes to the usb drive get buffered and then the whole pipeline suddenly stalls while it syncs to the drive. It then alternates between these two states.
What I would like to achieve is for
zstd --adapt -T1 to work properly. According to the man-page
zstd will dynamically adapt compression level to perceived I/O conditions.
This should give the optimal throughput so that the write is at the maximum (40 MB/s - tested with dd) to the thumb drive, and so that
zstd will not fill up it's buffer.
I suspect I can achieve this by adding
dd as the last stage of the pipeline; I've tried this with
sync options to
oflag, to no avail.
Is there some way to achieve this? Am I right in assuming the problem is with the write caching?