1

I have a backup script that runs the following commands:

tar -c dir1 dir2 | xz -9 -T0 | gpg -c --batch --passphrase xxx | aws s3 ...

The return values are always the same: tar fails with 141 (broken pipe error) and xz returns 137 (no other error messages, even in verbose mode).

The script is tested, runs as root and works well on other servers. Initially I thought that the data I'm backing up could be corrupt and deleted some socket files within the backup directory (which is a rsnapshot folder), but that hasn't helped as well.

Does anyone have an idea what the problem could be?

EDIT: If I remove xz from the pipe it works.

5
  • 3
    137 would indicate SIGKILL. Could be killed because it uses too much resources like RAM or CPU. Dec 1, 2021 at 12:02
  • Did you try the default compression level (6 instead of 9) or even a lower compression level? Sometimes I use -T0, and it works for me. Maybe try using fewer CPU cores in this particular computer. I use --block-size=24MiB.
    – sudodus
    Dec 1, 2021 at 12:18
  • 1
    @StéphaneChazelas @sudodus you were both right, it was a CPU and RAM problem. Confining the script with systemd-run --scope -p MemoryMax=1500M -p CPUWeight=20 -p CPUQuota=35% ${SCRIPT} helps. Thank you.
    – manifestor
    Dec 1, 2021 at 12:21
  • @manifestor are you really bound to XZ compression? zstd achieves similar results at a much lower resource usage / higher throughput. For example, in my experience, replacing xz -T0 -6 on a mixed picture/source code/binary backup with zstd -15 yielded 5% bigger files, but roughly 2 times faster compression, although I didn't use multithreading with zstd; you can still do that, if you want / need to, but seeing you're also doing gpg and TLS for the AWS transfer, you probably don't. Dec 1, 2021 at 14:26
  • @MarcusMüller thanks for your precious comment, I'll give zstd a try!
    – manifestor
    Dec 1, 2021 at 17:09

2 Answers 2

2

TL;DR: try

xz -9 -T{number of CPU cores - 1} --memlimit={reasonable amount of RAM}

or replace xz -T0 with zstd to get similar compression rates at higher speeds without even using multiple cores.


What happens here is quite likely that your xz gets killed by the out-of-memory killer of your operating system, so that the rest can survive. That of course breaks the pipe. (It's still a bit surprising; usually, xz -9 requires about at most 700 MB of RAM, which isn't that much per core). You can try with --memlimit=1000MiB to limit your RAM usage to 1000 MiB (or whatever). However, if this solves the problem, that means that your "reasonable CPU amount" can't fulfill the needs of your -9 compression setting, and xz had to choose a lower one. So, your problem is probably that you've got too little RAM for -9 and a thread for each CPU core, and nothing can solve that, other than reducing either one.

-T0 means "use as many threads as there are CPU cores", which is counterproductive, since you take the resulting data, and pass it through GPG (which isn't overly efficient itself and will quite possibly need about one CPU core for itself), and through the aws command, which in itself will do TLS encryption for the connection (and quite possibly unsuccessfully try to reduce the data volume using DEFLATE itself).

So, in the extreme case, -T should be used with at most the number of CPU cores you have minus one.

Generally, maybe don't use xz to begin with. It's an excellent compressor, for sure, but it's incredibly slow. I know you're probably paying per GB of storage, but:
zstd achieves similar results at a much lower resource usage / higher throughput.

For example, in my experience, replacing xz -T0 -6 on a mixed picture/source code/binary backup with zstd -15 yielded 5% bigger files, but roughly 2 times faster compression, although I didn't use multithreading with zstd (on an 8-core machine).

You can still enable multithreading, if you want / need to, but seeing you're also doing gpg and TLS for the AWS transfer, you probably don't (see above).

0

I recommend removing the -T0 or put a number there other than 0 (like maybe half your cpus or less). xz is almost certainly running out of memory and getting killed by OOM. Using -9 also increases memory use.

1
  • I'd like to add that -T0 means "use as many threads as there are CPU cores", which is counterproductive, since you take the resulting data, and pass it through GPG (which isn't overly efficient itself and will quite possibly need about one CPU core for itself), and through the aws command, which in itself will do TLS encryption for the connection (and quite possibly unsuccessfully try to reduce the data volume using DEFLATE itself). So, in the extreme case, -T should be used with at most the number of CPU cores you have minus one. Dec 1, 2021 at 14:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .