1

I'm trying to compress a large archive with multi-threading enabled, however, my system keeps freezing up and runs out of memory.

OS: Manjaro 21.1.0 Pahvo    
Kernel: x86_64 Linux 5.13.1-3-MANJARO
Shell: bash 5.1.9
RAM: 16GB
|swapon|
NAME      TYPE   SIZE   USED   PRIO
/swapfile file   32G    0B     -2

I've tried this with a /swapfile 2x the amount of RAM I have (32GB) but the system would always freeze once >90% of total RAM has been used, and would seem to not make use of the /swapfile.

|xz --info-memory|
Total amount of physical memory (RAM) : 15910 MiB
Memory usage limit for compression: Disabled
Memory usage limit for decompression: Disabled

I'm new to using xz so please bear with me, but is there a way to globally enable the memory usage limiter and for the Total amount of physical memory (RAM) to take into account the space made available by /swapfile?

2
  • 1
    What about --memlimit-compress=limit and/or --memlimit-decompress=limit?
    – taliezin
    Jul 23, 2021 at 22:37
  • 1
    xz really does excellent compression, but at high computational cost. I'd advise you to try zstd in a non-crazy compression setting, l do zstd -10 and rarely get files that are significantly larger than their xz brethren – just take a tenth of the time to compress and decompress. Jul 23, 2021 at 23:29

1 Answer 1

2

From man xz:

Memory usage

Especially users of older systems may find the possibility of very large memory usage annoying. To prevent uncomfortable surprises, xz has a built-in memory usage limiter, which is disabled by default.

The memory usage limiter can be enabled with the command line option --memlimit=limit. Often it is more convenient to enable the limiter by default by setting the environment variable XZ_DEFAULTS.

1
  • Works by setting either --memlimit or --threads to a lower number, thanks.
    – JKleinne
    Jul 23, 2021 at 23:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .