So I need to compress a directory with max compression.
How can I do it with xz? I mean I will need tar too because I can't compress a directory with only xz. Is there a oneliner to produce e.g. foo.tar.xz?
100
74
Assuming xz honors the standard set of commandline flags - including compression level flags, you could try:
tar -cf - foo/ | xz -9 -c - > foo.tar.xz
-
-
3
-
18
134
With a recent GNU tar on bash or derived shell:
XZ_OPT=-9 tar cJf tarfile.tar.xz directory
tar's lowercase j switch uses bzip, uppercase J switch uses xz.
The XZ_OPT environment variable lets you set xz options that cannot be passed via calling applications such as tar.
This is now maximal.
See man xz for other options you can set (-e/--extreme might give you some additional compression benefit for some datasets).
XZ_OPT=-e9 tar cJf tarfile.tar.xz directory
-
23No, you don't. That's the whole point. You can set the environment var for just that invocation. You can export it if you want to, but you don't have to. – bsd Apr 23 '13 at 9:36
-
1
-
6@anddam, that's supported by all shells of the Bourne family (Bourne, ksh, mksh, pdksh, ash, dash, bash, yash, zsh) and
rcandakanga.fish,csh,tcshandesbeing the major shells that don't support it. There, you'd use theenvcommand. – Stéphane Chazelas Jan 20 '15 at 10:33 -
1So to set both
-9and-exz opts, you wantXZ_OPT=-e9but as @krzyk pointed out, -e is extremely slow – hobs Dec 23 '16 at 20:48 -
2Just for the record:
XZ_OPTis not a feature implemented intar. It's a feature ofxz. Whentarcallsxz, the env-variable is simply passed on. – Sven Nov 20 '17 at 12:37
11
XZ_OPT=-9e tar cJf tarfile.tar.xz directory
is even better than
XZ_OPT=-9 tar cJf tarfile.tar.xz directory
-
5
-
2
option -e, --extremeModify the compression preset (-0 ... -9) so that a little bit better compression ratio can be achieved without increasing memory usage of the compressor or decompressor (exception: compressor memory usage may increase a little with presets -0 ... -2). The downside is that the compression time will increase dramatically (it can easily double). – Evandro Jr Apr 25 '16 at 8:46 -
So, If i'm compressing about 80GB of Software on my machine (when i want all the computers resources to go to the compression process for speed) i should use
-9not-9e, yeah? – nyxee Aug 28 '17 at 21:13 -
xz by default uses 1 core/thread, you can max that out (speed it all up) by adding -T0, eg
XZ_OPT="-9e -T0" tar -cJf ...– EkriirkE Jan 28 at 22:53
9
If you have 16 GiB of RAM (and nothing else running), you can try:
tar -cf - foo/ | xz --lzma2=dict=1536Mi,nice=273 -c - > foo.tar.xz
This will need 1.5 GiB for decompression, and about 11x that for compression. Adjust accordingly for lesser amounts of memory.
This will only help if the data is actually that big, and in any case it won't help THAT much, but still...
If you're compressing binaries, add --x86 as the first xz option. If you're playing with "multimedia" files (uncompressed audio or bitmaps), you can try with --delta=dist=2 (experiment with value, good values to try are 1..4).
If you're feeling very adventurous, you can try playing with more LZMA options, like
--lzma2=dict=1536Mi,nice=273,lc=3,lp=0,pb=2
(these are the default settings, you can try values between 0 and 4, and lc+lp must not exceed 4)
In order to see how the default presets map to these values, you can check the source file src/liblzma/lzma/lzma_encoder_presets.c. Nothing of much interest there though (-e sets the nice length to 273 and also adjusts the depth).
5
You might try different options, for me -4e works better
tar cf - wam_GG_${dir}.nc | xz -4e > wam_GG_${dir}.nc.tar.xz
I tested by running:
$ tar -cf - wam_GG.nc | xz -4e > wam_GG.nc.xz
$ tar -cf - wam_GG.nc | xz -9e > wam_GG.nc.xz.2
So, it seems that option -4e works a little bit better than -9e.
$ ll wam_GG.nc.xz*
-rw-rw-r--. 1 504 504 2707596 Jan 16 2015 wam_GG.nc.xz
-rw-rw-r--. 1 504 504 2708416 Jan 16 2015 wam_GG.nc.xz.2
-
3This really doesn't answer the question. This is just an observation that for your particular small data set, -4e already gets the best compression and so the higher levels don't get any more benefit ( and even an ever so slight penalty ). – psusi Jan 16 '15 at 16:00
-
Are you the same user as Szymon Roziewski? If so, please don't post multiple answers. Instead, edit your original answer. If you can't access your first account, please see here for how to merge your accounts. In the meantime, I am deleting your previous answer and including it here. – terdon♦ Jan 16 '15 at 16:35
-
Ok, I have done a more comprehensive study on that. What I got is here. I chose some files from my hardrive and made compression with option -4e and -9e. So, it's better to find your best solution by yourself. You were right, for some cases -9e is better whereas for another it's not:
no difference = 660 4e better than 9e = 74 9e better than 4e = 17 total files = 751 tar 2 html 2 csv 2 xml 2 gz 2 ppt 2 eps 2 docx 2 gif 2 rpm 3 png 3 asv 3 xlsx 3 exe 3 rar 4 nc 4 txt 5 odt 6 xls 7 zip 7 doc 9 m 12 dat 17 other 109 pdf 133 135 jpg 270– Szymon Roziewski Jan 20 '15 at 9:51 -
(comments may be edited only for 5 minutes)
txt 109 txt/pdf 135– Szymon Roziewski Jan 20 '15 at 9:59 -
2+1. This does help the OP find a way to determine maximum compression for
taring files usingxz. – cychoi Feb 10 '15 at 7:56
4
tar command uses J flag for xz files. An example:
tar -cJvf foo.tar.xz foo/
-
2
4
tar --help : -I, --use-compress-program=PROG
tar -I 'xz -9' -cvf foo.tar.xz foo/
tar -I 'gzip -9' -cvf foo.tar.gz foo/
also compress with external compressors:
tar -I 'lz4 -9' -cvf foo.tar.lz4 foo/
tar -I 'zstd -19' -cvf foo.tar.zst foo/
decompress external compressors:
tar -I lz4 -xvf foo.tar.lz4
tar -I zstd -xvf foo.tar.zst
list archive external compressors:
tar -I lz4 -tvf foo.tar.lz4
tar -I zstd -tvf foo.tar.zst
-
1This seems like a working answer, but, as it is, it would be greatly improved by having its formatting fixed and and explanation of option
-Iadded. – dhag Oct 27 '17 at 15:12
2
This is not an exact answer to your question but you could use one command instead of two:
7z a -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on archive.7z dir1
adds all files from directory "dir1" to archive archive.7z using "ultras ettings"
other formats supported are: zip, gzip, bzip2 or tar. for this just replace 7z after -t.
--source man 7z
NOTE: don't use this command to backup your system files except personal files because the 7z format doesn't store filesystem permissions.
-
3The question was about xz, not about 7z, even though they both use LZMA compression. – Amedee Van Gasse Jul 22 '15 at 7:57
2
For those interested, -e9 is 0.4% smaller, 20% slower at compression, 3% slower for decompression, compared to -9 on a typical laptop. Here're the timing runs on the Python source code directory structure.
Compression:
$ T0=`date +%s%3N` && XZ_OPT=-9 tar cJf python3.6.tar.9xz Python-3.6.0 && T1=`date +%s%3N`
$ python -c "print((float($T1) - float($T0)) / 1000.)"
43.87
$ T0=`date +%s%3N` && XZ_OPT=-e9 tar cJf python3.6.tar.e9xz Python-3.6.0 && T1=`date +%s%3N`
$ python -c "print((float($T1) - float($T0)) / 1000.)"
53.861
Decompression:
$ T0=`date +%s%3N` && tar xf python3.6.tar.9xz && T1=`date +%s%3N`
$ python -c "print((float($T1) - float($T0)) / 1000.)" && rm -rf Python-3.6.0
1.395
$ rm -rf Python-3.6.0
$ T0=`date +%s%3N` && tar xf python3.6.tar.e9xz && T1=`date +%s%3N`
$ python -c "print((float($T1) - float($T0)) / 1000.)" && rm -rf Python-3.6.0
1.443
File Size:
$ rm -rf Python-3.6.0
$ T0=`date +%s%3N` && tar xf Python-3.6.0.tar.xz && T1=`date +%s%3N`
$ python -c "print((float($T1) - float($T0)) / 1000.)" && rm -rf Python-3.6.0
1.49
$ ls -al ?ython*
-rw-rw-r-- 1 hobs hobs 16378500 Dec 23 13:06 python3.6.tar.9xz
-rw-rw-r-- 1 hobs hobs 16314420 Dec 23 13:05 python3.6.tar.e9xz
-rw-rw-r-- 1 hobs hobs 16805836 Dec 23 12:24 Python-3.6.0.tar.xz
1
If you would like this to complete faster, using multiple threads, but without slowing down your system while you perform other work, try adding -Tn where n is how many threads you want to use, as well as nice to demote the compression to idle priority.
Model (for 4 threads):
tar c foo/ | nice -n19 xz -9 -T4 > foo.tar.xz
Try watching in top or htop when you do this in a big directory (several GB). You should hopefully see several xz threads with Nice value of 19 (lowest priority).
I've also stripped this down be as terse as sensible, such as: the -f - in other answers is simply not needed, since tar's default output is stdout.
You can nice the tar process also, but I've never found it necessary, as xz always bottlenecks the CPU for the pipeline.
Practical note, I rarely use xz -9 for anything, not so much due to CPU or time, but because of the high memory demands. Take a look at https://catchchallenger.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO#Memory_requirements_on_compression. The xz compressor, like bzip2, but unlike gzip, uses more memory for higher compression factors. Put that together with that xz uses far more memory than any other compressor, you can easily use up 600+ MB of memory. And if you use the -T to enable threaded compression, the memory demands go up even further. Just something to be aware of, like if you're running some small service on a small VM with 1-2 GB memory, you could inadvertently cause an impact.
1
In a multicore machine from version v5.2.0 of xz-utils, check:
-T, --threads=NUM use at most NUM threads; the default is 1; set to 0
If you wish to use the maximum number of cores and maximum compression:
export XZ_DEFAULTS="-9 -T 0 "
Or set -T to the number of cores you wish to use.
Then:
tar cJf target.tar.xz source
Also this may useful in order to choose the compression level:
0
On Mac OS X, an alternate approach to pass in the parameter with tar is to use a --options= flag. For example,
tar Jcvf targetFileName.tar.xz --options='compression-level=9' directoryName
man 1 xzsaysit's not a good idea to blindly use -9 for everything like it often is with gzip(1) and bzip2(1).-7 ... -9 [...] These are useful only when compressing files bigger than 8 MiB, 16 MiB, and 32 MiB, respectively.RTFM for more info. – cychoi Feb 10 '15 at 7:42