30

I use

tar -cJvf resultfile.tar.xz files_to_compress

to create tar.xz and

tar -xzvf resultfile.tar.xz

to extract the archive in current directory. How to use multi threading in both cases? I don't want to install any utilities.

Improve this question

3 Answers 3

Reset to default
41 
tar -c -I 'xz -9 -T0' -f archive.tar.xz [list of files and folders]

This compresses a list of files and directories into an .tar.xz archive. It does so by specifying the arguments to be passed to the xz subprocess, which compresses the tar archive.

This is done using the -I argument to tar, which tells tar what program to use to compress the tar archive, and what arguments to pass to it. The -9 tells xz to use maximum compression. The -T0 tells xz to use as many threads as you have CPUs. Note that if you are on MacOS or a BSD distro, the -I option is available only on GNU tar, not on BSD tar. For MacOS users, this can be resolved by installing gnu-tar through Homebrew and using the gtar command instead of tar.

An update from January, 2024:

Even when using the multithreading option, xz barely scales beyond two threads, besides its compression/decompression performance is relatively low.

I highly recommend using ZSTD instead. The command will be:

tar -c -I 'zstd -22 --ultra --long -T0' -f archive.tar.xz [list of files and folders]

Caveats:

  • This commands needs a lot of RAM, at the very least 5GB. You may want to reduce the compression level, or remove ultra/long options to decrease RAM consumption.
  • Threading might not be used if you don't have enough source data (less than 1GB). If you still want to use threading for a small amount of data, decrease the compression level from 22 to say 20.
Improve this answer
8
  • 1
    On my TAR though, -I won't accept arguments, and expects a program only. Maybe you can make a simple shell script that is exec xz -9 -T0 $*?
    – ZiggyTheHamster
    Commented Oct 14, 2020 at 1:51
  • This program is used for piping, so $* is redundant. Also, not sure why you want to use exec.
    – Artem S. Tashkinov
    Commented Oct 14, 2020 at 9:39
  • it is freaking fast and very less filesize
    – Thamaraiselvam
    Commented Oct 28, 2020 at 6:53
  • 1
    Warning: If you have many CPUs, this will use a lot of RAM.....
    – starbeamrainbowlabs
    Commented Nov 24, 2020 at 17:54
  • 2
    This returns tar: Couldn't open xz -9 -T0: No such file or directory here (macOS). The following from another answer worked: tar -cf - list of files and folders| xz -9 -T0 >| archive.tar.T.xz
    – aaronk6
    Commented Jan 7, 2022 at 12:55
31

You can use XZ_DEFAULTS or XZ_OPT environment variables:

XZ_DEFAULTS is recommended to be used as a system wide configuration, typically set in a shell initialization script.

XZ_OPT is for passing options to xz when run by a script or tool, e.g. GNU tar. See man xz.

Example: using multiple threads (-T0) and max compression level (-9):

XZ_OPT='-T0 -9' tar -cJf resultfile.tar.xz files_to_compress

Reference with a recent GNU tar on bash or derived shell, also see the xz man page

Improve this answer
4

For older tars, this works:

tar -cf -  list of files and folders| xz -9 -T0 >| archive.tar.T.xz
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .