0

I'm excited to learn that xz now supports multithreading:

xz --threads=0

But now I want to utilise this as much as possible. For example, to recompress gzips as xz:

gzip -d -k -c myfile.gz | pv | xz -z --threads=0 - > myfile.xz

This results in my processor being more highly used (~260% CPU to xz, yay!).

However:

  • I realise that gzip is not (yet) multithreading,
  • I think that either pv or the pipes may be restricting the number of (IO?) threads.

Is this true and, if so, is there a way to make this more efficient (other than to remove pv)?

  • Can you give more details about the scenario? In my answer I point out that the number of .gz files matters a lot. – rastafile Oct 24 at 7:05
  • In my case it's actually one large disk image being recompressed. I came across this because of the gzip 32-bit size reference limit and I wanted my compressed files to show the right uncompressed size. There was a significant improvement in recompression, though, by about 40% compared to without threads, mostly during the nulled part of the uncompressed image where it reached 100MB/s (over USB3.0) according to pv. I put this down to lower IO wait times on the gzip end due to more "waiting" threads on the xz end but I wondered whether the pipe and pv were a bottleneck. – tu-Reinstate Monica-dor duh Oct 24 at 7:24
  • And did you time the difference at all between -T0 and without? Acoording to my trials it does not make a difference, you loose as much as you win. – rastafile Oct 24 at 7:25
  • Yes, that's how I got the ~40%, but I didn't keep a copy, sorry. – tu-Reinstate Monica-dor duh Oct 24 at 7:26
  • I got over 300%. So this shows that input (pipe) is the big bottleneck, not xz. It is in between my 300% and the 0.1% you get when you slow down the dd pipe with count=10 (only 10 bytes per read). – rastafile Oct 24 at 7:30
0

With the -T0 multithread option you tell xz two things at once. To use MT also means: wait until all input (data) is read into memory, and then start to compress in "parallel".

After including pigz into my tests I analyze the perfomance step by step; I have a 100M file f100.

$  time xz -c  f100 >/dev/null

real  0m2.658s
user  0m2.573s
sys   0m0.083s

99% of the time is spent compressing on one core. With all four cores activated with -T4 (or -T0)

$  time xz -c -T4 f100 >/dev/null

real  0m0.825s
user  0m2.714s
sys   0m0.284s

Overall result: 300% faster, almost linear per core. The "user" value must be divided by 4, the way it is reported/defined. "sys" now shows some overhead -- real is the sum of 1/4 user plus sys.

$  time gzip     -dc f100.gz >/dev/null
$  time pigz -p4 -dc f100.gz >/dev/null

This is 0.5 vs. 0.2 seconds; when I put all together:

$  time pigz -dc -p4 f100.gz | xz -c -T4 >out.xz

real  0m0.902s
user  0m3.237s
sys   0m0.363s

...it reduces 0.8 + 0.2 = 0.9.

With multiple files, but not too multiple, you can get highest overall parallelism with 4 shell background processes. Here I use four 25M files instead:

for f in f25-?.gz; do time pigz -p4 -dc "$f" | xz -c -T0 >"$f".xz & done

This seems even slightly faster with 0.7s. And even without multithreading, even for xz:

for f in f25-?.gz; do time gzip -dc "$f" | xz -c >"$f".xz & done

just by setting up four simple quarter pipelines with &, you get 0.8s, same as for a 100M file with xz -T4.

In my scenario it is about just as important to activate multithreading in xz than it is to parallelize the whole pipeline; if you can combine this with pigz and/or multiple files, you can even be a bit faster than a quarter of the sum of the single steps.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.