2

I am running a command (pngquant to be precise: https://github.com/pornel/pngquant) in a terminal window. I noticed, that if I open 4 terminal windows, and run pngquant command in each of them, I get 4x speed increase, effectively compressing 4 times as many images in the same time as before.

So I used this approach and assigned each pngqunat process a portion of images I want to compress, effectively creating multiple processes on multiple threads

Can you run command on multiple threads without doing this tricks that I did? I would like to just say "run pngquant compression on all this images and use all threads available."

3

Both moreutils parallel and GNU parallel will do this for you. With moreutils' parallel, it looks like:

parallel -j "$(nproc)" pngquant [pngquant-options] -- *.png

nproc outputs the number of available processors (threads), so that will run available-processors (-j "$(nproc)") pngquants at once, passing each a single PNG file. If the startup overhead is too high, you can pass more PNG files at once to each run with the -n option; -n 2 would pass two PNGs to each pngquant.

  • Thanks. Can every command be run with moreutils parallel? The extra command in question is github.com/krzysztof-o/spritesheet.js How do you know if the command can be run in parallel. I saw multicore support (via OpenMP) and Intel SSE optimizations line with pngquant, but not with spritesheet-js – sanjihan May 10 '17 at 17:37
  • @sanjihan parallel works by running the command multiple times at once, and dividing the input files between the different instances. When one run finishes, it fires up another one, giving it more of the input files. So it'll work with any command where that is a sensible thing to do. Sometimes these are called trivially parallelizable or embarrassingly parallel problems. Moreutils parallel is a (fairly simple, actually) trick, not magic. Spritesheet.js (from a very quick glance) seems to need all the input files given to one run, so it won't work. – derobert May 10 '17 at 17:42
  • oooh, I see. Thanks, your explanations are awesome. – sanjihan May 10 '17 at 17:49
2

With GNU xargs:

find . -type f -name '*.png' -print0 | xargs -r0 -n10 -P "$(nproc)" pngquant

(each pngquant invocation given up to 10 files to compress, up to "$(nproc)" (the number of processors on your system) running at the same time)

1

Using GNU Parallel it looks like this:

parallel pngquant --my-options ::: *.png

or:

ls | grep \\.png | parallel pngquant --my-options

It defaults to one job per CPU core. In your case you might want to run one more job than you have cores:

ls | grep \\.png | parallel -j+1 pngquant --my-options

This is because pngquant uses time to both read and write data. During this time it waits for the disk and hardly uses any CPU, and thus you might be having some idle CPU time. The only way to know for sure is to measure and see which is faster.

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

Simple scheduling

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

GNU Parallel scheduling

Installation

For security reasons you should install GNU Parallel with your package manager, but if GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.