3

I've found answers close to this but fail to understand how to use them in my case (I'm rather new to Bash)... so, I'm trying to process a folder containing a large image sequence (100k+ files) with Imagemagick and would like to use GNU Parallel to speed things up.

This is the code I use (processing 100 frames at a time to avoid running out of ram):

calcmethod1=mean;
allframes=(*.png)
cd out1

for (( i=0; i < "${#allframes[@]}" ; i+=100 )); do 
    convert "${allframes[@]:i:100}" -evaluate-sequence "$calcmethod1" \
        -channel RGB -normalize ../out2/"${allframes[i]}"
done

how would I 'parallelize' this? Most solutions I've found work with not using a loop but piping - but doing this I've run into the problem that my script would break because of my arguments list getting too long...

I guess what I would want to do is to have parallel splitting the load like handing the first 100 frames to core 1, frames 100-199 to core 2 etc.?

1
  • Does the order matter when you're calling convert with the allframes? The *.png gives the glob in a specific order that may not align w/ your intent, if order matters.
    – slm
    Commented Jul 25, 2018 at 4:38

3 Answers 3

3

Order

Your sample program did not seem to care about the order of the *.png for the allframes array that you were constructing, but your comments led me to believe that order would matter.

I guess what I would want to do is to have parallel splitting the load like handing the first 100 frames to core 1, frames 100-199 to core 2 etc.?

Bash

Therefore I'd start with a modification to your script like so, changing the construction of the allframes array so that the files are stored in numeric order.

allframes=($(printf "%s\n" *.png | sort -V | tr '\n' ' '))

This can be simplified further to this using sort -zV:

allframes=($(printf "%s\0" *.png | sort -zV | tr '\0' ' '))

This has the effect on constructing your convert ... commands so that they look like this now:

$ convert "0.png 1.png 2.png 3.png 4.png 5.png 6.png 7.png 8.png 9.png \
          10.png 11.png 12.png 13.png 14.png 15.png 16.png 17.png 18.png \
          19.png 20.png 21.png 22.png 23.png 24.png 25.png 26.png 27.png \
          28.png 29.png 30.png 31.png 32.png 33.png 34.png 35.png 36.png \
          37.png 38.png 39.png 40.png 41.png 42.png 43.png 44.png 45.png \
          46.png 47.png 48.png 49.png 50.png 51.png 52.png 53.png 54.png \
          55.png 56.png 57.png 58.png 59.png 60.png 61.png 62.png 63.png \
          64.png 65.png 66.png 67.png 68.png 69.png 70.png 71.png 72.png \
          73.png 74.png 75.png 76.png 77.png 78.png 79.png 80.png 81.png \
          82.png 83.png 84.png 85.png 86.png 87.png 88.png 89.png 90.png \
          91.png 92.png 93.png 94.png 95.png 96.png 97.png 98.png 99.png" \
          -evaluate-sequence "mean" -channel RGB -normalize ../out2/0.png

Parallels

Building off of eschwartz's example I put together a parallel example as follows:

$ printf '%s\n' *.png | sort -V | parallel -n100 --dryrun convert {} \
   -evaluate-sequence 'mean' -channel RGB -normalize ../out2/{1}

again, more simply using sort -zV:

$ printf '%s\0' *.png | sort -zV | parallel -0 -n100 --dryrun "convert {} \
   -evaluate-sequence 'mean' -channel RGB -normalize ../out2/{1}

NOTE: The above has an echo "..." as the parallel action to start. Doing it this way helps to visualize what's happening:

$ convert 0.png 1.png 2.png 3.png 4.png 5.png 6.png 7.png 8.png 9.png 10.png \
         11.png 12.png 13.png 14.png 15.png 16.png 17.png 18.png 19.png \
         20.png 21.png 22.png 23.png 24.png 25.png 26.png 27.png 28.png \
         29.png 30.png 31.png 32.png 33.png 34.png 35.png 36.png 37.png \
         38.png 39.png 40.png 41.png 42.png 43.png 44.png 45.png 46.png \
         47.png 48.png 49.png 50.png 51.png 52.png 53.png 54.png 55.png \ 
         56.png 57.png 58.png 59.png 60.png 61.png 62.png 63.png 64.png \ 
         65.png 66.png 67.png 68.png 69.png 70.png 71.png 72.png 73.png \ 
         74.png 75.png 76.png 77.png 78.png 79.png 80.png 81.png 82.png \
         83.png 84.png 85.png 86.png 87.png 88.png 89.png 90.png 91.png \
         92.png 93.png 94.png 95.png 96.png 97.png 98.png 99.png \
         -evaluate-sequence mean -channel RGB -normalize ../out2/0.png

If you're satisfied with this output, simply remove the --dryrun switch to parallel, and rerun it.

$ printf '%s\0' *.png | sort -zV | parallel -0 -n100 convert {} \ 
    -evaluate-sequence 'mean' -channel RGB -normalize

References

1
  • 1
    that does do the trick and is exactly, what i was looking for! thank you and eschwartz so much!!! interestingly, parallel does not speed up the process much when using all (8) threads, the best performance i get with using only two threads (parallel -j 2)... which does boost speed almost to the double though! i'm a happy camper now... Commented Jul 25, 2018 at 21:48
3

The correct solution is to print the filenames using a shell builtin like printf '%s\0' *.png which is not vulnerable to the ARG_MAX limitation on command-line argument length, and then pipe it to parallel --null which will read those filenames and batch the jobs however you wish.

Some features of parallel which we will use:

  • --null is required to sanely split filenames on null characters to prevent weird issues with weird filenames
  • -n 100 will, just like xargs, handle 100 files for each invocation
  • {} contains those 100 filenames
  • ../out2/{1} contains just the first one

So, this would become:

calcmethod1=mean
printf '%s\0' *.png | parallel --null -n 100 convert {} -evaluate-sequence $calcmethod1 -channel RGB -normalize {} ../out2/{1}

Why do you think piping would not work? Piping works fine, it is only externally forked commands that do not read from a pipe, which have issues with argument length. Piping is in fact the whole purpose of parallel.

9
  • thank you for your help!!! after some reading i do have a vague idea of what you are suggesting... but you would not happen to have some code illustrating this, would you? i manage to build this argument with xargs like this ls *.png | xargs -n 100 sh -c 'convert "$0" "$@" -evaluate-sequence $calcmethod1 ../out2/"$0"' (which produces the arg_max problem though) but how would i translate this to something starting with parallel --null?!? Commented Jul 24, 2018 at 21:05
  • ls is not a shell builtin, it is an externally forked command and therefore you get ARG_MAX errors from ls before the pipeline ever takes effect. Also relevant: don't use ls in scripts since it is a tool meant only for interactive use.
    – eschwartz
    Commented Jul 24, 2018 at 21:18
  • um, i'm afraid this doesn't work in this case. it processes the images individually - whereas i need to hand the convert process 100 images at a time to composite these together - this is what the xargs -n 100 is for and what i do not manage to translate into parallel... Commented Jul 24, 2018 at 22:04
  • parallel will default to one job per CPU core, that is why there is the --jobs N option, so I guess parallel --null --jobs 100 convert ... would do the trick... this is described in the manpage though.
    – eschwartz
    Commented Jul 24, 2018 at 22:13
  • This determines the number of processes spawned but does not get me anywhere closer, I'm afraid. Every time I execute this convert command, I need to hand it the next 100 files in my list - this is one operation. So what i need parallel to do is something like "convert these 100 pngs here using the first thread, the next 100 pngs on the second thread, etc."... In any case: Thank you so much for helping me with this!!! Commented Jul 24, 2018 at 22:25
2

It's possible to run every convert process in its own subshell:

#!/bin/bash

for (( i=1; i<=1000; i++ )) do
(
command --options ) &
disown
done

exit 0

To see how it works, try this script:

#!/bin/bash

echo "Hi!"

for (( i=1; i<=1000; i++ )) do
(
sleep 30
echo "Bye, "$i"!" ) &
disown
done

exit 0

Assign to the script name par.sh and check processes afterwards:

ps aux | grep par.sh

We can assume native Linux CPU load balancer should spread processes between CPU cores evenly since every subshell has a separated pid. Anyway, something like cpuset is always available to be used.

1
  • thanks for your solution bob! i tried your approach and it certainly does spread the operation in different processes - only would it spawn a process for each individual image, freezing up... i guess for a skilled person (= not me), it's probably easy to build these processes in batches of 100 as outlined by slm but i did not test this... Commented Jul 25, 2018 at 21:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .