3

I've found answers close to this but fail to understand how to use them in my case (I'm rather new to Bash)... so, I'm trying to process a folder containing a large image sequence (100k+ files) with Imagemagick and would like to use GNU Parallel to speed things up.

This is the code I use (processing 100 frames at a time to avoid running out of ram):

calcmethod1=mean;
allframes=(*.png)
cd out1

for (( i=0; i < "${#allframes[@]}" ; i+=100 )); do 
    convert "${allframes[@]:i:100}" -evaluate-sequence "$calcmethod1" \
        -channel RGB -normalize ../out2/"${allframes[i]}"
done

how would I 'parallelize' this? Most solutions I've found work with not using a loop but piping - but doing this I've run into the problem that my script would break because of my arguments list getting too long...

I guess what I would want to do is to have parallel splitting the load like handing the first 100 frames to core 1, frames 100-199 to core 2 etc.?

  • Does the order matter when you're calling convert with the allframes? The *.png gives the glob in a specific order that may not align w/ your intent, if order matters. – slm Jul 25 '18 at 4:38
3

Order

Your sample program did not seem to care about the order of the *.png for the allframes array that you were constructing, but your comments led me to believe that order would matter.

I guess what I would want to do is to have parallel splitting the load like handing the first 100 frames to core 1, frames 100-199 to core 2 etc.?

Bash

Therefore I'd start with a modification to your script like so, changing the construction of the allframes array so that the files are stored in numeric order.

allframes=($(printf "%s\n" *.png | sort -V | tr '\n' ' '))

This can be simplified further to this using sort -zV:

allframes=($(printf "%s\0" *.png | sort -zV | tr '\0' ' '))

This has the effect on constructing your convert ... commands so that they look like this now:

$ convert "0.png 1.png 2.png 3.png 4.png 5.png 6.png 7.png 8.png 9.png \
          10.png 11.png 12.png 13.png 14.png 15.png 16.png 17.png 18.png \
          19.png 20.png 21.png 22.png 23.png 24.png 25.png 26.png 27.png \
          28.png 29.png 30.png 31.png 32.png 33.png 34.png 35.png 36.png \
          37.png 38.png 39.png 40.png 41.png 42.png 43.png 44.png 45.png \
          46.png 47.png 48.png 49.png 50.png 51.png 52.png 53.png 54.png \
          55.png 56.png 57.png 58.png 59.png 60.png 61.png 62.png 63.png \
          64.png 65.png 66.png 67.png 68.png 69.png 70.png 71.png 72.png \
          73.png 74.png 75.png 76.png 77.png 78.png 79.png 80.png 81.png \
          82.png 83.png 84.png 85.png 86.png 87.png 88.png 89.png 90.png \
          91.png 92.png 93.png 94.png 95.png 96.png 97.png 98.png 99.png" \
          -evaluate-sequence "mean" -channel RGB -normalize ../out2/0.png

Parallels

Building off of eschwartz's example I put together a parallel example as follows:

$ printf '%s\n' *.png | sort -V | parallel -n100 --dryrun convert {} \
   -evaluate-sequence 'mean' -channel RGB -normalize ../out2/{1}

again, more simply using sort -zV:

$ printf '%s\0' *.png | sort -zV | parallel -0 -n100 --dryrun "convert {} \
   -evaluate-sequence 'mean' -channel RGB -normalize ../out2/{1}

NOTE: The above has an echo "..." as the parallel action to start. Doing it this way helps to visualize what's happening:

$ convert 0.png 1.png 2.png 3.png 4.png 5.png 6.png 7.png 8.png 9.png 10.png \
         11.png 12.png 13.png 14.png 15.png 16.png 17.png 18.png 19.png \
         20.png 21.png 22.png 23.png 24.png 25.png 26.png 27.png 28.png \
         29.png 30.png 31.png 32.png 33.png 34.png 35.png 36.png 37.png \
         38.png 39.png 40.png 41.png 42.png 43.png 44.png 45.png 46.png \
         47.png 48.png 49.png 50.png 51.png 52.png 53.png 54.png 55.png \ 
         56.png 57.png 58.png 59.png 60.png 61.png 62.png 63.png 64.png \ 
         65.png 66.png 67.png 68.png 69.png 70.png 71.png 72.png 73.png \ 
         74.png 75.png 76.png 77.png 78.png 79.png 80.png 81.png 82.png \
         83.png 84.png 85.png 86.png 87.png 88.png 89.png 90.png 91.png \
         92.png 93.png 94.png 95.png 96.png 97.png 98.png 99.png \
         -evaluate-sequence mean -channel RGB -normalize ../out2/0.png

If you're satisfied with this output, simply remove the --dryrun switch to parallel, and rerun it.

$ printf '%s\0' *.png | sort -zV | parallel -0 -n100 convert {} \ 
    -evaluate-sequence 'mean' -channel RGB -normalize

References

  • 1
    that does do the trick and is exactly, what i was looking for! thank you and eschwartz so much!!! interestingly, parallel does not speed up the process much when using all (8) threads, the best performance i get with using only two threads (parallel -j 2)... which does boost speed almost to the double though! i'm a happy camper now... – user3647558 Jul 25 '18 at 21:48
3

The correct solution is to print the filenames using a shell builtin like printf '%s\0' *.png which is not vulnerable to the ARG_MAX limitation on command-line argument length, and then pipe it to parallel --null which will read those filenames and batch the jobs however you wish.

Some features of parallel which we will use:

  • --null is required to sanely split filenames on null characters to prevent weird issues with weird filenames
  • -n 100 will, just like xargs, handle 100 files for each invocation
  • {} contains those 100 filenames
  • ../out2/{1} contains just the first one

So, this would become:

calcmethod1=mean
printf '%s\0' *.png | parallel --null -n 100 convert {} -evaluate-sequence $calcmethod1 -channel RGB -normalize {} ../out2/{1}

Why do you think piping would not work? Piping works fine, it is only externally forked commands that do not read from a pipe, which have issues with argument length. Piping is in fact the whole purpose of parallel.

  • thank you for your help!!! after some reading i do have a vague idea of what you are suggesting... but you would not happen to have some code illustrating this, would you? i manage to build this argument with xargs like this ls *.png | xargs -n 100 sh -c 'convert "$0" "$@" -evaluate-sequence $calcmethod1 ../out2/"$0"' (which produces the arg_max problem though) but how would i translate this to something starting with parallel --null?!? – user3647558 Jul 24 '18 at 21:05
  • ls is not a shell builtin, it is an externally forked command and therefore you get ARG_MAX errors from ls before the pipeline ever takes effect. Also relevant: don't use ls in scripts since it is a tool meant only for interactive use. – eschwartz Jul 24 '18 at 21:18
  • um, i'm afraid this doesn't work in this case. it processes the images individually - whereas i need to hand the convert process 100 images at a time to composite these together - this is what the xargs -n 100 is for and what i do not manage to translate into parallel... – user3647558 Jul 24 '18 at 22:04
  • parallel will default to one job per CPU core, that is why there is the --jobs N option, so I guess parallel --null --jobs 100 convert ... would do the trick... this is described in the manpage though. – eschwartz Jul 24 '18 at 22:13
  • This determines the number of processes spawned but does not get me anywhere closer, I'm afraid. Every time I execute this convert command, I need to hand it the next 100 files in my list - this is one operation. So what i need parallel to do is something like "convert these 100 pngs here using the first thread, the next 100 pngs on the second thread, etc."... In any case: Thank you so much for helping me with this!!! – user3647558 Jul 24 '18 at 22:25
2

It's possible to run every convert process in its own subshell:

#!/bin/bash

for (( i=1; i<=1000; i++ )) do
(
command --options ) &
disown
done

exit 0

To see how it works, try this script:

#!/bin/bash

echo "Hi!"

for (( i=1; i<=1000; i++ )) do
(
sleep 30
echo "Bye, "$i"!" ) &
disown
done

exit 0

Assign to the script name par.sh and check processes afterwards:

ps aux | grep par.sh

We can assume native Linux CPU load balancer should spread processes between CPU cores evenly since every subshell has a separated pid. Anyway, something like cpuset is always available to be used.

  • thanks for your solution bob! i tried your approach and it certainly does spread the operation in different processes - only would it spawn a process for each individual image, freezing up... i guess for a skilled person (= not me), it's probably easy to build these processes in batches of 100 as outlined by slm but i did not test this... – user3647558 Jul 25 '18 at 21:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.