I have the below bash script that I created with my yet very poor bash knowledge to batch convert JPG/PNG files to JPEG XL, the script so far works fine for my needs with no problems.

The only thing I could not solve was optimizing the part of the loop that is responsible for checking if the images have an ICC profile that is not compatible with "visually lossless" JPEG XL.

My initial idea was to combine Find and Parallel together with IF and ELSE but I didn't have any success, resulting only in a lot of syntax error outputs, so as an alternative method I opted for using loop, but in folders with several files the checking process is slow and sometimes takes longer than the conversion itself, so I ask, how can I can optimize this part of the script?


# create a copy of all folders and subfolder inside a path called jxl #
find . -type d -not -path "./jxl/*" -exec mkdir -p ./jxl/{} \; -exec mkdir -p ./jxl/icc/{} \;
rmdir ./jxl/jxl
rmdir ./jxl/icc/jxl

# move images with a NOT compatible icc profile to a directory called icc inside jxl path #
icc1="Device Model                    : "
icc2="Device Model                    : NONE"
icc3="Device Model                    : MS30"
shopt -s globstar
for f in **/*.jpg **/*.jpeg **/*.jpe **/*.png
   check=$(exiftool -devicemodel "$f")
   if [ "$check" = "$icc1" ] || [ "$check" = "$icc2" ] || [ "$check" = "$icc3" ]; then      
      echo "$f = icc profile NOT compatible"
      mv "$f" "$dir/$f"
      echo "$f = icc profile compatible"

# Run cjxl encoder e ignore all files inside the jxl folder
find ./ -type f \( -iname \*.jpg -o -iname \*.jpeg -o -iname \*.jpe -o -iname \*.png \) -not -path "./jxl/*" -print0 | parallel --jobs 8 -0 cjxl '{}' './jxl/{.}.jxl' -d 1 -e 7 -E 3 -I 1 --lossless_jpeg 0\;

# copy all files that are not a image to the jxl folder
find ./ -type f \( -iname \*.* ! -iname \*.jpg ! -iname \*.jpeg ! -iname \*.jpe ! -iname \*.png ! -iname \*.sh ! -iname \*.html \) -not -path "./jxl/*" -print0 | parallel --jobs 5 -0 mv '{}' './jxl/{}' \;

#delete all empty folders inside the jxl folder
find ./jxl -type d -empty -delete
  • Please edit: It's not very obvious what your shell script is and what the individual interactive commands are. Maybe also provide the number of files to process as well as timex timing info. Also note that executing I/O intensive tasks in parallel does not necessarily make total response time faster.
    – U. Windl
    Dec 12, 2022 at 7:47

2 Answers 2


The only really obvious improvement I can see is to avoid forking external programs multiple times. Whether this is worth the bother of "fixing" or not depends on how many times each command is forked by find or bash....if it's only a handful or tens of files, probably not. If it's hundreds or thousands, then definitely yes.

e.g. on the first find, you're running mkdir twice for each file found. You could optimise that by writing it as:

find . -type d -not -path "./jxl/*" \
  -exec bash -c 'for d; do
                   printf './jxl/%s\0./jxl/icc/%s\0' "$d" "$d";
                 done | xargs -0r mkdir -p ' bash {} +

This uses printf to send a NUL separated list of directories to create with xargs -0r mkdir -p.

Later in the script, in the for f in **/*.jpg **/*.jpeg **/*.jpe **/*.png loop, you're executing mv once for every file non- ICC-compatible file. You could optimise that by building an array containing the files to move and, again, using printf to send a NUL-separated list to xargs -0r mv -t "$dir/".

For example:

declare -a mvfiles=()

for f in **/*.jpg **/*.jpeg **/*.jpe **/*.png
   check=$(exiftool -devicemodel "$f")
   if [ "$check" = "$icc1" ] || [ "$check" = "$icc2" ] || [ "$check" = "$icc3" ]; then      
      echo "$f = icc profile NOT compatible"
      echo "$f = icc profile compatible"
printf '%s\0' "${mvfiles[@]}" | xargs -0r mv -t "$dir/"

Note this assumes you're using GNU mv with the -t option which allows you to specify the target directory before all the source pathnames. It's not worth doing with other versions of mv because you'd have to use xargs' -I option and that would run one mv per filename, defeating the purpose of doing this. Unless, of course, you're using FreeBSD's version of xargs with the -J option (which works like -I but allows multiple arguments per command).

You could do it without xargs, with just mv "${mvfiles[@]}" "$dir/" but then you'd be running the risk of exceeding ARG_MAX if you have many thousands of files. Using xargs avoids any risk of that.

BTW, I'm not sure how parallel handles {} (I mostly use xargs -P to parallelise tasks) but you might want to look into that. Given that you're using parallel to run 5 mv processes at once, it probably runs one command per argument. This might be exactly what you need when running cjxl (I don't have it installed and have no idea how it handles its filename args, so I can't comment) but for mv you're better off just using mv -t by piping into xargs -0r mv -t ./jxl/

Also worth noting: using parallel to run multiple processes at once may not provide the performance boost you expect. It depends on whether the processes are constrained by I/O or by available CPU power.

If they're not starved of I/O bandwidth then there'll be a huge performance boost running them in parallel.

If the processes are I/O-starved, they're all going to be idling most of the time while waiting for input, especially if they end up competing with each other for IO. So, experiment to find out the optimal number of cjxl jobs you can run without exceeding the available IO bandwidth of your hardware - 8 might be the right number, or it might be less. It could possibly even be more if you have more than 8 cores, but that's unlikely.

  • Thanks for your time, and for the super detailed answer, but there is still the problem of the Loop that is the biggest cause of slowness in this script for me, because I'm using Loop to check the ICC profile of a image the process is extremely slow since it checks only one file at a time, is it possible to perform this check in multiple files at the same time using another solution? something like parallel or xargs?
    – tchewinc
    Dec 11, 2022 at 20:17
  • Parallel execution is unlikely to help with exiftool -devicemodel as it is almost certainly I/O-bound. BTW, you must have a different version of exiftool, the version on my system (12.51) doesn't have a -devicemodel option. Does that option take multiple filename args? Can you extract the filename and device type from its output? can you make it produce NUL-separated output? Can you use something else instead (e.g. file, which can output manufacturer and model info from exif data AND has a -0 option for NUL-separated output)?
    – cas
    Dec 12, 2022 at 5:37

I will highly recommend that you change your script to include a single bash function which you call from GNU Parallel:

doit() {
   # Do all processing of a single file here
   # including if statements
export -f doit
find ... | parallel doit

A benefit from this is that you can test your function on a single file at a time.

Another benefit is that you may be mixing parts that are CPU and I/O intensive. So with a bit of luck one job will be using your CPU while another is using your disk.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .