Process multiple inputs with multiple equivalent commands (multiple thread pools) in GNU parallel

Question

I have a large dataset (>200k files) that I would like to process (convert files into another format). The algorithm is mostly single-threaded, so it would be natural to use parallel processing. However, I want to do an unusual thing. Each file can be converted using one of two methods (CPU- and GPU-based), and I would like to utilize both CPU and GPU at the same time.

Speaking abstractly, I have two different commands (foo and bar), which are supposed to produce equivalent results. I would like to organize two thread pools with fixed capacity that would run up to N instances of foo and M instances of bar respectively, and process each input file with either of those pools depending on which one has free slots (determinism is not required or desired).

Is it possible to do something like that in GNU parallel or with any other tool?

Answer 1

Something like this:

gpus=2

find files |
  parallel -j +$gpus '{= $_ = slot() > '$gpus' ? "foo" : "bar" =}' {}

Less scary:

  parallel -j +$gpus '{=
      if(slot() > '$gpus') {
        $_ = "foo"
      } else {
        $_ = "bar"
      }
    =}' {}

-j +$gpus Run one job per CPU thread + $gpus

{= ... =} Use perl code to set $_.

slot() Job slot number (1..cpu_threads+$gpus).

Answer 2

#!/usr/bin/env bash

values=( a b c d e f g h i j k l m n o p q r s t u v w x y z)
start=0

function foo()
{
        sleep 4
        echo "foo ${1}"
}

function bar()
{
        sleep 4
        echo "bar ${1}"
}

function baz()
{
        sleep 4
        echo "baz ${1}"
}

while [ ${#values[@]} -gt 0 ]
do 
        if [ ${#values[@]} -gt 0 ]
        then
                foo "${values[0]}" &
                values=("${values[@]:1}")
        fi

        if [ ${#values[@]} -gt 0 ]
        then
                bar "${values[0]}" &
                values=("${values[@]:1}")
        fi

        if [ ${#values[@]} -gt 0 ]
        then
                baz "${values[0]}" &
                values=("${values[@]:1}")
        fi

        wait
done

values values=( a b c d e f g h i j k l m n o p q r s t u v w x y z) you can replace by values=$(ls -1)