2

I have a large dataset (>200k files) that I would like to process (convert files into another format). The algorithm is mostly single-threaded, so it would be natural to use parallel processing. However, I want to do an unusual thing. Each file can be converted using one of two methods (CPU- and GPU-based), and I would like to utilize both CPU and GPU at the same time.

Speaking abstractly, I have two different commands (foo and bar), which are supposed to produce equivalent results. I would like to organize two thread pools with fixed capacity that would run up to N instances of foo and M instances of bar respectively, and process each input file with either of those pools depending on which one has free slots (determinism is not required or desired).

Is it possible to do something like that in GNU parallel or with any other tool?

1

Something like this:

gpus=2

find files |
  parallel -j +$gpus '{= $_ = slot() > '$gpus' ? "foo" : "bar" =}' {}

Less scary:

  parallel -j +$gpus '{=
      if(slot() > '$gpus') {
        $_ = "foo"
      } else {
        $_ = "bar"
      }
    =}' {}

-j +$gpus Run one job per CPU thread + $gpus

{= ... =} Use perl code to set $_.

slot() Job slot number (1..cpu_threads+$gpus).

1
  • Thanks! I forgot that parallel is perl-scriptable. Looks scary but does exactly what I need. – intelfx Sep 12 '20 at 22:07
0
#!/usr/bin/env bash

values=( a b c d e f g h i j k l m n o p q r s t u v w x y z)
start=0

function foo()
{
        sleep 4
        echo "foo ${1}"
}

function bar()
{
        sleep 4
        echo "bar ${1}"
}

function baz()
{
        sleep 4
        echo "baz ${1}"
}

while [ ${#values[@]} -gt 0 ]
do 
        if [ ${#values[@]} -gt 0 ]
        then
                foo "${values[0]}" &
                values=("${values[@]:1}")
        fi

        if [ ${#values[@]} -gt 0 ]
        then
                bar "${values[0]}" &
                values=("${values[@]:1}")
        fi

        if [ ${#values[@]} -gt 0 ]
        then
                baz "${values[0]}" &
                values=("${values[@]:1}")
        fi

        wait
done

values values=( a b c d e f g h i j k l m n o p q r s t u v w x y z) you can replace by values=$(ls -1)

5
  • Interesting. But I'm not seeing any parallelization and it's not immediately obvious how to implement that short of copy-pasting. Let's say I want to launch up to 32 instances of foo simultaneously and up to 8 instances of bar simultaneously, how would I do that? – intelfx Sep 12 '20 at 7:42
  • Inside loop as for i in {1..32}; do ...; done where ... is "if" with foo and second one for i in {1..8} with "if" bar function, all process are going background, so it is pseudo parallelization, but works fine – Marek Lisiecki Sep 12 '20 at 9:26
  • Also this can be written as one liner ;) if you want, also... You can do "copy paste" version for shell (here bash) – Marek Lisiecki Sep 12 '20 at 9:35
  • Also as I think it should be possible to run foo and bar loops independent now main loop waiting for finish all background jobs – Marek Lisiecki Sep 12 '20 at 9:45
  • Ah, well, argumentless wait waits for all children. I see. This should superficially work, but doesn't really suit my problem because foo and bar have significantly different (and variable) performance. I'm afraid this won't be too efficient. – intelfx Sep 12 '20 at 9:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.