2

I have a large dataset (>200k files) that I would like to process (convert files into another format). The algorithm is mostly single-threaded, so it would be natural to use parallel processing. However, I want to do an unusual thing. Each file can be converted using one of two methods (CPU- and GPU-based), and I would like to utilize both CPU and GPU at the same time.

Speaking abstractly, I have two different commands (foo and bar), which are supposed to produce equivalent results. I would like to organize two thread pools with fixed capacity that would run up to N instances of foo and M instances of bar respectively, and process each input file with either of those pools depending on which one has free slots (determinism is not required or desired).

Is it possible to do something like that in GNU parallel or with any other tool?

2 Answers 2

1

Something like this:

gpus=2

find files |
  parallel -j +$gpus '{= $_ = slot() > '$gpus' ? "foo" : "bar" =}' {}

Less scary:

  parallel -j +$gpus '{=
      if(slot() > '$gpus') {
        $_ = "foo"
      } else {
        $_ = "bar"
      }
    =}' {}

-j +$gpus Run one job per CPU thread + $gpus

{= ... =} Use perl code to set $_.

slot() Job slot number (1..cpu_threads+$gpus).

1
  • Thanks! I forgot that parallel is perl-scriptable. Looks scary but does exactly what I need.
    – intelfx
    Sep 12, 2020 at 22:07
0
#!/usr/bin/env bash

values=( a b c d e f g h i j k l m n o p q r s t u v w x y z)
start=0

function foo()
{
        sleep 4
        echo "foo ${1}"
}

function bar()
{
        sleep 4
        echo "bar ${1}"
}

function baz()
{
        sleep 4
        echo "baz ${1}"
}

while [ ${#values[@]} -gt 0 ]
do 
        if [ ${#values[@]} -gt 0 ]
        then
                foo "${values[0]}" &
                values=("${values[@]:1}")
        fi

        if [ ${#values[@]} -gt 0 ]
        then
                bar "${values[0]}" &
                values=("${values[@]:1}")
        fi

        if [ ${#values[@]} -gt 0 ]
        then
                baz "${values[0]}" &
                values=("${values[@]:1}")
        fi

        wait
done

values values=( a b c d e f g h i j k l m n o p q r s t u v w x y z) you can replace by values=$(ls -1)

5
  • Interesting. But I'm not seeing any parallelization and it's not immediately obvious how to implement that short of copy-pasting. Let's say I want to launch up to 32 instances of foo simultaneously and up to 8 instances of bar simultaneously, how would I do that?
    – intelfx
    Sep 12, 2020 at 7:42
  • Inside loop as for i in {1..32}; do ...; done where ... is "if" with foo and second one for i in {1..8} with "if" bar function, all process are going background, so it is pseudo parallelization, but works fine Sep 12, 2020 at 9:26
  • Also this can be written as one liner ;) if you want, also... You can do "copy paste" version for shell (here bash) Sep 12, 2020 at 9:35
  • Also as I think it should be possible to run foo and bar loops independent now main loop waiting for finish all background jobs Sep 12, 2020 at 9:45
  • Ah, well, argumentless wait waits for all children. I see. This should superficially work, but doesn't really suit my problem because foo and bar have significantly different (and variable) performance. I'm afraid this won't be too efficient.
    – intelfx
    Sep 12, 2020 at 9:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .