I'm trying to wrap my head around running parallel in parallel, and it seems like I have a situation where that would be an ideal solution.

I want to run a set of jobs in series - call them A-1, A-2, A-3 and so on. These would be run with --jobs 1 (or sem?).

I want to run sets of those in parallel - call them A, B, C, and so on. These would be run with the default number of jobs (cores).

The “A” sets of jobs may have a different number of jobs in them than the “B” sets of jobs; similarly for C or others.

Visually, where the horizontal axis is time and the vertical is job sets:


For this, let's assume all jobs are sleep $((RANDOM % 10)).

I assume there will have to be some sort of link (ala --link) between job sets and jobs - A with 1, 2 and 3; B with 1, 2, 3 and 4; C with 1 and 2; and D with just 1, using the above visual.

This may be a better example of what I was trying to do, using @ole-tang's solution

$ declare -fp apples bananas cherries dates
apples () 
    echo -n grannysmith fiji pinklady | parallel -d' ' -j1 'echo apples-{#}: {};sleep $((RANDOM % 3))'
declare -fx apples
bananas () 
    echo -n plantain cavadish red manzano | parallel -d' ' -j1 'echo bananas-{#}: {};sleep $((RANDOM % 3))'
declare -fx bananas
cherries () 
    echo -n sweet sour red yellow bing | parallel -d' ' -j1 'echo cherries-{#}: {};sleep $((RANDOM % 3))'
declare -fx cherries
dates () 
    echo -n medjool khola | parallel -d' ' -j1 'echo dates-{#}: {};sleep $((RANDOM % 3))'
declare -fx dates
$ parallel ::: apples bananas cherries dates
bananas-1: plantain
bananas-2: cavadish
bananas-3: red
bananas-4: manzano
dates-1: medjool
dates-2: khola
apples-1: grannysmith
apples-2: fiji
apples-3: pinklady
cherries-1: sweet
cherries-2: sour
cherries-3: red
cherries-4: yellow
cherries-5: bing

1 Answer 1

a() {
   seq 10 | parallel -j1 'echo A-{#};sleep $((RANDOM % 10))'
b() {
   seq 10 | parallel -j1 'echo B-{#};sleep $((RANDOM % 10))'
c() {
   seq 10 | parallel -j1 'echo C-{#};sleep $((RANDOM % 10))'
d() {
   seq 10 | parallel -j1 'echo D-{#};sleep $((RANDOM % 10))'
export -f a b c d

parallel ::: a b c d

If you want to see the output:

parallel --lb ::: a b c d

It can also be done:

doit() {
    seq 10 | parallel -j1 'echo '$1'-{};sleep $((RANDOM % 10))'
export -f doit
parallel --lb doit ::: A B C D
  • Using functions instead of parallel parallel makes it cleaner. Thanks!
    – Larry
    Commented May 30, 2023 at 18:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .