3

I have multiple "sets" of arguments for one command that I need to run in sequence (well, in parallel, technically). I also need to repeat the same logic after running each command.

#!/bin/bash

local pids=()

# Run commands in parallel.
my_command "$URL_ONE"   "$URL_ONE_TEXT"    "${TMP_DIR}/some_dir" &
pids+=("$!")

my_command "$URL_ONE"   "$URL_TWO_TEXT"    "${TMP_DIR}/some_similar_dir" &
pids+=("$!")

my_command "$URL_TWO"   "$URL_TWO_TEXT"    "${TMP_DIR}/third_dir" &
pids+=("$!")

my_command "$URL_THREE" "$URL_THREE_TEXT"  "${TMP_DIR}/fourth_dir" &
pids+=("$!")

# ...

# Wait for parallel commands to complete and exit if any fail.
for pid in "${pids[@]}"; do
    wait "$pid"
    if [[ $? -ne 0 ]]; then
        errecho "Failed."
        exit 1
    fi
done

Rather than repeating pids+=("$!") and other portions so frequently, what I'd like to do is define an array/set of arguments, loop through it, and execute the same logic for each set of arguments. For example:

#!/bin/bash

# This wouldn't actually work...
ARG_SETS=(
    ("$URL_ONE"   "$URL_ONE_TEXT"   "${TMP_DIR}/some_dir")
    ("$URL_ONE"   "$URL_TWO_TEXT"   "${TMP_DIR}/some_similar_dir")
    ("$URL_TWO"   "$URL_TWO_TEXT"   "${TMP_DIR}/third_dir")
    ("$URL_THREE" "$URL_THREE_TEXT" "${TMP_DIR}/fourth_dir")
)
for arg1 arg2 arg3 in "$ARG_SETS[@]"; do
    my_command "$arg1" "$arg2" "$arg3" &
    pids+=("$!")
done

But Bash does not support multidimensional arrays. Does anyone have any ideas for a good pattern to make this cleaner, or do something similar in design to my second example? Thanks!

3
  • 1
    I suggest you consider using python for this.
    – Ketan
    Commented Oct 11, 2015 at 3:38
  • I would love to. I think you can see how my mind jumps straight to a Python-ish paradigm in my second example ;) Unfortunately, this is part of a large system written in Bash and I can't really move it to Python at this time.
    – Will
    Commented Oct 11, 2015 at 3:39
  • 1
    A simpler solution in bash could be to have a set as a single string separated by special characters eg. "--" and then use awk to separate them before processing.
    – Ketan
    Commented Oct 11, 2015 at 3:43

2 Answers 2

2

This approach uses three arrays, one for each argument of the my_command:

pids=()

a=("$URL_ONE"               "$URL_ONE"                      "$URL_TWO"                  "$URL_THREE")
b=("$URL_ONE_TEXT"          "$URL_TWO_TEXT"                 "$URL_TWO_TEXT"         "$URL_THREE_TEXT")
c=("${TMP_DIR}/some_dir"    "${TMP_DIR}/some_similar_dir"   "${TMP_DIR}/third_dir"  "${TMP_DIR}/fourth_dir")

for i in ${!a[@]}
do
    my_command "${a[$i]}" "${b[$i]}" "${c[$i]}" &
    pids+=("$!")
done

# Wait for parallel commands to complete and exit if any fail.
for pid in "${pids[@]}"
do
    if ! wait "$pid"
    then
        errecho "Failed."
        exit 1
    fi
done

Alternate style

Depending on how many commands are to be run, you may want to consider the following alternative for defining the arrays:

a=();              b=();                   c=()
a+=("$URL_ONE");   b+=("$URL_ONE_TEXT");   c+=("${TMP_DIR}/some_dir")
a+=("$URL_ONE");   b+=("$URL_TWO_TEXT");   c+=("${TMP_DIR}/some_similar_dir")
a+=("$URL_TWO");   b+=("$URL_TWO_TEXT");   c+=("${TMP_DIR}/third_dir")
a+=("$URL_THREE"); b+=("$URL_THREE_TEXT"); c+=("${TMP_DIR}/fourth_dir")
1
  • Ah, the second style here is great. I was having an issue with too many commands and lines too long.
    – Will
    Commented Oct 11, 2015 at 6:23
2

I don't really see the need for arrays, and would go for something simple like:

f(){
  my_command "$@" &
  pids+=("$!")
}

f "$URL_ONE"   "$URL_ONE_TEXT"   "${TMP_DIR}/some_dir"
f "$URL_ONE"   "$URL_TWO_TEXT"   "${TMP_DIR}/some_similar_dir"
f "$URL_TWO"   "$URL_TWO_TEXT"   "${TMP_DIR}/third_dir"
f "$URL_THREE" "$URL_THREE_TEXT" "${TMP_DIR}/fourth_dir"
1
  • Yeah, I like this too. I want to avoid global variables though. But I guess I could just define the function within another function and declare $pids as a local inside the parent function.
    – Will
    Commented Oct 11, 2015 at 6:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .