3

I have multiple "sets" of arguments for one command that I need to run in sequence (well, in parallel, technically). I also need to repeat the same logic after running each command.

#!/bin/bash

local pids=()

# Run commands in parallel.
my_command "$URL_ONE"   "$URL_ONE_TEXT"    "${TMP_DIR}/some_dir" &
pids+=("$!")

my_command "$URL_ONE"   "$URL_TWO_TEXT"    "${TMP_DIR}/some_similar_dir" &
pids+=("$!")

my_command "$URL_TWO"   "$URL_TWO_TEXT"    "${TMP_DIR}/third_dir" &
pids+=("$!")

my_command "$URL_THREE" "$URL_THREE_TEXT"  "${TMP_DIR}/fourth_dir" &
pids+=("$!")

# ...

# Wait for parallel commands to complete and exit if any fail.
for pid in "${pids[@]}"; do
    wait "$pid"
    if [[ $? -ne 0 ]]; then
        errecho "Failed."
        exit 1
    fi
done

Rather than repeating pids+=("$!") and other portions so frequently, what I'd like to do is define an array/set of arguments, loop through it, and execute the same logic for each set of arguments. For example:

#!/bin/bash

# This wouldn't actually work...
ARG_SETS=(
    ("$URL_ONE"   "$URL_ONE_TEXT"   "${TMP_DIR}/some_dir")
    ("$URL_ONE"   "$URL_TWO_TEXT"   "${TMP_DIR}/some_similar_dir")
    ("$URL_TWO"   "$URL_TWO_TEXT"   "${TMP_DIR}/third_dir")
    ("$URL_THREE" "$URL_THREE_TEXT" "${TMP_DIR}/fourth_dir")
)
for arg1 arg2 arg3 in "$ARG_SETS[@]"; do
    my_command "$arg1" "$arg2" "$arg3" &
    pids+=("$!")
done

But Bash does not support multidimensional arrays. Does anyone have any ideas for a good pattern to make this cleaner, or do something similar in design to my second example? Thanks!

  • 1
    I suggest you consider using python for this. – mkc Oct 11 '15 at 3:38
  • I would love to. I think you can see how my mind jumps straight to a Python-ish paradigm in my second example ;) Unfortunately, this is part of a large system written in Bash and I can't really move it to Python at this time. – Will Oct 11 '15 at 3:39
  • 1
    A simpler solution in bash could be to have a set as a single string separated by special characters eg. "--" and then use awk to separate them before processing. – mkc Oct 11 '15 at 3:43
2

This approach uses three arrays, one for each argument of the my_command:

pids=()

a=("$URL_ONE"               "$URL_ONE"                      "$URL_TWO"                  "$URL_THREE")
b=("$URL_ONE_TEXT"          "$URL_TWO_TEXT"                 "$URL_TWO_TEXT"         "$URL_THREE_TEXT")
c=("${TMP_DIR}/some_dir"    "${TMP_DIR}/some_similar_dir"   "${TMP_DIR}/third_dir"  "${TMP_DIR}/fourth_dir")

for i in ${!a[@]}
do
    my_command "${a[$i]}" "${b[$i]}" "${c[$i]}" &
    pids+=("$!")
done

# Wait for parallel commands to complete and exit if any fail.
for pid in "${pids[@]}"
do
    if ! wait "$pid"
    then
        errecho "Failed."
        exit 1
    fi
done

Alternate style

Depending on how many commands are to be run, you may want to consider the following alternative for defining the arrays:

a=();              b=();                   c=()
a+=("$URL_ONE");   b+=("$URL_ONE_TEXT");   c+=("${TMP_DIR}/some_dir")
a+=("$URL_ONE");   b+=("$URL_TWO_TEXT");   c+=("${TMP_DIR}/some_similar_dir")
a+=("$URL_TWO");   b+=("$URL_TWO_TEXT");   c+=("${TMP_DIR}/third_dir")
a+=("$URL_THREE"); b+=("$URL_THREE_TEXT"); c+=("${TMP_DIR}/fourth_dir")
  • Ah, the second style here is great. I was having an issue with too many commands and lines too long. – Will Oct 11 '15 at 6:23
2

I don't really see the need for arrays, and would go for something simple like:

f(){
  my_command "$@" &
  pids+=("$!")
}

f "$URL_ONE"   "$URL_ONE_TEXT"   "${TMP_DIR}/some_dir"
f "$URL_ONE"   "$URL_TWO_TEXT"   "${TMP_DIR}/some_similar_dir"
f "$URL_TWO"   "$URL_TWO_TEXT"   "${TMP_DIR}/third_dir"
f "$URL_THREE" "$URL_THREE_TEXT" "${TMP_DIR}/fourth_dir"
  • Yeah, I like this too. I want to avoid global variables though. But I guess I could just define the function within another function and declare $pids as a local inside the parent function. – Will Oct 11 '15 at 6:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.