5

I'm trying to set several environment variables with the results from command substitution. I want to run the commands in parallel with & and wait. What I've got currently looks something like

export foo=`somecommand bar` &
export fizz=`somecommand baz` &
export rick=`somecommand morty` &
wait

But apparently when using & variable assignments don't stick. So after the wait, all those variables are unassigned.

How can I assign these variables in parallel?

UPDATE: Here's what I ended up using based off the accepted answer

declare -a data
declare -a output
declare -a processes

var_names=(
    foo
    fizz
    rick
)

for name in "${var_names[@]}"
do
    processes+=("./get_me_a_value_for $name")
done

index=0
for process in "${processes[@]}"; do
    output+=("$(mktemp)")
    ${process} > ${output[$index]} &
    index=$((index+1))
done
wait

index=0
for out in "${output[@]}"; do
    val="$(<"${out}")"
    rm -f "${out}"

    export ${var_names[index]}="$val"

    index=$((index+1))
done

unset data
unset output
unset processes
  • 2
    What exactly would the benefit of this complication be, assuming it somehow actually worked? – thrig Apr 5 '17 at 22:11
  • 1
    If bar, baz, and morty take (say) ten seconds each to run, assigning the variables in parallel would take ten seconds, rather than thirty if run in series. – DopeGhoti Apr 5 '17 at 22:12
  • I've even tried such constructs as foo=$( ( sleep 3, echo bar ) & ) to work around the problem, but that just makes the assignment not return until the subshell has completed (which in retrospect makes sense). – DopeGhoti Apr 5 '17 at 22:15
6

After some ruminations, I came up with an ugly workaround:

#!/bin/bash
proc1=$(mktemp)
proc2=$(mktemp)
proc3=$(mktemp)

/path/to/longprocess1 > "$proc1" &
pid1=$!
/path/to/longprocess2 > "$proc2" &
pid2=$!
/path/to/longprocess3 > "$proc3" &
pid3=$!

wait "$pid1" "$pid2" "$pid3"
export var1="<("$proc1")"
export var2="<("$proc2")"
export var3="<("$proc3")"
rm -f "$proc1" "$proc2" "$proc3"

As requested in a comment, here is how to make this more extensible for an arbitrarily large list:

#!/bin/bash
declare -a pids
declare -a data
declare -a output
declare -a processes

# Generate the list of processes for demonstrative purposes
processes+=("/path/to/longprocess1")
processes+=("/path/to/longprocess2")
processes+=("/path/to/longprocess3")

index=0
for process in "${processes[@]}"; do
    output+=("$(mktemp")
    $process > ${output[$index]} &
    pids+=("$!")
    index=$((index+1))
done
wait ${pids[@]}
index=0
for process in "${processes[@]}"; do
    data+="$(<"${output[index]}")"
    rm -f "${output[index]}"
    index=$((index+1))
done
export data

The resultant output will be in the data array.

  • I think you mean export var1=$(<"$proc1") etc? I don't think you need to save the PID's of the children, doesn't wait wait for all of them if none are listed – ilkkachu Apr 5 '17 at 22:47
  • I tracked the PIDs and wait on them explicitly because it's possible there are other backgrounded tasks we don't need to wait for the completion of. – DopeGhoti Apr 6 '17 at 0:11
  • That worked with the suggestion from @ilkkachu. To add another constraint, is it possible to make this work for an arbitrary number of variables (such as a list being passed in and iterated over by a for loop)? – kag0 Apr 6 '17 at 18:52
  • Certainly, I will modify the answer to show that alternative. – DopeGhoti Apr 6 '17 at 19:28
  • @DopeGhoti awesome. This should be thread safe right? Also, I just had to make one change, ("<${output[index]}") had to be $(<"${output[index]}"). – kag0 Apr 7 '17 at 22:47
0

If you have more jobs than can safely be run in parallel at the same time, you can use GNU Parallel:

i="$(parallel --files ::: "${cmd[@]}" | perl -pe '$_="\"\`cat $_;rm $_\`\" "')"
eval my_result_array=($i)
unset i

Or in general:

parset() {
  local vname
  vname="$1"
  shift
  if [[ "$(declare -p $vname 2>/dev/null)" =~ "declare -a" ]]; then
      # vname refers to an array
      # The array elements refers to variable names to put output into
      eval $(
          parallel --files "$@" |
              perl -pe 'chop;$_="\"\`cat $_; rm $_\`\"\n"' |
              parallel echo {2}={1} :::: - :::+ $(eval echo '${'$vname'[@]}')
      )
  else
    # vname is not an existing array: Put output into that
    eval $vname="( $( parallel --files "$@" |
      perl -pe 'chop;$_="\"\`cat $_; rm $_\`\" "' ) )"
  fi
}

# Put output into vars $seq, $pwd, $ls
into_vars=(seq pwd ls)
parset into_vars ::: "seq 10" pwd ls
echo "$ls"

# Put output into aaa
unset aaa    
parset aaa seq 3 ::: 4 5 6
echo "${aaa[1]}"

# Output can contain spaces
parset out ::: "echo '<<joe  \"double  space\"  cartoon>>'" "pwd"
echo "${out[0]}"
echo "${out[1]}"

# The commands to run can be an array
cmd=("echo '<<joe  \"double  space\"  cartoon>>'" "pwd")
parset data ::: "${cmd[@]}"
echo "${data[0]}"
echo "${data[1]}"

# You cannot pipe into parset, but must use a tempfile
seq 10 > parallel_input
parset res echo :::: parallel_input
echo "${res[0]}"
echo "${res[9]}"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.