Shell parallel processing: appending values

Question

I am trying to understand parallel processing in shell scripting and sequentially appending values deterministically (no random order) in the output through a simple example. Below is the code snippet:

x=""
appendnum() {
    num=$1; x=`echo $x$num`
}
for no in {0..10}
do
    appendnum $no &
done
wait $(jobs -rp)
echo $x

The expected output is 012345678910, but it’s resulting in a null value. I even tried it with iterating the PID to wait until it completes, but was unsuccessful. I want the main thread to wait till every parallel process completes. Appending number was just an example.

My problem statement looks like this: considering I have 3 tasks, I want list of responses like [responseof(task1),responseof(task2),responseof(task3)]. Count of tasks can be up to 50. Irrespective of number of tasks, my response time should be same. What is the most efficient and correct way of doing this?

Answer 1

If I understand you correctly you want to:

• Run the tasks in parallel, ie. ideally making sure that all of them complete in the time that it takes a single task to execute. (this is not realistic, but we can make a best effort)
• Keep the order of the output, even if some later tasks finish before some earlier tasks

With that in mind, you could try the following:

parallel -k -j10 'sleep {}; echo -n {}' ::: {10..1}

The first task executed takes the longest, but since we added the -k option the parallel utility will keep the order and eventually output

10987654321

without the -k option the output is reversed and comes as the commands finish

12345678910

Have a look at the tutorial if you need more info: https://www.gnu.org/software/parallel/parallel_tutorial.html

Answer 2

Some things you’re missing:

• Shell variables are stored in shell memory; i.e., the memory of the shell process.
• Most commands that you run from a shell are run in a child process (or processes).  The only exceptions are “built-in commands”.
• Asynchronous commands are always run in a child process — even if they don’t run any programs.  An asynchronous command that doesn’t run any programs is a child process that only runs the shell.  This is known as a “sub-shell”.
• Generally speaking, processes can’t change other processes’ memory.  In particular, sub-shells can’t modify variables in the main shell process.  So when you say appendnum $no &, the appendnum function cannot modify the x variable in the main shell process.

You can get something like the behavior you’re trying to get with this:

x=TR007.out
> "$x"
appendnum() {
    echo "$1" >> "$x"
}
for no in {0..10}
do
    appendnum $no &
done
wait

You will get the numbers 0 through 10 written to the file TR007.out.

• The scheduling (sequencing) of asynchronous processes is indeterminate.  Therefore, in the above example script, while you will get the numbers 0 through 10 written to the file, they might not be in order.
• As you might know, wait by itself (with no arguments) will wait for all child processes.
• “Irrespective of number of task, my response time should be same.”  That’s a very bold expectation / request.  Whether it is reasonable depends on context.  If the task is a single-threaded compute-intensive one, and you have three or more (logical) CPUs, then, yes, it may be reasonable to expect three tasks run in parallel to take little more time than one by itself.  But if you have four logical CPUs, it is totally unreasonable to expect to run 50 tasks in the same amount of time it takes to run one.
• I mentioned that the child (asynchronous) processes run in an expected order.  Since they are running concurrently (i.e., in parallel), their execution will likely overlap.  So, if we change the above script to do

  appendnum() {
      echo "$1"a >> "$x"
      echo "$1"b >> "$x"
  }
  for no in {1..3}
  do
      appendnum $no &
  done

  then you might get 1a/1b/2a/2b/3a/3b in the file — or you might get 2a/2b/1a/1b/3a/3b, or you might get 2a/1a/2b/3a/3b/1b, or worse.  Having asynchronous processes writing to the same file is a bad idea.

You should probably do something like

for no in {1..3}
do
    task"$no" > file"$no" &
done
wait
cat file1 file2 file3 > combined_result

Other notes:

• $(command) does the same thing as `command`.  You should stick with the $(command) form.
• It doesn’t make sense to say x=`echo $x$num` or x=$(echo $x$num).  Just say x="$x$num".
• You should always quote shell variables unless you have a good reason not to, and you’re sure you know what you’re doing.  So don’t do appendnum $no; do appendnum "$no", etc.

Answer 3

[responseof(task1),responseof(task2),responseof(task3)]

parset is built for that:

parset result responseof ::: task1 task2 task3
echo "${result[1]}"

E.g:

parset res seq ::: 3 2 1
echo "${res[1]}"

parset is part of GNU Parallel.