GNU parallel export function output to variables failed

Question

This script is to determine when destination file already exist, source file will update the destination one or be removed according to flag "$dup_act".

#!/bin/bash
dup_chk()
{
  # $1: f_src, $2: f_dest, $3: dup_act (keep file u/pdate, l/arge)
  # check source file and destination file status 
  
  [[ "$3" = u && $(stat -c%Y "$1") -gt $(stat -c%Y "$2") || "$3" = l && $(stat -c%s "$1") -gt $(stat -c%s "$2") ]] \
  && echo -e "$1~$2~\n" >> mv_f.tmp || echo -e "$1\n" >> rm_f.tmp
  
  # mv_f.tmp: a list of source file replace destination one
  # rm_f.tmp: a list of source file to be removed
}

[[ -f mv_f.tmp ]] && rm mv_f.tmp ; [[ -f rm_f.tmp ]] && rm rm_f.tmp

dup_act=u # or dup_act=l

export dup_act
export -f dup_chk
cat dup_files.txt |  parallel -j10 --no-run-if-empty --colsep '~'  dup_chk {1} {2} "$dup_act"

The output files mv_f.tmp and rm_f.tmp are generated correctly.

Now, I want to the script output variables instead of files:

#!/bin/bash
dup_chk()
{
  # $1: f_src, $2: f_dest, $3: dup_act (keep file u/pdate, l/arge)
  # check source file and destination file status 
  
  [[ "$3" = u && $(stat -c%Y "$1") -gt $(stat -c%Y "$2") || "$3" = l && $(stat -c%s "$1") -gt $(stat -c%s "$2") ]] \
  && mv_f+="$1~$2~\n" || || rm_f+="$1\n"
  
  # mv_f: a variable of source file replace destination one
  # rm_f: a variable of source file to be removed
}

mv_f= ; rm_f= 

dup_act=u # or dup_act=l

export dup_act
export -f dup_chk
cat dup_files.txt |  parallel -j10 --no-run-if-empty --colsep '~'  dup_chk {1} {2} "$dup_act"
 

The result: $mv_f and $rm_f variables are empty.

I found out in other post: "Environment variables can only be passed from parent to child (as part of environment export/inheritance), not the other way around.". Is this the reason?

Please help. Thanks.

Answer 1

GNU Parallel spawns jobs in shells. Think of it as:

bash -c 'the job'

You cannot get variables back from that:

i=1
bash -c 'i=2'
# prints 1
echo $i

You add values to an array in the bash -c part and this will not make it back to the parent.

But you might be able to change your script so that you can use parset.

# NB: parset will not work correctly if reading from a pipe
parset myresults dup_chk < dup_files.txt

and then do some postprocessing on myresults.