Using parallel processes to speed up an interative bash loop, but I need to create an associative array inside the loop

Question

Using Bash, I have an indexed array that represents a list of files:

a=("1.json" "2.json" "3.json" ... "5309.json")

I am parsing two data fields from those JSON files into two associative arrays:

declare -A idArr
declare -A valueArr

for i in "${a[@]}"; do
    jqId="$(jq -M ".fileId" <"${i}")"
    jqValue="$(jq -M ".value" <"${i}")"
    # If there are already items in the associative array, add the new items separated by a newline
    idArr[${i}]="${idArr[${i}]}${idArr[${i}]:+$'\n'}${jqId}"
    valueArr[${jqId}]="${valueArr[${jqId}]}${valueArr[${jqId}]:+$'\n'}${jqValue}"
done

Since I'm iterating through one file at a time, it takes a considerable amount of time to process all the files. I require the associative arrays created within the loop to persist beyond its scope, even after the loop has finished.

Is there a method, such as using parallel processing or any other approach, that would allow me to concurrently process multiple array items and still enable them to contribute data to the associative arrays?

Answer 1

What's likely slow here is that you're running two jqs per file and forking a process and executing jq in it is likely orders of magnitude more work than processing a small json file.

read0() { IFS= read -rd '' "$@"; }
add_line() {
  typeset -n _var="$1"
  _var+=${_var:+$'\n'}$2
}
shopt -s extglob failglob lastpipe
set -o pipefail

typeset -A idArr valueArr

jq -j '[input_filename, .fileId, .value] |
       map(gsub("\u0000"; "") + "\u0000") | add
      ' -- +([0123456789]).json |
  while read0 file && read0 id && read0 value; do
    add_line "idArr[$file]"    "$id"
    add_line "valueArr[$file]" "$var"
  done

Would run jq only once. jq prints the filename, id and value (stripped of their NULs if any) NUL-delimited which bash reads in a loop in parallel.

Important note: do not call that add_line with arbitrary file names as that would be a command injection vulnerability as typeset -n is a bit of an eval in disguise. For instance, if you used *.json in place of that +([0123456789]).json, and there was a file called $(reboot).json, that would reboot!

In current versions of bash, that can be worked around by using single quotes instead of double-quotes around the idArr[$file], valueArr[$file], but that may not be future-proof as future versions of bash might decide to no longer do those expansions upon nameref dereferencing to avoid this kind of vulnerability.

Or you could remove those vulnerabilities by doing away with those mis-designed namerefs and using eval explicitly:

add_line() {
  eval "$1+=\${$1:+\$'\\n'}\$2"
}

And make sure to call it as:

    add_line 'idArr[$file]'    "$id"
    add_line 'valueArr[$file]' "$var"

Then you can use *.json or "${a[@]}" without the need to do some sanitisation¹.

If you run into a "argument list too long" error, replace jq ... +([0123456789]).json with printf '%s\0' +([0123456789]).json | xargs -r0 jq ....

While you may be tempted to also use GNU xargs' -P to run some of those jqs in parallel, I would advise against it as xargs doesn't guarantee the serialisation of the output of the commands so the outputs of the individual jqs would likely end-up intertwined. GNU parallel does but also has a large overhead compared to something as simple as parsing a (presumably) short JSON file so may not add much benefit.

---

^{¹ Well, jq treats - as meaning stdin, so strictly speaking, if there is file named like that in your $a array (or the expansion of a * glob instead of *.json), you'd need to transform it to ./-.}