Using Bash, I have an indexed array that represents a list of files:

a=("1.json" "2.json" "3.json" ... "5309.json")

I am parsing two data fields from those JSON files into two associative arrays:

declare -A idArr
declare -A valueArr

for i in "${a[@]}"; do
    jqId="$(jq -M ".fileId" <"${i}")"
    jqValue="$(jq -M ".value" <"${i}")"
    # If there are already items in the associative array, add the new items separated by a newline

Since I'm iterating through one file at a time, it takes a considerable amount of time to process all the files. I require the associative arrays created within the loop to persist beyond its scope, even after the loop has finished.

Is there a method, such as using parallel processing or any other approach, that would allow me to concurrently process multiple array items and still enable them to contribute data to the associative arrays?

  • 1
    Shell scripts tend to have limitations in the ways data can be exchanged between a parent process and its children. The workarounds I've seen most often for this use case involve data storage that's external to the parent/child processes, and can handle simultaneous writes. The child processes write their results to per-child files in a subdirectory; or they append rows to a table in a SQL database; or they write keys to a key/value store like Redis. When the children are done, the parent reads the results from the data store and assembles them into the required data structures.
    – Sotto Voce
    Jun 28 at 4:17
  • Or in other words, shells don't tend to support threads with shared memory. Between processes, you could also use pipes from the children to the parent, but shells also tend to have a hard time waiting for data from multiple fds at the same time (i.e. with the select() system call or such, though AFAIK zsh does have support for it). Hence writing the assignments out to a file or several and reading them back from the parent might make most sense. Or consider using another programming language, one where you can handle the JSON directly instead of calling another process.
    – ilkkachu
    Jun 28 at 6:38

1 Answer 1


What's likely slow here is that you're running two jqs per file and forking a process and executing jq in it is likely orders of magnitude more work than processing a small json file.

read0() { IFS= read -rd '' "$@"; }
add_line() {
  typeset -n _var="$1"
shopt -s extglob failglob lastpipe
set -o pipefail

typeset -A idArr valueArr

jq -j '[input_filename, .fileId, .value] |
       map(gsub("\u0000"; "") + "\u0000") | add
      ' -- +([0123456789]).json |
  while read0 file && read0 id && read0 value; do
    add_line "idArr[$file]"    "$id"
    add_line "valueArr[$file]" "$var"

Would run jq only once. jq prints the filename, id and value (stripped of their NULs if any) NUL-delimited which bash reads in a loop in parallel.

Important note: do not call that add_line with arbitrary file names as that would be a command injection vulnerability as typeset -n is a bit of an eval in disguise. For instance, if you used *.json in place of that +([0123456789]).json, and there was a file called $(reboot).json, that would reboot!

In current versions of bash, that can be worked around by using single quotes instead of double-quotes around the idArr[$file], valueArr[$file], but that may not be future-proof as future versions of bash might decide to no longer do those expansions upon nameref dereferencing to avoid this kind of vulnerability.

Or you could remove those vulnerabilities by doing away with those mis-designed namerefs and using eval explicitly:

add_line() {
  eval "$1+=\${$1:+\$'\\n'}\$2"

And make sure to call it as:

    add_line 'idArr[$file]'    "$id"
    add_line 'valueArr[$file]' "$var"

Then you can use *.json or "${a[@]}" without the need to do some sanitisation¹.

If you run into a "argument list too long" error, replace jq ... +([0123456789]).json with printf '%s\0' +([0123456789]).json | xargs -r0 jq ....

While you may be tempted to also use GNU xargs' -P to run some of those jqs in parallel, I would advise against it as xargs doesn't guarantee the serialisation of the output of the commands so the outputs of the individual jqs would likely end-up intertwined. GNU parallel does but also has a large overhead compared to something as simple as parsing a (presumably) short JSON file so may not add much benefit.

¹ Well, jq treats - as meaning stdin, so strictly speaking, if there is file named like that in your $a array (or the expansion of a * glob instead of *.json), you'd need to transform it to ./-.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .