This is me revisiting an old answer after gaining a bit more experience with
The user in the question says
I'm not concerned about the jq command [...]
... but in fact, the
jq command is the key to making this work efficiently.
To recap: The user wants to extract all the values for all keys listed in some text file,
keys.txt. They want to do this for all JSON files listed in another file,
To get all (decoded) values for a set of keys,
c, from a set of files,
jq, you may do it like this:
jq -r '.[$ARGS.positional] // empty' file1 file2 file3 --args a b c
This passes the strings to be used as keys into
jq, which accepts them as elements in the array
$ARGS.positional. That array is expanded, and its elements are used as the keys for getting the values from the input. Here, I've chosen to return nothing (
empty) if the key's value does not exist or is
Adding the shell plumbing around this kernel of operation, assuming that both the list of files and the list of keys may be arbitrarily long and that the two files contain lines that are quoted as necessary:
xargs sh -c '
xargs jq -r ".[\$ARGS.positional] // empty" "$@" --args <keys.txt
' sh <list.txt >result.txt
This takes the list of filenames and calls a short inline shell script with batches of these. The inline script uses
xargs to call
jq with the filenames given (
"$@") and set of keys read from the key list.
The resulting list of values is written to
Old answer is below:
After reading what it is you say you want to do:
while read json_path; do
while read key; do
printf 'File="%s", Key="%s"\n' "$json_path" "$key"
jq ". .'$key'" "$json_path"
done <"$master" >list.txt
The list of JSON files are read from
$master and each path is assigned to
json_path in turn.
In the inner loop, the keys are read from
key.txt and assigned to
key in turn. The
jq utility is invoked on
$json_path with the argument
. .'$key' where
$key is expanded with the value of the current key.
All output is put into
There might be some optimization to be done with the inner loop. It seem unnecessary to read all the keys in every single iteration of the outer loop, for example.
Annotated version of your script:
master="path/masterlist.txt" # this is never used
while read master # this will set $master to something from list.txt
for KEY in 'key.txt' # $KEY will be the string 'key.txt' (nothing else)
jq '. .'$KEY'' $f # $f is empty
done >> list.txt # better to >list.txt after "done" of outer loop
done < list.txt # you probably don't want to read from this file