0

I'm trying to make a bash script that loops through a file that contains a list of files (so it goes through each one of these files). While the files are open, I want to use another program (jq) to extract the values from the key value pair of the JSON files. Here's what I'm working with so far, kept pretty basic so I can try to figure out where I'm going wrong. I'm not concerned about the jq command, I've had it working where I am able to loop through all the files and pull out the values of a single key, but this ends up being extremely tedious because I have to manually change it for each key.

TL;DR: 'master' is the file that contains the full path of each one of the JSON files I want to open. 'key.txt' is a file that contains all of the possible keys within these JSON files. $KEY is the syntax for jq to search through the open file for that specific key and return the value of that key. I want to open up a file in the master list, loop through all the possible keys, and return the values of these keys into a text file called 'list.txt'.

Any help would be greatly appreciated.

#!/bin/bash

master="path/masterlist.txt"
while read master
do
        for KEY in 'key.txt'
        do
                jq '. .'$KEY'' $f
                echo $KEY
        done >> list.txt
        echo $master
done < list.txt
  • In your script, master is not a file; it is a variable. This does not match the prose of your question. Are you sure your script is trying to do what you describe? – DopeGhoti Feb 6 '17 at 17:48
  • Would it make more sense for master to be a variable? Bash syntax isn't my strongest, that's for sure. I want to be able to open the files in 'master' and do something with them while they're open. – user131935 Feb 6 '17 at 17:49
  • I was primarily pointing out that what you describe to be what you're doing and the script you present don't match, which makes it difficult to know what your actual goal is. – DopeGhoti Feb 6 '17 at 17:55
  • The actual goal is what I described. I would then assume the script isn't working because it isn't doing what I described. – user131935 Feb 6 '17 at 17:58
1

After reading what it is you say you want to do:

master="path/masterlist.txt"

while read json_path; do
    while read key; do
        printf 'File="%s", Key="%s"\n' "$json_path" "$key"
        jq ". .'$key'" "$json_path"
    done <key.txt
done <"$master" >list.txt

The list of JSON files are read from $master and each path is assigned to json_path in turn.

In the inner loop, the keys are read from key.txt and assigned to key in turn. The jq utility is invoked on $json_path with the argument . .'$key' where $key is expanded with the value of the current key.

All output is put into list.txt.

There might be some optimization to be done with the inner loop. It seem unnecessary to read all the keys in every single iteration of the outer loop, for example.


Annotated version of your script:

master="path/masterlist.txt"        # this is never used
while read master                   # this will set $master to something from list.txt
do
        for KEY in 'key.txt'        # $KEY will be the string 'key.txt' (nothing else)
        do
                jq '. .'$KEY'' $f   # $f is empty
                echo $KEY
        done >> list.txt            # better to >list.txt after "done" of outer loop
        echo $master
done < list.txt                     # you probably don't want to read from this file
0

From reading the script you posted, I wouldn't think that $f is being assigned properly. Also, the quoting in the jq command may be causing a problem. And then there's the fact that you are attempting to read a bare word master in a while loop, which may also be trying to read in list.txt at the same time as it's trying to append out to list.txt inside the loop.

#!/bin/bash

master="path/masterlist.txt"

cat $master | while read f; do
    for KEY in 'key.txt'; do
        echo $(jq ". .'$KEY'" $f) >> list.txt
    done
    echo $master
done

This will take each line from the $master file, and assign that into the variable $f in the loop, and run the jq query, appending the output to list.txt.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.