2

I have a JSON master config file whose value may be overwritten by a specific account's config file (also in JSON). The Master file has this structure:

{
    "section1Configs": {
        "setting01": true,
        "setting02": true,
        "setting03": false
    },
    section2Configs: {
        "setting01": true,
        "setting02": true,
        "setting03": false
    },
    section3Configs: {
        "setting01": true,
        "setting02": true,
        "setting03": false
    },
    section4Configs: {
        "setting01": true,
        "setting02": true,
        "setting03": false
    }
}

The config file for a specific account may look like this:

{
    "section1Configs": {
        "setting01": true,
        "setting02": true,
        "setting03": true
    },
    section2Configs: {
        "setting01": false,
        "setting02": true,
        "setting03": false
    },
    section3Configs: {
        "setting01": true,
        "setting02": false,
        "setting03": false
    },
    section4Configs: {
        "setting01": true,
        "setting02": true,
        "setting03": false
    }
}

Note that they are identical except certain values (section01Config.setting03, section02Config.setting01, and section03Config.setting02) are different. Note also that the entire section4Configs block is the same in both files.

The ones that are the same are not needed since the application loads both and overwrites the Master file with the ones that are different in the account config.

What I would like to do is have a script that iterates through a directory of such account files and deletes the entry in each file that are the same key and value as the Master file. From this example I would end up with a file like this:

{
    section1Configs: {
        setting03: true
    },
    section2Configs: {
        setting01: false
    },
    section3Configs: {
        setting02: false
    }
}

There are over 200 account config files and doing this manually will take a lifetime. Any help is greatly appreciated!

------ EDIT -------

I tried to keep the question simple but I think a missing detail will have an impact on the answer.

The account and config JSON file consist of sections and their corresponding settings. This means that the JSON is one level deeper than what's mentioned above:

{
    "section1Configs": {
        "level1Settings": {
            "section01Lev01_01": true,
            "section01Lev01_012": true,
            "section01Lev01_02": true
        }
    }
}

Thanks!

0

So I copied your bottom thing into two files.

File 1:

{
    "section1Configs": {
        "level1Settings": {
            "section01Lev01_01": true,
            "section01Lev01_012": true,
            "section01Lev01_02": true
        }
    }
}

And File 2...

{
    "section1Configs": {
        "level1Settings": {
            "section01Lev01_01": true,
            "section01Lev01_012": false,
            "section01Lev01_02": true
        }
    }
}

They only differ on the Level01_012 line.

Then I did:

grep -Ev '[{}]|^$' file1 | grep -Fxvf- file2

{
    "section1Configs": {
        "level1Settings": {
            "section01Lev01_012": false,
        }
    }
}

That command is broken down as follows:

  1. grep -Ev '[{}]|^$' file1

    • Here we ask grep to please print every line in file1 which does not match either a { character or a } character, but it must match at least one character.
    • Its output looks like:

              "section01Lev01_01": true,
              "section01Lev01_012": true,
              "section01Lev01_02": true
      
  2. grep -Fxvf- file2

    • That is read by this second grep which interprets its - stdin input pattern -file as a -Fixed-string pattern -x entire-line match which it should -v not print to output. And so this second grep prints every line in file2 which doesn't exactly match the true/false lines in file1.

Now if your grep does not understand - as stdin we have a few options:

grep -Ev '[{}]|^$' file1 | grep -Fxvf /dev/fd/0 file2

...will work on most systems. A similar option is to use /dev/stdin, which is also pretty likely to work, if a little less so.

There is also a shell-specific implementation know as process-substitution which works in just about exactly the same way...

grep -Fxvf <(grep -Ev '[{}]|^$' file1) file2

...which may or may not work for you depending on your shell.

Another possibility is...

grep -Fxv "$(grep -Ev '[{}]|^$' file1)" file2

...which may or may not work for you depending on how large the output of the command-subbed grep is.

Try a few and probably you'll find one that works for you.

  • the first part of this command works "grep -Ev '[{}]|^$' file1" but the other part "grep -Fxvf- file2" produces the following error: grep: -: No such file or directory. Running the first command with the second file works too so I'm not sure why it says "No such file..." – vdiaz1130 Jul 24 '15 at 19:16
  • Ok. So I'll change it. Most greps refer to - as stdin. Some do not. And so we can do something else. – mikeserv Jul 24 '15 at 19:17
  • @vdiaz1130 - please see the edit I have just made. It should cover most bases. – mikeserv Jul 24 '15 at 19:24
  • This works. Thanks! Just needs a little bit more work to get it to do what I need. Basically iterate through a directory of files. Also need to remove the comma at the end of this example. { "section1Configs": { "level1Settings": { "section01Lev01_02": false, } } } Thanks! – vdiaz1130 Jul 24 '15 at 19:40
  • @vdiaz130 - I don't know what this last thing means. – mikeserv Jul 24 '15 at 19:43
0

May be better to use tools specialized for json like jq or jshon?
But if you'd like it can be done with paste and sed

paste config master | sed '/[{}]/! {/\(.\+\)\t\1/d;};s/\t.*//'

or awk

awk '{getline a < "master"} /[{}]/ || $0 != a' config

for each line in config-file gets corresponding(by number) row from master-file into variable a and if line consist {or} or not the same with ones from master-file print it.
Result:

{
    "section1Configs": {
        "setting03": true
    },
    section2Configs: {
        "setting01": false,
    },
    section3Configs: {
        "setting02": false,
    },
    section4Configs: {
    }
}
  • paste acct.json master.json | sed '/[{}]/! {/(.\+)\t\1/d};s/\t.*//' produces the following error: sed: 1: "/[{}]/! {/(.\+)\t\1/d ...": extra characters at the end of d command – vdiaz1130 Jul 24 '15 at 16:53
  • Can you explain the awk command? I think this is working but the JSON object can be another level deep like { "section1Configs": { "level1Settings":{ "section01Lev01_01": true, "section01Lev01_012": true, "section01Lev01_02": true } } } – vdiaz1130 Jul 24 '15 at 17:20
  • @vdiaz1130 1) some sed implementations require ;before} 2)added – Costas Jul 24 '15 at 18:06
0

As was pointed out elsewhere on this page, a robust and efficient way to solve the problem is to use a JSON-aware tool such as jq. The following provides a solution to the original problem; it can easily be adapted to solve variants.

(1) Put the following in a file, say minusConfig.jq:

def minus(o1;o2):
  o1 | with_entries( select(o2[.key] != .value) );

def minusConfig(o1;o2):
  reduce (o1|keys)[] as $key ({};
    minus( o1[$key] ; o2[$key]) as $v
    | if $v == {} then . else . + {($key): $v} end );

minusConfig($user; $master)

(2) Assuming the master and user config files are master.json and user.json respectively,

$ jq -n --arg master master.json --arg user user.json -f minusConfig.jq

produces:

{
  "section1Configs": {
    "setting03": true
  },
  "section2Configs": {
    "setting01": false
  },
  "section3Configs": {
    "setting02": false
  }
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.