Delete Entries in File if the Same Entry Exists in a Master File

Question

I have a JSON master config file whose value may be overwritten by a specific account's config file (also in JSON). The Master file has this structure:

{
    "section1Configs": {
        "setting01": true,
        "setting02": true,
        "setting03": false
    },
    section2Configs: {
        "setting01": true,
        "setting02": true,
        "setting03": false
    },
    section3Configs: {
        "setting01": true,
        "setting02": true,
        "setting03": false
    },
    section4Configs: {
        "setting01": true,
        "setting02": true,
        "setting03": false
    }
}

The config file for a specific account may look like this:

{
    "section1Configs": {
        "setting01": true,
        "setting02": true,
        "setting03": true
    },
    section2Configs: {
        "setting01": false,
        "setting02": true,
        "setting03": false
    },
    section3Configs: {
        "setting01": true,
        "setting02": false,
        "setting03": false
    },
    section4Configs: {
        "setting01": true,
        "setting02": true,
        "setting03": false
    }
}

Note that they are identical except certain values (section01Config.setting03, section02Config.setting01, and section03Config.setting02) are different. Note also that the entire section4Configs block is the same in both files.

The ones that are the same are not needed since the application loads both and overwrites the Master file with the ones that are different in the account config.

What I would like to do is have a script that iterates through a directory of such account files and deletes the entry in each file that are the same key and value as the Master file. From this example I would end up with a file like this:

{
    section1Configs: {
        setting03: true
    },
    section2Configs: {
        setting01: false
    },
    section3Configs: {
        setting02: false
    }
}

There are over 200 account config files and doing this manually will take a lifetime. Any help is greatly appreciated!

------ EDIT -------

I tried to keep the question simple but I think a missing detail will have an impact on the answer.

The account and config JSON file consist of sections and their corresponding settings. This means that the JSON is one level deeper than what's mentioned above:

{
    "section1Configs": {
        "level1Settings": {
            "section01Lev01_01": true,
            "section01Lev01_012": true,
            "section01Lev01_02": true
        }
    }
}

Thanks!

Answer 1

So I copied your bottom thing into two files.

File 1:

{
    "section1Configs": {
        "level1Settings": {
            "section01Lev01_01": true,
            "section01Lev01_012": true,
            "section01Lev01_02": true
        }
    }
}

And File 2...

{
    "section1Configs": {
        "level1Settings": {
            "section01Lev01_01": true,
            "section01Lev01_012": false,
            "section01Lev01_02": true
        }
    }
}

They only differ on the Level01_012 line.

Then I did:

grep -Ev '[{}]|^$' file1 | grep -Fxvf- file2

---

{
    "section1Configs": {
        "level1Settings": {
            "section01Lev01_012": false,
        }
    }
}

That command is broken down as follows:

1. grep -Ev '[{}]|^$' file1

   • Here we ask grep to please print every line in file1 which does not match either a { character or a } character, but it must match at least one character.

   • Its output looks like:

             "section01Lev01_01": true,
             "section01Lev01_012": true,
             "section01Lev01_02": true
     

2. grep -Fxvf- file2

   • That is read by this second grep which interprets its - stdin input pattern -file as a -Fixed-string pattern -x entire-line match which it should -v not print to output. And so this second grep prints every line in file2 which doesn't exactly match the true/false lines in file1.

---

Now if your grep does not understand - as stdin we have a few options:

grep -Ev '[{}]|^$' file1 | grep -Fxvf /dev/fd/0 file2

...will work on most systems. A similar option is to use /dev/stdin, which is also pretty likely to work, if a little less so.

There is also a shell-specific implementation know as process-substitution which works in just about exactly the same way...

grep -Fxvf <(grep -Ev '[{}]|^$' file1) file2

...which may or may not work for you depending on your shell.

Another possibility is...

grep -Fxv "$(grep -Ev '[{}]|^$' file1)" file2

...which may or may not work for you depending on how large the output of the command-subbed grep is.

Try a few and probably you'll find one that works for you.

Answer 2

May be better to use tools specialized for json like jq or jshon?
But if you'd like it can be done with paste and sed

paste config master | sed '/[{}]/! {/\(.\+\)\t\1/d;};s/\t.*//'

or awk

awk '{getline a < "master"} /[{}]/ || $0 != a' config

for each line in config-file gets corresponding(by number) row from master-file into variable a and if line consist {or} or not the same with ones from master-file print it.
Result:

{
    "section1Configs": {
        "setting03": true
    },
    section2Configs: {
        "setting01": false,
    },
    section3Configs: {
        "setting02": false,
    },
    section4Configs: {
    }
}

Answer 3

As was pointed out elsewhere on this page, a robust and efficient way to solve the problem is to use a JSON-aware tool such as jq. The following provides a solution to the original problem; it can easily be adapted to solve variants.

(1) Put the following in a file, say minusConfig.jq:

def minus(o1;o2):
  o1 | with_entries( select(o2[.key] != .value) );

def minusConfig(o1;o2):
  reduce (o1|keys)[] as $key ({};
    minus( o1[$key] ; o2[$key]) as $v
    | if $v == {} then . else . + {($key): $v} end );

minusConfig($user; $master)

(2) Assuming the master and user config files are master.json and user.json respectively,

$ jq -n --arg master master.json --arg user user.json -f minusConfig.jq

produces:

{
  "section1Configs": {
    "setting03": true
  },
  "section2Configs": {
    "setting01": false
  },
  "section3Configs": {
    "setting02": false
  }
}