Remove large chunks from json using vim

Question

I have a huge file (1/2 million lines) of json.

I need to delete a set of entries that contain a certain string.

{
    "bla1": {
        "Part1": "Plop1",
        "Part2": "Plop2",
        "Part3": "BadFling1<stuff>",
        "part4": "Plop4",
    },
    "bla2": {
        "Part1": "Plop1",
        "Part2": "Plop2",
        "Part3": "<stuff>",
        "part4": "Plop4",
    },
    // etc for many more entries
}

All the entries have "BadFling1" as the prefix to the "Part3" entry.

I was wondering how the best way to automate removing all entries that include "BadFling1". So for example the result of removing the bad entry from above is:

{
    "bla2": {
        "Part1": "Plop1",
        "Part2": "Plop2",
        "Part3": "<stuff>",
        "part4": "Plop4",
    },
    // etc for many more entries
}

My first attempt was working but not fast enough (as it was slightly manual).

/BadFling1
qan3k5ddq
:map z n@a

Now hold down the 'z' key.

My vim foo is not strong enough, so I am unsure how to automate the processes better in vim. Any help appreciated.

Alternative approaches in bash (or other command line tools also welcome).

Answer 1

Try this in vim:

:g/BadFling/normal [{V]}d

The :global command runs a command on all lines that match the pattern (I used BadFling as an example - adjust it if necessary). The command being run in this case is the :normal command, which runs normal mode commands. The purpose of this is to use the power of the [{ and ]} vim movement commands which move between brace pairs. The Vd is combo is used to do a linewise delete. This is not as robust as a JSON parser but works assuming that each "blah1" part is contained within its own set of lines and so a linewise delete won't accidentally delete anything that belongs to another block. For example, the linewise delete approach would not work if you had something like

    ... end of block you want to keep
}, "blah1" : {
    block you want removed
}, "blah2" : {
    start of block you want to keep ...
}

Also, the [{ only uses the immediate parent block so if you have further levels of nesting it also won't work.

Answer 2

You can do this with grep and diff, if your version of diff is recent enough:

ire@localhost$ grep -B 3 -A 2 BadFling1 huge.json | diff --changed-group-format="%>" --unchanged-group-format="" - huge.json 
{
    "bla2": {
        "Part1": "Plop1",
        "Part2": "Plop2",
        "Part3": "<stuff>",
        "part4": "Plop4",
    },
    // etc for many more entries
}

The grep pulls out the bad records, by extracting the lines surrounding your match. The diff removes them from the original. As mentioned in the comments, this solution requires that the block sizes be consistent, and the matching line to be in the same place within each block (as in your example).

If that were not the case (varying record size, or unreliable positioning of record elements), I would take that as the cue to write a quick parsing script. You can easily and safely remove these records with only a few lines of Python, which has a built-in JSON parser.

Answer 3

Here's a solution in awk:

awk '/".*":\ {/             { open=line; skip_block=0 }
     /"Part3":\ "BadFling1/ { skip_block=1 }
     /},/                   { if (skip_block) { line=open; next } }
     { lines[line++]=$0 }
     END { for (i=0;i<=line;i++) { print lines[i] } }' yourfile > clean

This is not very well tested, but it should get you started. It will work even if the blocks are of variable length and does not care where in the block the disqualifying line is positioned.

Explanation:

line 1: if line matches the start of a block, note the position in the array, mark the block as good so far

line 2: if line matches the disqualifying line, and mark the block

line 3: match the end of a block. if the block is marked, reset the position in the array to where the block started, and skip to the next line

line 4: add the current line to array and increment line counter

line 5: when done reading file, print the array, containing only the "good" blocks

You can implement the same thing in bash, but awk will be a lot faster, and in my opinion this is the sort of things awk is built for, without having to pull out a "heavier" language.

Answer 4

using vim:

:%s/BadFling1//g

will search for all occurrences of "BadFling1" and replace it with "".