Command to remove a portion of JSON data from each line?

Question

I have test.json file with different lengths of rows. Some fictitious example:

{ a: 123, b: sd, c: x45, d: 1, e: '' }
{ a: 5, b: bfgg, c: x4c, d: 31, e: '' }

I want to cut the whole substring after d - part and get back just for every line:

{ a: 123, b: sd, c: x45 }
{ a: 5, b: bfgg, c: x4c }

I found here a similar question and tried to adapt my problem to it:

echo test.json |  sed 's/. d:/' > newtest.json

I need to do it for the whole file, not only one line.

Answer 1

If you are doing much with transforming json on the command line, it is well worth your time to get the jq tool and learn to use it.

http://stedolan.github.io/jq/

While the answers above show that you can do minimal transformations without actually parsing the json, eventually you will either reinvent a buggy json parser via regexp or revert to using the native json parser in your language of choice.

jq is fast, simple to use and a very handy tool to have in your toolbox.

BWT, your test data is invalid json, which will complicate the solutions above. If you fix it to

{ "a":123 , "b": "sd", "c": "x45", "d": 1, "e": "" }
{ "a":5 , "b": "bfgg", "c": "x4c", "d": 31, "e": "" }

Then this jq command will do want you want

 jq -c '{a,b,c}' test.json

{"a":123,"b":"sd","c":"x45"}
{"a":5,"b":"bfgg","c":"x4c"}

Answer 2

sed '/d:/s/, d:[^}]*/ /' test.json

it will walk through the whole file and remove on each line with d: all , d:.* part till } symbol (} symbol will stay on the line).

Answer 3

@Rush's answer using sed is likely the best way to solve this but here's how you can do it using awk too:

$ awk -F ', d.* ' '{print $1, $2}' file.txt 
{ a: 123, b: sd, c: x45 }
{ a: 5, b: bfgg, c: x4c }

The above uses awk to split the data on , d.*. This produces 2 fields of data, $1 and $2 which contain the data resulting from AWK's splitting of the strings.

Answer 4

Through Perl which uses Positive lookahead assertion,

$ perl -pe 's/,\s+d:\s+.*?(?= })//' file
{ a: 123, b: sd, c: x45 }
{ a: 5, b: bfgg, c: x4c }

Through Python,

#!/usr/bin/python3
import sys
import re
file = sys.argv[1]
with open(file, 'r') as f:
    for line in f:
        line = re.sub(r',\s+d:\s+.*?(?= })', r'', line)
        print(line, end='')

Paste the above code in a file script.py and run the srcipt on your terminal by,

python3 script.py /path/to/the/input/file

Output:

{ a: 123, b: sd, c: x45 }
{ a: 5, b: bfgg, c: x4c }

Answer 5

Assuming you have a valid JSON file containing a set of objects, like

{"a":123,"b":"sd","c":"x45","d":1,"e":"''"}
{"a":5,"b":"bfgg","c":"x4c","d":31,"e":"''"}

or the equivalent

{
  "a": 123,
  "b": "sd",
  "c": "x45",
  "d": 1,
  "e": "''"
}
{
  "a": 5,
  "b": "bfgg",
  "c": "x4c",
  "d": 31,
  "e": "''"
}

and you'd like to remove the d and e keys from each object.

Using jq, deleting one key at a time:

jq -c 'del(.d) | del(.e)' file.json

Deleting both keys at once:

jq -c 'del(.d, .e)' file.json

The result of either of these would be

{"a":123,"b":"sd","c":"x45"}
{"a":5,"b":"bfgg","c":"x4c"}

A third way of doing this, which doesn't mention the actual keys by name, would be to convert the objects to lists of "entries" using to_entries, then delete the last two entries and convert the list back to a modified object:

jq -c 'to_entries | del(.[-2:]) | from_entries' file.json

This is most like what the text of the question proposes and the result depends on the ordering of the keys in the object.