Convert text file to JSON

Question

Using curl, I get some statistical information from my TP-Link router. I put that information to a text file. However, I need this info in JSON format so that I can use it elsewhere. The format of the text file is like this:

[0,0,0,0,0,0]0
enable=1
interval=10
action=0
[1,0,0,0,0,0]1
ipAddress=3232235848
macAddress=EC:0E:C4:4F:XX:XX
totalPkts=201
totalBytes=22914
currPkts=0
currBytes=0
currIcmp=0
currUdp=0
currSyn=0
currIcmpMax=0
currUdpMax=15
currSynMax=0
[2,0,0,0,0,0]1
ipAddress=3232235829
macAddress=00:16:E8:22:XX:XX
totalPkts=972
totalBytes=98730
currPkts=0
currBytes=0
currIcmp=0
currUdp=0
currSyn=0
currIcmpMax=0
currUdpMax=14
currSynMax=4
[3,0,0,0,0,0]1
ipAddress=3232235842
macAddress=70:14:A6:8B:XX:XX
totalPkts=154566
totalBytes=15116490
currPkts=0
currBytes=0
currIcmp=0
currUdp=0
currSyn=0
currIcmpMax=9
currUdpMax=58
currSynMax=60

I need a JSON something like:

{"ipAddress" : "3232235848",
"macAddress" : "EC:0E:C4:4F:XX:XX",
"totalPkts" : "201",
"totalBytes" : "22914",
"currPkts" : "0",
"currBytes" : "0",
"currIcmp" : "0",
"currUdp" : "0",
"currSyn" : "0",
"currIcmpMax" : "0",
"currUdpMax" : "15",
"currSynMax" : "0"}
{"ipAddress" : "3232235829", ...

Could you show me the most efficient way of doing this? I prefer a bash script if it is possible? Tried putting "| jq -R -s -c 'split("\n")'" to the end of curl command but then it looks like: {"ipAddress=3232235848","macAddress=EC:0E:C4:4F:XX:XX" ...

Answer 1

sed hack:

sed -E -e 's/\[([0-9],){5}[0-9]\][0-9]+/}{/g; s/([[:alnum:]]+)=([[:alnum:]]+)/"\1" : "\2",/; 
           s/,\n}/}/;  1s/.*/{/; $s/.*/&}/' yourfile | sed -Ez 's/,\n?}/ }\n/g'

The output:

{
"enable" : "1",
"interval" : "10",
"action" : "0" }
{
"ipAddress" : "3232235848",
"macAddress" : "EC",:0E:C4:4F:XX:XX
"totalPkts" : "201",
"totalBytes" : "22914",
"currPkts" : "0",
"currBytes" : "0",
"currIcmp" : "0",
"currUdp" : "0",
"currSyn" : "0",
"currIcmpMax" : "0",
"currUdpMax" : "15",
"currSynMax" : "0" }
{
"ipAddress" : "3232235829",
"macAddress" : "00",:16:E8:22:XX:XX
"totalPkts" : "972",
"totalBytes" : "98730",
"currPkts" : "0",
"currBytes" : "0",
"currIcmp" : "0",
"currUdp" : "0",
"currSyn" : "0",
"currIcmpMax" : "0",
"currUdpMax" : "14",
"currSynMax" : "4" }
{
"ipAddress" : "3232235842",
"macAddress" : "70",:14:A6:8B:XX:XX
"totalPkts" : "154566",
"totalBytes" : "15116490",
"currPkts" : "0",
"currBytes" : "0",
"currIcmp" : "0",
"currUdp" : "0",
"currSyn" : "0",
"currIcmpMax" : "9",
"currUdpMax" : "58",
"currSynMax" : "60" }

Answer 2

perl -lne 'BEGIN{*1 = sub {print q<{>, join(",\n", splice @A, 0, @A), q<}>}}
   next if $. == 1;
   /^\[(?:\d+,?)+\]\d+$/ and &1,next;
   push @A, join q/ : /, map qq/"$_"/, split /=/;
   eof && &1;
' yourfile

Answer 3

Using Miller (mlr) after some light preprocessing with sed:

sed 's/^\[.*//' file | mlr --irs lflf --ifs lf --ojson cat

This first uses sed to clear the lines that start with [. The empty lines that this creates later serve as record separators for Miller.

Miller then converts the input to JSON without further modifying the data. To make the data understandable to Miller, we specify that the input record separator is a double line-feed, and that the input field separator is a single line-feed.

This command, given the example data in the question, would produce the following output:

[
{
  "enable": 1,
  "interval": 10,
  "action": 0
},
{
  "ipAddress": 3232235848,
  "macAddress": "EC:0E:C4:4F:XX:XX",
  "totalPkts": 201,
  "totalBytes": 22914,
  "currPkts": 0,
  "currBytes": 0,
  "currIcmp": 0,
  "currUdp": 0,
  "currSyn": 0,
  "currIcmpMax": 0,
  "currUdpMax": 15,
  "currSynMax": 0
},
{
  "ipAddress": 3232235829,
  "macAddress": "00:16:E8:22:XX:XX",
  "totalPkts": 972,
  "totalBytes": 98730,
  "currPkts": 0,
  "currBytes": 0,
  "currIcmp": 0,
  "currUdp": 0,
  "currSyn": 0,
  "currIcmpMax": 0,
  "currUdpMax": 14,
  "currSynMax": 4
},
{
  "ipAddress": 3232235842,
  "macAddress": "70:14:A6:8B:XX:XX",
  "totalPkts": 154566,
  "totalBytes": 15116490,
  "currPkts": 0,
  "currBytes": 0,
  "currIcmp": 0,
  "currUdp": 0,
  "currSyn": 0,
  "currIcmpMax": 9,
  "currUdpMax": 58,
  "currSynMax": 60
}
]

To avoid that first object in the resulting array, change the cat operation to a filter operation that avoids outputting the first record:

sed 's/^\[.*//' file | mlr --irs lflf --ifs lf --ojson filter 'NR > 1'

Would you additionally prefer to not have the resulting JSON objects wrapped in an array, then use Miller with --no-jlistwrap:

sed 's/^\[.*//' file | mlr --irs lflf --ifs lf --ojson --no-jlistwrap filter 'NR > 1'

... or change to the one-object-per-line JSON output format by changing --ojson to --ojsonl:

sed 's/^\[.*//' file | mlr --irs lflf --ifs lf --ojsonl filter 'NR > 1'

... which would produce

{"ipAddress": 3232235848, "macAddress": "EC:0E:C4:4F:XX:XX", "totalPkts": 201, "totalBytes": 22914, "currPkts": 0, "currBytes": 0, "currIcmp": 0, "currUdp": 0, "currSyn": 0, "currIcmpMax": 0, "currUdpMax": 15, "currSynMax": 0}
{"ipAddress": 3232235829, "macAddress": "00:16:E8:22:XX:XX", "totalPkts": 972, "totalBytes": 98730, "currPkts": 0, "currBytes": 0, "currIcmp": 0, "currUdp": 0, "currSyn": 0, "currIcmpMax": 0, "currUdpMax": 14, "currSynMax": 4}
{"ipAddress": 3232235842, "macAddress": "70:14:A6:8B:XX:XX", "totalPkts": 154566, "totalBytes": 15116490, "currPkts": 0, "currBytes": 0, "currIcmp": 0, "currUdp": 0, "currSyn": 0, "currIcmpMax": 9, "currUdpMax": 58, "currSynMax": 60}