5

I've written a script that converts nmcli --mode multiline dev wifi into json but I'm finding it's inconsistent (breaks when results have a space), long, and hard to read.

I'm wondering if it is possible to pipe the results directly into jq. The output looks like:

*:                                       
SSID:                                   VIDEOTRON2255
MODE:                                   Infra
CHAN:                                   11
RATE:                                   54 Mbit/s
SIGNAL:                                 69
BARS:                                   ▂▄▆_
SECURITY:                               WPA1 WPA2
*:                                      * 
SSID:                                   VIDEOTRON2947
MODE:                                   Infra
CHAN:                                   6
RATE:                                   54 Mbit/s
SIGNAL:                                 49
BARS:                                   ▂▄__
SECURITY:                               WPA1 WPA2

I'm looking to generate something like this:

[{
    "network": "VIDEOTRON2255",
    "mode": "Infra",
    "chan": "11",
    "rate": "54 Mbit/s",
    "signal": "69",
    "bars": "▂▄▆_",
    "security": "WPA1 WPA2"
},
{
    "network": "VIDEOTRON2947",
    "mode": "Infra",
    "chan": "6",
    "rate": "54 Mbit/s",
    "signal": "49",
    "bars": "▂▄__",
    "security": "WPA1 WPA2"
}]

Yes, I did ask a related question earlier. This is the first script I wrote based on Gilles answer, it worked for some of the values but not security or rate which have spaces.

  • 1
    there would a problem: I got error on encountering that crooked bars value "▂▄▆_" – RomanPerekhrest Oct 19 '17 at 22:23
  • @RomanPerekhrest could you post your solution and just remove lines with BARS: using grep? It's not that important since bars is likely just determined by signal. – Philip Kirkbride Oct 19 '17 at 22:27
  • you got my solution – RomanPerekhrest Oct 19 '17 at 23:02
  • Stupid question: Why insist on using jq to parse this? jq is made for parsing JSON, and this is not JSON. OTOH, any other program that can parse this can often easily produce JSON ... – dirkt Oct 21 '17 at 22:00
  • @dirkt I guess you're right, maybe I was wrong assuming jq would be a good tool for the job. There is no reason that I would have to use it in my situation. – Philip Kirkbride Oct 21 '17 at 22:04
6
+50

The script that you linked to is extremely inefficient - you're doing a lot of useless pre-processing...
Use nmcli in --terse mode since, per the manual, "this mode is designed and suitable for computer (script) processing", specify the desired fields and pipe the output to jq -sR e.g.

printf '%s' "$(nmcli -f ssid,mode,chan,rate,signal,bars,security -t dev wifi)" | \
jq -sR 'split("\n") | map(split(":")) | map({"network": .[0],
                                             "mode": .[1],
                                             "channel": .[2],
                                             "rate": .[3],
                                             "signal": .[4],
                                             "bars": .[5],
                                             "security": .[6]})'
  • Wow, that is a really short script, nice job. On my end I always get an extra entry with all null values. – Philip Kirkbride Oct 22 '17 at 13:36
  • 1
    @PhilipKirkbride - if your jq doesn't support those options you can always do something like nmcli -f ssid,mode,chan,rate,signal,bars,security -t dev wifi | awk -F: 'BEGIN{z[1]="network";z[2]="mode";z[3]="channel";z[4]="rate";z[5]="signal";z[6]="bars";z[7]="security";}{if (NR==1){printf "%s", "[{"}else{printf "%s",",{"}}{for (i=1; i<NF; i++){ printf "\"%s\":\"%s\",",z[i],$i};{ printf "\"%s\":\"%s\"}",z[NF],$NF}}END{print "]"}' | jq . – don_crissti Oct 26 '17 at 16:11
2

This GNU sed code isn't jq, (it isn't a complex conversion), but it seems to work well enough, (even the bars come out OK):

nmcli --mode multiline dev wifi | 
sed    '/^*/! {s/^[A-Z]*/\L&/
               s/ssid/network/
               s/: */": "/
               s/$/"/
               {/^sec/!s/$/,/}
               s/^/\t"/}
        1     s/^\*.*/[{/
        /^\*/ s/.*/},\n{/
        $  {p;s/.*/}]/}'

Easier to read standalone pcsvp.sed script, (save to file, then run chmod +x pcsvp.sed):

#!/bin/sed -f
# Text lines (the non "*:" lines.)
/^*/! {s/^[A-Z]*/\L&/
       s/ssid/network/
       s/: */": "/
       s/$/"/
       {/^sec/!s/$/,/}
       s/^/\t"/}

# First JSON line
1     s/^\*.*/[{/

# Middle JSON lines.  If a line begins with a '*'...
/^\*/ s/.*/},\n{/

# Last line, close up the JSON.
$     {p;s/.*/}]/}

To run that do:

nmcli --mode multiline dev wifi | ./pcsvp.sed

Note: Since there are doubts about the input file, I've opted to use nmcli for input instead. At my location this shows about 50 networks, which makes the resulting output too long to quote here.

If the input sample typo is corrected, ./pcsvp.sed input.txt outputs:

[{
    "network": "VIDEOTRON2255",
    "mode": "Infra",
    "chan": "11",
    "rate": "54 Mbit/s",
    "signal": "69",
    "bars": "▂▄▆_",
    "security": "WPA1 WPA2"
},
{
    "network": "VIDEOTRON2947",
    "mode": "Infra",
    "chan": "6",
    "rate": "54 Mbit/s",
    "signal": "49",
    "bars": "▂▄__",
    "security": "WEP"
}]
  • This can generate errors with empty lines. I had to use nmcli --mode multiline dev wifi | sed '/^$/d' | ./scan.sed. In the end this answer was more reliable because the end product was on Ubuntu 14.04 and the default version of jq didn't have the flags used in @don_crissti's answer. – Philip Kirkbride Oct 25 '17 at 19:57
  • @PhilipKirkbride, I didn't know that nmcli could output any empty lines. – agc Oct 26 '17 at 4:15
  • I think it's the very last line that is empty – Philip Kirkbride Oct 26 '17 at 10:30
  • @PhilipKirkbride, With network-manager v1.4.4-1ubuntu3.2, running nmcli --mode multiline dev wifi | grep '^$' | wc -l consistently returns 0. – agc Oct 26 '17 at 14:37
1

Complex jq solution (with BARS line removed as it contains irregular/non-ASCII characters):

Input file input.txt:

*:                                       
SSID:                                   VIDEOTRON2255
MODE:                                   Infra
CHAN:                                   11
RATE:                                   54 Mbit/s
SIGNAL:                                 69
SECURITY:                               WPA1 WPA2
*:                                      * 
SSID:                                   VIDEOTRON2947
MODE:                                   Infra
CHAN:                                   6
RATE:                                   54 Mbit/s
SIGNAL:                                 49
SECURITY:                               WPA1 WPA2

The job:

jq -sR '[ gsub("[*]: *\n| {2,}";"") | gsub("SSID";"network") | split("\n[*]:[*] +\n";"n")[] 
    | [ capture("(?<key>[^:\n]+):(?<value>[^:\n]+)";"g") | .key |= (. | ascii_downcase) ] 
    | from_entries ]' input.txt

The output:

[
  {
    "network": "VIDEOTRON2255",
    "mode": "Infra",
    "chan": "11",
    "rate": "54 Mbit/s",
    "signal": "69",
    "security": "WPA1 WPA2"
  },
  {
    "network": "VIDEOTRON2947",
    "mode": "Infra",
    "chan": "6",
    "rate": "54 Mbit/s",
    "signal": "49",
    "security": "WPA1 WPA2"
  }
]

Additional approach for another particular input presented/posted on https://pastebin.com/8stHSUeu:

jq -sR '[sub("[*]: *[*]\n";"") | gsub(" {2,}";"") | gsub("SSID";"network") 
  | split("\n[*]: *\n";"n")[] 
  | [ capture("(?<key>[^:\n]+):(?<value>[^:\n]+)";"g") | .key |= (. | ascii_downcase) ] 
  | from_entries]' input.txt
  • Thank you this is perfect. I just have one question it fails when I actually use it. Do you think this Cocotte-invité causes the issue? pastebin.com/8stHSUeu – Philip Kirkbride Oct 19 '17 at 23:14
  • @PhilipKirkbride, welcome. What is the error message? – RomanPerekhrest Oct 19 '17 at 23:16
  • src/scripts/scan2.sh: line 12: 4640 Segmentation fault (core dumped) jq -sR '[ gsub("[*]: *\n| {2,}";"") | gsub("SSID";"network") | split("\n[*]:[*] +\n";"n")[] | [ capture("(?<key>[^:\n]+):(?<value>[^:\n]+)";"g") | .key |= (. | ascii_downcase) ] | from_entries ]' /tmp/json_gen – Philip Kirkbride Oct 19 '17 at 23:16
  • 1
    @PhilipKirkbride, Cocotte-invité should not be an issue, but I have noted that your new input is somehow differs from that one in the question. The 1st line in your new input has * at the end: *: * while the starting line in question's input does not have it – RomanPerekhrest Oct 19 '17 at 23:32
  • @PhilipKirkbride, check my additional approach – RomanPerekhrest Oct 19 '17 at 23:38
1

If you can, use a tool that understands JSON back and forth. I'd use Python:

#! /usr/bin/env python3
import json
import re
import sys

objects = []
obj = {}
for line in sys.stdin:
    entry = re.split(':\s*', line.strip(), maxsplit=1) # split on first `:`
    if entry[0] == '*':
        if obj:  # skip a null entry (the first, here)
            obj['network'] = obj.pop('ssid') # rename the SSID entry
            objects.append(obj)
        obj = {} # start a new object for each `*`
        continue
    obj[entry[0].lower()] = entry[1]  # lowercase the key
objects.append(obj)
json.dump(objects, sys.stdout)

Gets me:

[{"mode": "Infra", "chan": "11", "rate": "54 Mbit/s", "signal": "69", "bars": "\u2582\u2584\u2586_", "security": "WPA1 WPA2", "network": "VIDEOTRON2255"}, {"ssid": "VIDEOTRON2947", "mode": "Infra", "chan": "6", "rate": "54 Mbit/s", "signal": "49", "bars": "\u2582\u2584__", "security": "WPA1 WPA2"}]

which, when pretty-printed by jq is:

[
  {
    "mode": "Infra",
    "chan": "11",
    "rate": "54 Mbit/s",
    "signal": "69",
    "bars": "▂▄▆_",
    "security": "WPA1 WPA2",
    "network": "VIDEOTRON2255"
  },
  {
    "ssid": "VIDEOTRON2947",
    "mode": "Infra",
    "chan": "6",
    "rate": "54 Mbit/s",
    "signal": "49",
    "bars": "▂▄__",
    "security": "WPA1 WPA2"
  }
]
1
  1. GNU sed and mlr method:

    nmcli dev wifi | sed 'y/*/ /;1{s/.*/\L&/;s/ssid/network/};s/   */\t/g'  | 
    mlr --p2j --fs '\t' --jvstack --jlistwrap cat
    
  2. bash, text mode nmcli, (swiped from don_chrissti's answer), and mlr:

    h=ssid:mode:chan:rate:signal:bars:security
    { echo ${h/ssid/network} ; nmcli -f ${h//:/,} -t dev wifi ; } | \
    mlr --c2j --fs ':' --jvstack --jlistwrap cat
    
  • The right combination of nmcli's output formats and mlr's many input formats might provide a way to avoid the use of bash variables. – agc Oct 22 '17 at 17:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.