9

I've written a script that converts the output of nmcli --mode multiline dev wifi into JSON, but I'm finding it's inconsistent (breaks when results have a space), long, and hard to read.

I wonder if it is possible to pipe the results directly into jq.

The nmcli output (input to my script) looks like this:

*:                                       
SSID:                                   VIDEOTRON2255
MODE:                                   Infra
CHAN:                                   11
RATE:                                   54 Mbit/s
SIGNAL:                                 69
BARS:                                   ▂▄▆_
SECURITY:                               WPA1 WPA2
*:                                      * 
SSID:                                   VIDEOTRON2947
MODE:                                   Infra
CHAN:                                   6
RATE:                                   54 Mbit/s
SIGNAL:                                 49
BARS:                                   ▂▄__
SECURITY:                               WPA1 WPA2

I'm looking to generate something like this:

[{
    "network": "VIDEOTRON2255",
    "mode": "Infra",
    "chan": "11",
    "rate": "54 Mbit/s",
    "signal": "69",
    "bars": "▂▄▆_",
    "security": "WPA1 WPA2"
},
{
    "network": "VIDEOTRON2947",
    "mode": "Infra",
    "chan": "6",
    "rate": "54 Mbit/s",
    "signal": "49",
    "bars": "▂▄__",
    "security": "WPA1 WPA2"
}]

I asked a related question earlier. This is the first script I wrote based on Gilles's answer. It worked for some of the values but not security or rate, which have spaces.

6
  • 1
    there would a problem: I got error on encountering that crooked bars value "▂▄▆_" Oct 19, 2017 at 22:23
  • @RomanPerekhrest could you post your solution and just remove lines with BARS: using grep? It's not that important since bars is likely just determined by signal. Oct 19, 2017 at 22:27
  • you got my solution Oct 19, 2017 at 23:02
  • Stupid question: Why insist on using jq to parse this? jq is made for parsing JSON, and this is not JSON. OTOH, any other program that can parse this can often easily produce JSON ...
    – dirkt
    Oct 21, 2017 at 22:00
  • @dirkt I guess you're right, maybe I was wrong assuming jq would be a good tool for the job. There is no reason that I would have to use it in my situation. Oct 21, 2017 at 22:04

6 Answers 6

8
+50

The script that you linked to is extremely inefficient - you're doing a lot of useless pre-processing...
Use nmcli in --terse mode since, per the manual, "this mode is designed and suitable for computer (script) processing", specify the desired fields and pipe the output to jq -sR e.g.

printf '%s' "$(nmcli -f ssid,mode,chan,rate,signal,bars,security -t dev wifi)" | \
jq -sR 'split("\n") | map(split(":")) | map({"network": .[0],
                                             "mode": .[1],
                                             "channel": .[2],
                                             "rate": .[3],
                                             "signal": .[4],
                                             "bars": .[5],
                                             "security": .[6]})'
7
  • Wow, that is a really short script, nice job. On my end I always get an extra entry with all null values. Oct 22, 2017 at 13:36
  • 1
    @PhilipKirkbride - if your jq doesn't support those options you can always do something like nmcli -f ssid,mode,chan,rate,signal,bars,security -t dev wifi | awk -F: 'BEGIN{z[1]="network";z[2]="mode";z[3]="channel";z[4]="rate";z[5]="signal";z[6]="bars";z[7]="security";}{if (NR==1){printf "%s", "[{"}else{printf "%s",",{"}}{for (i=1; i<NF; i++){ printf "\"%s\":\"%s\",",z[i],$i};{ printf "\"%s\":\"%s\"}",z[NF],$NF}}END{print "]"}' | jq . Oct 26, 2017 at 16:11
  • Why would you do printf '%s' "$(nmcli …)" | … instead of just nmcli … | …? Is that to strip off trailing newline(s)? Feb 5, 2023 at 18:41
  • @G-ManSays'ReinstateMonica' - not my code (see edits); I guess it's related to OP's first comment above (re: the additional entry with all null values)... and also his comments on agc's answer... apparently his version of nmcli was printing additional empty lines Feb 5, 2023 at 19:03
  • @agc: Why would you do printf '%s' "$(nmcli …)" | … instead of just nmcli … | …? Is that to strip off trailing newline(s)? Feb 5, 2023 at 19:09
2

This GNU sed code isn't jq, (it isn't a complex conversion), but it seems to work well enough, (even the bars come out OK):

nmcli --mode multiline dev wifi | 
sed    '/^*/! {s/^[A-Z]*/\L&/
               s/ssid/network/
               s/: */": "/
               s/$/"/
               {/^sec/!s/$/,/}
               s/^/\t"/}
        1     s/^\*.*/[{/
        /^\*/ s/.*/},\n{/
        $  {p;s/.*/}]/}'

Easier to read standalone pcsvp.sed script, (save to file, then run chmod +x pcsvp.sed):

#!/bin/sed -f
# Text lines (the non "*:" lines.)
/^*/! {s/^[A-Z]*/\L&/
       s/ssid/network/
       s/: */": "/
       s/$/"/
       {/^sec/!s/$/,/}
       s/^/\t"/}

# First JSON line
1     s/^\*.*/[{/

# Middle JSON lines.  If a line begins with a '*'...
/^\*/ s/.*/},\n{/

# Last line, close up the JSON.
$     {p;s/.*/}]/}

To run that do:

nmcli --mode multiline dev wifi | ./pcsvp.sed

Note: Since there are doubts about the input file, I've opted to use nmcli for input instead. At my location this shows about 50 networks, which makes the resulting output too long to quote here.

If the input sample typo is corrected, ./pcsvp.sed input.txt outputs:

[{
    "network": "VIDEOTRON2255",
    "mode": "Infra",
    "chan": "11",
    "rate": "54 Mbit/s",
    "signal": "69",
    "bars": "▂▄▆_",
    "security": "WPA1 WPA2"
},
{
    "network": "VIDEOTRON2947",
    "mode": "Infra",
    "chan": "6",
    "rate": "54 Mbit/s",
    "signal": "49",
    "bars": "▂▄__",
    "security": "WEP"
}]
4
  • This can generate errors with empty lines. I had to use nmcli --mode multiline dev wifi | sed '/^$/d' | ./scan.sed. In the end this answer was more reliable because the end product was on Ubuntu 14.04 and the default version of jq didn't have the flags used in @don_crissti's answer. Oct 25, 2017 at 19:57
  • @PhilipKirkbride, I didn't know that nmcli could output any empty lines.
    – agc
    Oct 26, 2017 at 4:15
  • I think it's the very last line that is empty Oct 26, 2017 at 10:30
  • @PhilipKirkbride, With network-manager v1.4.4-1ubuntu3.2, running nmcli --mode multiline dev wifi | grep '^$' | wc -l consistently returns 0.
    – agc
    Oct 26, 2017 at 14:37
1

Complex jq solution (with BARS line removed as it contains irregular/non-ASCII characters):

Input file input.txt:

*:                                       
SSID:                                   VIDEOTRON2255
MODE:                                   Infra
CHAN:                                   11
RATE:                                   54 Mbit/s
SIGNAL:                                 69
SECURITY:                               WPA1 WPA2
*:                                      * 
SSID:                                   VIDEOTRON2947
MODE:                                   Infra
CHAN:                                   6
RATE:                                   54 Mbit/s
SIGNAL:                                 49
SECURITY:                               WPA1 WPA2

The job:

jq -sR '[ gsub("[*]: *\n| {2,}";"") | gsub("SSID";"network") | split("\n[*]:[*] +\n";"n")[] 
    | [ capture("(?<key>[^:\n]+):(?<value>[^:\n]+)";"g") | .key |= (. | ascii_downcase) ] 
    | from_entries ]' input.txt

The output:

[
  {
    "network": "VIDEOTRON2255",
    "mode": "Infra",
    "chan": "11",
    "rate": "54 Mbit/s",
    "signal": "69",
    "security": "WPA1 WPA2"
  },
  {
    "network": "VIDEOTRON2947",
    "mode": "Infra",
    "chan": "6",
    "rate": "54 Mbit/s",
    "signal": "49",
    "security": "WPA1 WPA2"
  }
]

Additional approach for another particular input presented/posted on https://pastebin.com/8stHSUeu:

jq -sR '[sub("[*]: *[*]\n";"") | gsub(" {2,}";"") | gsub("SSID";"network") 
  | split("\n[*]: *\n";"n")[] 
  | [ capture("(?<key>[^:\n]+):(?<value>[^:\n]+)";"g") | .key |= (. | ascii_downcase) ] 
  | from_entries]' input.txt
8
  • Thank you this is perfect. I just have one question it fails when I actually use it. Do you think this Cocotte-invité causes the issue? pastebin.com/8stHSUeu Oct 19, 2017 at 23:14
  • @PhilipKirkbride, welcome. What is the error message? Oct 19, 2017 at 23:16
  • src/scripts/scan2.sh: line 12: 4640 Segmentation fault (core dumped) jq -sR '[ gsub("[*]: *\n| {2,}";"") | gsub("SSID";"network") | split("\n[*]:[*] +\n";"n")[] | [ capture("(?<key>[^:\n]+):(?<value>[^:\n]+)";"g") | .key |= (. | ascii_downcase) ] | from_entries ]' /tmp/json_gen Oct 19, 2017 at 23:16
  • 1
    @PhilipKirkbride, Cocotte-invité should not be an issue, but I have noted that your new input is somehow differs from that one in the question. The 1st line in your new input has * at the end: *: * while the starting line in question's input does not have it Oct 19, 2017 at 23:32
  • @PhilipKirkbride, check my additional approach Oct 19, 2017 at 23:38
1
  1. GNU sed and mlr method:

    nmcli dev wifi | sed 'y/*/ /;1{s/.*/\L&/;s/ssid/network/};s/   */\t/g'  | 
    mlr --p2j --fs '\t' --jvstack --jlistwrap cat
    
  2. bash, text mode nmcli, (swiped from don_chrissti's answer), and mlr:

    h=ssid:mode:chan:rate:signal:bars:security
    { echo ${h/ssid/network} ; nmcli -f ${h//:/,} -t dev wifi ; } | \
    mlr --c2j --fs ':' --jvstack --jlistwrap cat
    
1
  • The right combination of nmcli's output formats and mlr's many input formats might provide a way to avoid the use of bash variables.
    – agc
    Oct 22, 2017 at 17:48
1

If you can, use a tool that understands JSON back and forth. I'd use Python:

#! /usr/bin/env python3
import json
import re
import sys

objects = []
obj = {}
for line in sys.stdin:
    entry = re.split(':\s*', line.strip(), maxsplit=1) # split on first `:`
    if entry[0] == '*':
        if obj:  # skip a null entry (the first, here)
            obj['network'] = obj.pop('ssid') # rename the SSID entry
            objects.append(obj)
        obj = {} # start a new object for each `*`
        continue
    obj[entry[0].lower()] = entry[1]  # lowercase the key
obj['network'] = obj.pop('ssid') # rename the SSID entry
objects.append(obj)
json.dump(objects, sys.stdout)

Gets me:

[{"mode": "Infra", "chan": "11", "rate": "54 Mbit/s", "signal": "69", "bars": "\u2582\u2584\u2586_", "security": "WPA1 WPA2", "network": "VIDEOTRON2255"}, {"ssid": "VIDEOTRON2947", "mode": "Infra", "chan": "6", "rate": "54 Mbit/s", "signal": "49", "bars": "\u2582\u2584__", "security": "WPA1 WPA2"}]

which, when pretty-printed by jq is:

[
  {
    "mode": "Infra",
    "chan": "11",
    "rate": "54 Mbit/s",
    "signal": "69",
    "bars": "▂▄▆_",
    "security": "WPA1 WPA2",
    "network": "VIDEOTRON2255"
  },
  {
    "ssid": "VIDEOTRON2947",
    "mode": "Infra",
    "chan": "6",
    "rate": "54 Mbit/s",
    "signal": "49",
    "bars": "▂▄__",
    "security": "WPA1 WPA2"
  }
]
2
  • I'm just noticing that your output contains an element with a network key and one with a ssid key.
    – Kusalananda
    Jul 11, 2021 at 11:05
  • Ah, I see now. In each case I do the change for the previous entry, which means the last one is left as-is.
    – muru
    Jul 11, 2021 at 14:50
0

Using csvjson from the csvkit toolkit to create the JSON document directly from the nmcli command:

#!/bin/sh

set -- SSID MODE CHAN RATE SIGNAL BARS SECURITY

( IFS=:; printf '%s\n' "$*"
  IFS=,; nmcli -g "$*" device wifi ) |
csvjson -d :

The first printf creates a :-delimited CSV header for the cvsjson command created from the positional parameters set at the start of the script, and the nmcli command gets a comma-delimited list of the same parameters as the option-argument to its -g (--get-values) option. The nmcli command will create what amounts to :-delimited CSV output, and this, together with the simplistic CSV header, is parsed by csvjson to produce a JSON document.

The JSON document created will contain an array of objects like

{
  "SSID": "The wireless network",
  "MODE": "Infra",
  "CHAN": 6,
  "RATE": "195 Mbit/s",
  "SIGNAL": 45,
  "BARS": "▂▄__",
  "SECURITY": "WPA2"
}

To lowercase the keys and to change SSID into network, we may use jq:

#!/bin/sh

set -- SSID MODE CHAN RATE SIGNAL BARS SECURITY

( IFS=:; printf '%s\n' "$*"
  IFS=,; nmcli -g "$*" device wifi ) |
csvjson -d : |
jq '.[] | with_entries(if .key == "SSID" then .key |= "network" else .key |= ascii_downcase end)'

This is expected to produce an array with entries like

{
  "network": "The wireless network",
  "mode": "Infra",
  "chan": 6,
  "rate": "195 Mbit/s",
  "signal": 34,
  "bars": "▂▄__",
  "security": "WPA2"
}

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .