1

I type mosquitto_sub -d -t +/# from the Ubuntu terminal to access the MQTT stream.

However I would like to see only a specific set of data from the MQTT stream. I've added | grep -v PING so everything still be printed except for lines with PING in them but that didn't work.

I've tried | grep -A1 PUBLISH so I can see lines from the MQTT with PUBLISH in them, nothing changed: I just get the whole MQTT stream nothing gets filtered out.

How can I see the certain thing that I want to see from an MQTT stream? Alternatively; how can I filter certain things out of the MQTT stream or how can I extract data from the MQTT stream?

My input from the ubuntu terminal is as follows $ mosquitto_sub -d -t +/# and I've tried $ mosquitto_sub -d -t +/# | grep -v PING and $ mosquitto_sub -d -t +/# | grep -A1 PUBLISH but it just streams everything, doesn't grep anything. The output is as follows:

ed@agharta:~$ mosquitto_sub -d -t +/#
Received CONNACK
Received SUBACK
Subscribed (mid: 1): 0
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Received PUBLISH (d0, q0, r0, m0, 'm/jsyd/TEST/001/d/status', ... (34 bytes))
Sending PINGREQ
Received PINGRESP
Received PUBLISH (d0, q0, r0, m0, 'm/jsyd/TEST/001/d/SVlts', ... (28 bytes))
Sending PINGREQ
Received PINGRESP

For example how can I only see the PUBLISH from the live stream and not the PINGs?

  • Please edit your question to include sample input and output... – jasonwryan Feb 24 '15 at 2:49
  • @jasonwryan I've included the input and output from the terminal – 3kstc Feb 24 '15 at 3:01
  • sed -n '/Subscribed/p' stream – jasonwryan Feb 24 '15 at 3:13
  • 1
    @3kstc - Your commands are OK but note that grep doesn't operate on stderr. Apparently mosquitto_sub logs to stderr so (with bash) you could try something like mosquitto_sub -d -t +/# 2> >(grep PUBLISH). For more details & options read the answers here. – don_crissti Feb 24 '15 at 12:42
  • 1
    @3kstc - redirection and process substitution... here is some stuff for you to read; as to grep more patterns, here's some more to read :) HTH. – don_crissti Feb 25 '15 at 2:42
2

I was in the process of writing the following answer to the other, related question here when it was deleted. I don't know if you'll be able to follow that link, but (an abbreviation of) the example data referred to below is this:

Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Received PUBLISH (d0, q0, r0, m0, 'm/gf/TMX6BP/075/d/SVlts', ... (28 bytes))
86,1224830,27610 27869 17565
Received PUBLISH (d0, q0, r0, m0, 'm/gf/TMX6BP/075/d/status', ... (39 bytes))
86,1243000,164573,-33.836465,151.051189
Sending PINGREQ
Received PINGRESP

So I wrote the following to make it work with sed like this:

mosquitto ... 2>&1 | 
sed -u "/.*PUBLISH[^']*\([^,]*,\).*/!d 
       s//\1/;N;s/[\%]/&&/g;s/'/&\\\\&&/g
       s/.*/printf '&'/;s/\n/'&date '+[%s,]/"|  
sh -s

Now, I don't have your mosquitto program, but if I copy your whole example thing to my clipboard, and do:

xsel -bo | sed ... | sh ...

It prints:

'm/gf/TMX6BP/075/d/SVlts',[1425002404,]86,1224830,27610 27869 17565
'm/gf/TMX6BP/075/d/status',[1425002404,]86,1243000,164573,-33.836465,151.051189
'm/NWRL/TMX/098/d/SVlts',[1425002404,]806,3040421,7549 7750 3904
'm/NWRL/TMX/098/d/status',[1425002404,]806,3069000,59666,-33.836465,151.051189
'm/NWRL/TMX/098/d/SVlts',[1425002404,]810,5440995,6143 7807 4076
'm/NWRL/TMX/098/d/status',[1425002404,]810,5489000,59897,-33.836465,151.051189

...which is, I think, what you want. It works by ensuring all of the input is safely quoted for both the shell and for date and printf's format strings, and then constructing shell commands out of each line pair.

For each line pair printf first prints the contents of the first line w/out a trailing \newline, then date prints the [timestamp] in seconds since the epoch followed by the contents of the second line and a trailing \newline. sed's output looks something like this:

printf ''\''m/gf/TMX6BP/075/d/SVlts'\'','
date '+[%s,]86,1224830,27610 27869 17565'

None of the commands printed at sh, of course, are executed until sed prints them at it. So if sed were reading mosquitto's output and one line came in later than a previous one then the date line would reflect that.

For example:

(set -f; IFS='
';  for l in $(xsel -bo)
    do  printf %s\\n "$l"
        sleep 1
    done) |
sed -u "/.*PUBLISH[^']*\([^,]*,\).*/!d 
       s//\1/;N;s/[\%]/&&/g;s/'/&\\\\&&/g
       s/.*/printf '&'/;s/\n/'&date '+[%s,]/"|  
sh -s

...which waits a second before printing every output line at sed in order to more faithfully emulate a live stream. It takes advantage of the GNU sed -unbuffered switch to get sed's output flushed as often as possible, but if that is not available to you let me know and I will show you how to block out a pipe correctly w/ dd to avoid buffering issues.

Anyway, the above prints:

'm/gf/TMX6BP/075/d/SVlts',[1425002863,]86,1224830,27610 27869 17565
'm/gf/TMX6BP/075/d/status',[1425002865,]86,1243000,164573,-33.836465,151.051189
'm/NWRL/TMX/098/d/SVlts',[1425002869,]806,3040421,7549 7750 3904
'm/NWRL/TMX/098/d/status',[1425002871,]806,3069000,59666,-33.836465,151.051189
'm/NWRL/TMX/098/d/SVlts',[1425002881,]810,5440995,6143 7807 4076
'm/NWRL/TMX/098/d/status',[1425002885,]810,5489000,59897,-33.836465,151.051189

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.