1

I type mosquitto_sub -d -t +/# from the Ubuntu terminal to access the MQTT stream.

Real output from the live MQTT stream is this:

Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Received PUBLISH (d0, q0, r0, m0, 'm/gf/TMX6BP/075/d/SVlts', ... (28 bytes))
86,1224830,27610 27869 17565
Received PUBLISH (d0, q0, r0, m0, 'm/gf/TMX6BP/075/d/status', ... (39 bytes))
86,1243000,164573,-33.836465,151.051189
Sending PINGREQ
Received PINGRESP
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/SVlts', ... (26 bytes))
806,3040421,7549 7750 3904
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/status', ... (39 bytes))
806,3069000,59666,-33.836465,151.051189
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/SVlts', ... (26 bytes))
810,5440995,6143 7807 4076
Sending PINGREQ
Received PINGRESP
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/status', ... (39 bytes))
810,5489000,59897,-33.836465,151.051189
Sending PINGREQ
Received PINGRESP

There is no way of predicting when the next PUBLISH will be seen as they are only seen in the stream when the vehicle has transmission/reception from the GSM/3G towers

To filter I add mosquitto_sub -d -t +/# 2> >(grep PUBLISH) this will only allow lines with PUBLISH in it, hence the ouput is:

Received PUBLISH (d0, q0, r0, m0, 'm/gf/TMX6BP/075/d/status', ... (38 bytes))
86,637999,164563,-33.836465,151.051189
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/SVlts', ... (26 bytes))
806,3040421,7549 7750 3904
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/status', ... (39 bytes))
806,3069000,59666,-33.836465,151.051189
Received PUBLISH (d0, q0, r0, m0, 'm/gf/TMX6BP/075/d/SVlts', ... (28 bytes))
86,1224830,27610 27869 17565
Received PUBLISH (d0, q0, r0, m0, 'm/gf/TMX6BP/075/d/status', ... (39 bytes))
86,1243000,164573,-33.836465,151.051189
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/SVlts', ... (26 bytes))
806,3640483,7463 7721 3933
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/status', ... (39 bytes))
806,3674000,59676,-33.836465,151.051189
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/SVlts', ... (26 bytes))
806,4240543,7291 7750 3933
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/status', ... (39 bytes))
806,4279000,59687,-33.836465,151.051189
Received PUBLISH (d0, q0, r0, m0, 'm/gf/MXE/065/d/SVlts', ... (25 bytes))
455,24715,28041 28041 967

How would I be able to eliminate a few of the fields and also add a time stamp everytime I receive something; I 've tried using sed but had no luck. I entered $ mosquitto_sub -d -t +/# 2< <(grep PUBLISH) 2< <(sed "s/^/ date/") , $ mosquitto_sub -d -t +/# 2< <(grep PUBLISH) 2< <(sed "s/^/$date`/")

Q: How can I change my input to the terminal so that the output from the above live feed would be:

[timestamp],m,gf,TMX6BP,075,d,status,86,637999,164563,-33.836465,151.051189
[timestamp],m,NWRL,TMX,098,d,SVlts,806,3040421,7549 7750 3904
[timestamp],m,NWRL,TMX,098,d,status,806,3069000,59666,-33.836465,151.051189
[timestamp],m,gf,TMX6BP,075,d,SVlts,86,1224830,27610 27869 17565
[timestamp],m,gf,TMX6BP,075,d,status,86,1243000,164573,-33.836465,151.051189
[timestamp],m,NWRL,TMX,098,d,SVlts,806,3640483,7463 7721 3933
[timestamp],m,NWRL,TMX,098,d,status,806,3674000,59676,-33.836465,151.051189
[timestamp],m,NWRL,TMX,098,d,SVlts,806,4240543,7291 7750 3933
[timestamp],m,NWRL,TMX,098,d,status,806,4279000,59687,-33.836465,151.051189
[timestamp],m,gf,MXE,065,d,SVlts,455,24715,28041 28041 967

Possible solutions (future referencing):

Using the mosquitto_sub -d -t +/# 2> >(sed -n "s|.*\('.*',\).*|\1|p") | sed "N;s/\n/ /;s/$/ $(date)/" The output is:

0 810,5440995,6143 7807 4076 Wed Feb 25 23:23:51 UTC 2015 810,5489000,59897,-33.836465,151.051189 810,6041055,7606 7693 4076 Wed Feb 25 23:23:51 UTC 2015

Using the mosquitto_sub -d -t +/# 2> >(grep PUBLISH) | sed "N;s/\n/ /;s/$/ $(date)/" command from the terminal the output is

817,3069000,60045,-33.836465,151.051189 609,24570,27553 27553 955 Thu Feb 26 00:06:26 UTC 2015

Using the mosquitto_sub -d -t +/# 2>&1 | sed -n "/PUBLISH/{s|.*\('.*',\).*|\1|;N;s/\n/ /;s/$/ $(date)/;p}" The output is

'm/gf/MX3/122/d/status', 610,33000,28162,-33.836465,151.051189 Thu Feb 26 01:18:17 UTC 2015

  • Why is the terminal input relevant? Do you mean, what command should you type or is there reason a filter should read from the terminal itself to get this information rather than mosquito's stdout? Also, why do you think is the 86,6666...... appended line following a PUBLISH line in the second code block but not in the first one? Is a line like that not always going to follow a PUBLISH line? – mikeserv Feb 25 '15 at 5:05
  • @don_crissti -wilco; ALAS! mosquitto_sub -d -t +/# 2>&1 | sed -n "/PUBLISH/{s|.*\('.*',\).*|\1|;N;s/\n/ /;s/$/ $(date)/;p}" has worked I just need to understand what does what now :/ and why the others didn't work? – 3kstc Feb 26 '15 at 1:47
2

Well, rather confusing but anyway... Judging by the output of
mosquitto_sub -d -t +/# 2> >(grep PUBLISH)
your app seems to output to both stderr and stdout (otherwise you should only get lines matching PUBLISH in your output). It prints the debug messages (Sending... and Received...) to stderr and the actual data (810,5440995,6143...) to stdout.


Apparently, you need comma separated values so you could try the following, if you need the timestamp from the line matching PUBLISH:

mosquitto_sub -d -t +/# 2>&1 | xargs -d$'\n' -L1 sh -c 'date "+%s,$0"' | \
sed -n "/PUBLISH/{N;s|[ /]|,|g;s|^\([^,]*,\)[^']*'\([^']*\)',.*\n[^,]*,\(.*\)|\1\2,\3|;p}"

or, if you need the timestamp from the next line:

mosquitto_sub -d -t +/# 2>&1 | xargs -d$'\n' -L1 sh -c 'date "+%s,$0"' | \
sed -n "/PUBLISH/{N;s|[ /]|,|g;s|^[^,]*,[^']*'\([^']*\)',.*\n\([^,]*,\)\(.*\)|\2\1,\3|;p}"

2>&1 redirects stderr to stdout, the output is then piped to xargs which passes each line as an argument to the next command sh -c 'date "+%s,$0"' so each line is prepended with a timestamp+comma, e.g.:

[timestamp],Sending PINGREQ
[timestamp],Received PINGRESP
[timestamp],Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/status', ... (39 bytes))
[timestamp],871,40114,4536 4536 323

This is then piped to sed suppressing the automatic printing (-n).
For each line matching PUBLISH, append the Next line, replace each space and / with comma then via grouping, retain only the first or second timestamp, the values in between quotes and the values after the second timestamp and finally, print the result:

[timestamp],m,NWRL,TMX,098,d,status,871,40114,4536,4536,323
  • I've looked into xargs I am understanding it a bit. How can I shift the the time stamp between before \1\2? ie [timestamp],m/NWRL/TMX/098/d/SVlts, 806,3040421,7549 7750 3904 – 3kstc Mar 2 '15 at 3:02
  • it works :( but it endlines the output here – 3kstc Mar 2 '15 at 3:51
  • you've got every right to be angry. but the guy responsible of me, is currently overseas, and we can't really communicate, so everything is a 2-3 days delayed. I will abide by you, and make it into a CSV file, I will look into it when I go home... don_cristi, thank you for your help thus far.. :) – 3kstc Mar 2 '15 at 5:16
  • 1
    @ChuckClaunch - well, yeah, as I suspected, this is because of the pipe buffering - not much you can do other than use sed with -u (if you're using gnu sed that is) to force it unbuffer but that's all... – don_crissti Apr 6 '16 at 21:27
  • 1
    @ChuckClaunch - no problem buddy, anytime ! – don_crissti Apr 6 '16 at 21:31
0

I simulated mosquitto in bash/ksh environment and found

stub_mosquitto | 
   sed -e 's/.*m0, //' -e 's/, .*bytes))//' | while read line; do
      if [[ "${line}" = \'* ]]; then
         echo -n "${line}, "
      else
         echo "${line}, $(date)"
      fi
   done

I used sed here for cutting the interesting parts.
I use while read line for a simple way of joining 2 lines (and adding a date). This is possible with sed or tr, but that might need some work to (different versions of sed).

I do not understand the call to mosquitto_sub exactly, so I only show the solution when the strings come through a pipe:

  • What is the # doing there?
    Without quotes it should be the start of the comment
  • How do you get a line with numbers when you grep for a string?
    Are the numbers written to stderr? After redirecting them to stdout with 2>&1 grep will skip them.
  • would you be able to elaborate where and how I could implement this? (Apologies - totally new to Unix/programming) – 3kstc Feb 25 '15 at 22:27
  • Walter - why would you sed | read? – mikeserv Feb 26 '15 at 0:41
  • @mikeserv See updated answer with some explanation. – Walter A Feb 26 '15 at 7:54
  • It still doesn't make sense - do the join with sed - with read you only floor your efficiency and possibly mangle your input. – mikeserv Feb 26 '15 at 10:53
  • @mikeserv You are right. Joining lines with sed is much more efficient. I could have found the special syntax for removing \n on SO, I just tried to show a general approach in a "simple" way. – Walter A Feb 26 '15 at 21:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.