Insert Newlines into serial stream before writing to text file

Question

I have an RFID reader which sends the Unique ID of the RFID tag to the serial port of my linux (raspberry pi) box every time it gets a successful read. The problem is that it is sending it without any newline characters. Here's an example of the output:

root@scalepi:~/scale# ./grabserial -d "/dev/ttyUSB4" -b 9600 
$A0112OKD9990001200005643E#$A0112OKD9990001200005643E#$A0112OKD9990001200005643E#$A0112OKD9990001200005643E#$A0112OKD9990001200005643E#$A0112OKD9990001200005643E#$A0112OKD9990001200005643E#$A0112OKD9990001200005643E#$A0112OKD9990001200005643E#$A0112OKD9990001200005643E#$A0112OKD9990001200005643E#$A0112OKD9990001200005643E#

I'd like to write each entry to a text file with a new line, like this:

$A0112OKD9990001200005643E#
$A0112OKD9990001200005643E#
$A0112OKD9990001200005643E#

As you can see each entry begins with $ and ends with #

I have tried just piping the outout to sed using this line to add a new line after the #, but I get zero output, because I think sed is looking for a new line before it processes the data:

./grabserial -d "/dev/ttyUSB4" -b 9600 | sed 's/#/#\r\n/g'

If I redirect the output to a file, and then cat the file, piped through that same sed command, I get what I expect, but I need to be able to write the data to a file in this format, so I can read it in realtime.

root@scalepi:~/scale# ./grabserial -d "/dev/ttyUSB4" -b 9600 > /tmp/test2.txt
^C
root@scalepi:~/scale# cat /tmp/test2.txt | sed 's/#/#\r\n/g'
$A0112OKD9990001200005643E#
$A0112OKD9990001200005643E#
$A0112OKD9990001200005643E#
root@scalepi:~/scale# 

If anyone has any suggestions for me, I'd appreciate it. Thanks!

EDIT - UPDATED

After trying the "tr" solution below, I was able to get real time output, but if I try to redirect via pipe, or via stdout/stderr redirection, I get buffered results of 50-60 lines at a time instead of getting a line by line result.

The following command will give me real time output as it comes in on the serial port:

./grabserial -d "/dev/ttyUSB4" -b 9600 | tr '#' '\n'

The following command will give me output in chunks of 50 or 60 lines at a time (not real time):

./grabserial -d "/dev/ttyUSB4" -b 9600 | tr '#' '\n' | sed -u 's/$/#\r/g'

The following command will give me output to a text file in chunks of 50 or 60 lines at a time (not real time):

./grabserial -d "/dev/ttyUSB4" -b 9600 | tr '#' '\n'  > /tmp/test1.txt

FWIW, I get the same results using cat to read the serial port, instead of "grabserial".

The following command gives me realtime results:

cat /dev/ttyUSB4 | tr '#' '\n' 

But when I try to redirect output via either pipe or stdout redirection, I get buffered chunks of data in the text file:

cat /dev/ttyUSB4 | tr '#' '\n' > /tmp/test4.txt

Answer 1

You can translate (or transliterate) the # characters into newlines with

 tr '#' '\n'

so

./grabserial -d "/dev/ttyUSB4" -b 9600 | tr '#' '\n'

should give you

$A0112OKD9990001200005643E
$A0112OKD9990001200005643E
$A0112OKD9990001200005643E
$A0112OKD9990001200005643E
$A0112OKD9990001200005643E
$A0112OKD9990001200005643E
$A0112OKD9990001200005643E
$A0112OKD9990001200005643E
$A0112OKD9990001200005643E
$A0112OKD9990001200005643E
$A0112OKD9990001200005643E
$A0112OKD9990001200005643E
            ︙

in real time.  If you need the pound signs and/or the carriage returns, you can pipe the above into

        … sed 's/$/#\r/'

(using the fact that sed (and regular expressions in general) use $ to represent the end of a line.

Gotcha

You may have a problem that tr buffers its output when that output is going to a pipe or a file.  Perhaps the best way to handle this is to follow αғsнιη’s advice to turn off buffering with stdbuf; e.g.,

./grabserial -d "/dev/ttyUSB4" -b 9600 | stdbuf -oL tr '#' '\n' | sed 's/$/#\r/' > RFIDs.txt

In case it’s unclear, the option to stdbuf is lower-case o (oh) and upper-case L (ell), which says that the standard output should be line-buffered.

Bottom line (TL;DR) #1:

So you should be able to use

./grabserial -d "/dev/ttyUSB4" -b 9600 | stdbuf -oL tr '#' '\n' > RFIDs.txt

to get the entries without the # at the end, or

./grabserial -d "/dev/ttyUSB4" -b 9600 | stdbuf -oL tr '#' '\n' | sed 's/$/#/' > RFIDs.txt

to get the entries with the # at the end, but without the \r, or

./grabserial -d "/dev/ttyUSB4" -b 9600 | stdbuf -oL tr '#' '\n' | sed 's/$/#\r/' > RFIDs.txt

to get the entries with the # and the \r at the end.

Bottom line (TL;DR) #2:

You haven’t said how you’re terminating the command.  If you’re typing Ctrl+C, that might be causing a problem¹.  If so, try:

• putting one of the above command lines into the background with &, and then
• when you’re ready to quit, do a ps and get the PID of the grabserial process, and
• kill it with the kill command.

Bottom line (TL;DR) #3:

You say you need to be able to read the data in real time, without going into specifics.  If you have some program / script that is capable of reading from stdin, try

./grabserial -d "/dev/ttyUSB4" -b 9600 | stdbuf -oL tr '#' '\n' | sed 's/$/#\r/' | tee RFIDs.txt | your_program 

This should write the data to the output file while simultaneously piping it to your program.

Bottom line (TL;DR) #4:

If, in fact, cat works as well as grabserial, then you don’t actually need either one; things like

stdbuf -oL tr '#' '\n' < /dev/ttyUSB4 | sed 's/$/#\r/' > RFIDs.txt

should work just as well.
____________
¹ Based on your update, this is probably not your problem, but I’m leaving this here in case it helps somebody else.

---

Note that tr basically operates on a one-to-one basis.  (There are some options to do some slightly fancier stuff.)  So, for example,

tr 'abc…' 'ABC…'

will replace a with A, b with B, c with C, etc.  So you can’t do something like

tr '#' '#\r\n'

Answer 2

Because sed normally buffering its output until a buffer size fills up ( Probably 4K bytes), you could use -u as GNU specific option to unbuffer the output.

Please see Turn off buffering in pipe and related answer for sed.