1

I have an Arduino Nano connected to a Raspberry Pi which streams text data (the output of a GY-91 sensor) to a listener on the Raspberry Pi.

P: 974.044 | T: 22.65 | AG: -0.17  -0.10   1.01 | GD:    1.16     0.91     0.98 | MT:  -59.0    94.4   105.1
P: 974.044 | T: 22.65 | AG: -0.17  -0.10   1.00 | GD:    1.34     0.85     1.04 | MT:  -58.3    94.7   105.1
P: 974.066 | T: 22.64 | AG: -0.17  -0.10   1.00 | GD:    0.98     1.10     1.04 | MT:  -58.5    94.9   104.2
P: 974.066 | T: 22.64 | AG: -0.17  -0.10   1.01 | GD:    0.98     0.85     1.16 | MT:  -59.2    94.2   104.2

The receiving program is little cpp program which just forwards the data to cout.

With ./a.out > log.txt I am able to dump the sensor data into the log.txt file.

The sample rate is about 50 lines per second (50 full readouts of the sensor), and I want to let this logging run for around 12 hours. Needless to say, this will create a relatively big log file.

How can I compress this data on the fly before it gets written into the log file?

Something like a.out | gzip > log.txt, but that does not work.

Ideally it's something which I can then download and "unzip" with a program on a Windows machine, but decompressing with some cygwin tool is also ok.


I intend to create just one such a log file in order to run different algorithms on it to decide which sort of aggregation/sumarization/analysis would work best on the data.

1 Answer 1

4

You can try something like:

./a.out | gzip -c - > log.gz

and you can decompress it with 7-zip But to avoid broken archive you should implement timer in to your program. If you interrupt it with Ctrl-C the archive will not be closed properly.

7
  • 1
    or have the program trap a SIGINT and close the archive safely.
    – DopeGhoti
    Jan 10, 2019 at 19:22
  • Thanks to both of you. Why, when I don't close the archive, will the file be empty? Is this stuff cached somewhere? In memory?
    – Daniel F
    Jan 10, 2019 at 20:51
  • @DopeGhoti How would I close the the archive from the cpp program when it is upstream from the pipe?
    – Daniel F
    Jan 10, 2019 at 20:53
  • stackoverflow.com/a/1641223/1421075
    – DopeGhoti
    Jan 10, 2019 at 20:56
  • @DopeGhoti Thanks, I think I won't have problems implementing the intercept of signals. The question is what do I do once I am in the handler? It's not the little program which is compressing the data.
    – Daniel F
    Jan 10, 2019 at 21:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .