I have an Arduino Nano connected to a Raspberry Pi which streams text data (the output of a GY-91 sensor) to a listener on the Raspberry Pi.

P: 974.044 | T: 22.65 | AG: -0.17  -0.10   1.01 | GD:    1.16     0.91     0.98 | MT:  -59.0    94.4   105.1
P: 974.044 | T: 22.65 | AG: -0.17  -0.10   1.00 | GD:    1.34     0.85     1.04 | MT:  -58.3    94.7   105.1
P: 974.066 | T: 22.64 | AG: -0.17  -0.10   1.00 | GD:    0.98     1.10     1.04 | MT:  -58.5    94.9   104.2
P: 974.066 | T: 22.64 | AG: -0.17  -0.10   1.01 | GD:    0.98     0.85     1.16 | MT:  -59.2    94.2   104.2

The receiving program is little cpp program which just forwards the data to cout.

With ./a.out > log.txt I am able to dump the sensor data into the log.txt file.

The sample rate is about 50 lines per second (50 full readouts of the sensor), and I want to let this logging run for around 12 hours. Needless to say, this will create a relatively big log file.

How can I compress this data on the fly before it gets written into the log file?

Something like a.out | gzip > log.txt, but that does not work.

Ideally it's something which I can then download and "unzip" with a program on a Windows machine, but decompressing with some cygwin tool is also ok.

I intend to create just one such a log file in order to run different algorithms on it to decide which sort of aggregation/sumarization/analysis would work best on the data.


You can try something like:

./a.out | gzip -c - > log.gz

and you can decompress it with 7-zip But to avoid broken archive you should implement timer in to your program. If you interrupt it with Ctrl-C the archive will not be closed properly.

  • 1
    or have the program trap a SIGINT and close the archive safely. – DopeGhoti Jan 10 at 19:22
  • Thanks to both of you. Why, when I don't close the archive, will the file be empty? Is this stuff cached somewhere? In memory? – Daniel F Jan 10 at 20:51
  • @DopeGhoti How would I close the the archive from the cpp program when it is upstream from the pipe? – Daniel F Jan 10 at 20:53
  • @DopeGhoti Thanks, I think I won't have problems implementing the intercept of signals. The question is what do I do once I am in the handler? It's not the little program which is compressing the data. – Daniel F Jan 10 at 21:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.