I am writing a data logging app, all programs are started like:
./program > out.bin
The data collector periodically pools the stdout output files and reads the data.
The issue is that the IO streams are buffered and if some program outputs data in like 1 byte per second, it takes a lot of time (up too 4k seconds with default 4kB buffer size) before the data are actually flushed out.
My question is how to force the stdout/pipe/printf buffer to flush externally, i.e. call externally something like
I have read various sites like Turn off buffering in pipe , but I can not disable the buffers as it have huge IO performace impact (measured).
I am looking for high performace solution for production and these following conditions are always met:
- the program (data producer) PID is always known
- the output is always a file with known path
- the data logging process has full root access