Flush the pipe/printf buffers externally for already running process with known PID

Question

I am writing a data logging app, all programs are started like:

./program > out.bin

The data collector periodically pools the stdout output files and reads the data.

The issue is that the IO streams are buffered and if some program outputs data in like 1 byte per second, it takes a lot of time (up too 4k seconds with default 4kB buffer size) before the data are actually flushed out.

My question is how to force the stdout/pipe/printf buffer to flush externally, i.e. call externally something like fflush(stdout).

I have read various sites like Turn off buffering in pipe , but I can not disable the buffers as it have huge IO performace impact (measured).

I am looking for high performace solution for production and these following conditions are always met:

• the program (data producer) PID is always known
• the output is always a file with known path
• the data logging process has full root access

Answer 1

gdb -p PID -batch -ex 'p fflush(stdout)'

As with any debugging and hacking, YMMV.

Answer 2

Do you have access to the source of the running programs?

Forcing a flush of an arbitrary executable is, while not theoretically impossible, very difficult. You would need to find the fflush function in the code and the stdout argument, then interrupt the execution of the program, arrange for the call to fflush, then continue execution. If the program is using a shared library, that at least makes the first part easier, finding fflush and stdout, but you still need to simulate the call. Also, for an unknown binary, you can't know whether it used stdio or whether it implements its own buffering mechanism. Knowing the path of the output file will not help you.

You can try to use gdb, then use the attach command to attach to the process. Maybe gdb can call the fflush function.

If you have the source to the program, implement a signal handler that flushes the buffer and just send the signal when you want the buffer flushed.

You can try a pipe, maybe the programs don't buffer if the output is a pipe. Change it to

./program | your_program > out.bin

Your program can accept the input, buffer it and flush the buffer when it receives a signal. It would still add CPU overhead, but not disk overhead.