cat
is blocking from what I see in the code, i.e. it uses blocking read()
and then it uses blocking write()
.
I want to call some tool where I disable all stdout buffering by purpose (e.g. like described here), because that tool might call subprocesses and I want that all writings from all subprocesses on stdout will occur at the same time when they are written at the same time.
Then, I want to pipe that stdout into a multi-threaded version of cat
(or so). The purpose is that the real stdout will be slow (it's a file on disk) but I don't want the tool to hang when it tries to write on stdout. If I would just do
stdbuf -oL mytool
then it would hang when the disk is busy or so. When I do
stdbuf -oL mytool | cat
I'm actually not perfectly sure what will happen. I might get some additional buffering by the kernel pipe buffer, although I think that will not be used when I disable the stdout buffering of mytool
. And then the stdout of cat
will also be buffered by default, but at the time when cat
really writes out its stdout, it might hang. mytool
will hang when it writes something and cat
is not reading at the same time.
That is why I'm searching for a multi-threaded cat
which reads at the same time when it writes, so writing to the stdin of multi-threaded-cat
will never be blocking (or only soft-blocking or whatever you would call that). It basically introduces another buffer in user-space in multi-threaded-cat
. When multi-threaded-cat
hangs while trying to write to stdout, it does not matter because it will still read from stdin in parallel.
So I want to do:
stdbuf -oL mytool | multi-threaded-cat
I want that multi-threaded-cat
consumes the incoming data always as soon as possible. That is why I think that it probably should be multi-threaded. Otherwise, if it uses write(), that might block or at least give a small hickup and in the mean-while, it cannot read() from stdin.
I also want that multi-threaded-cat
writes out the data as soon as possible. So it should not first fill its own buffer and then write, I want that it writes always immediately.
My use case is this: mytool, including some subprocesses, will write some logging information on stdout. stdbuf
is important so that I don't get any delay in the output and also that all subprocesses stdouts are synced. All the stdout will be redirected to a log file on a fileserver which is a bit slow and it will greatly reduce the performance when it would wait for all writes to complete. So that is why I want something like multi-threaded-cat
in between.
Is there such a tool?
I just implemented my own such tool here. Using this already gives me a speedup of 800%, compared to not using it. But maybe there are other tools or other ways to do what I want?