I have 2 applications:
- Producer (N instances)
- Consumer (1 instance)
I currently write out intermediate results from the producers, and then the consumer reads these files from disk and produces a final result.
I would like to minimize this I/O by "streaming" the outputs from the producers directly to the consumer.
I came across named pipes (mkfifo) and a subsequent example here. This looks great, but what I can't determine is how this is actually implemented? Is the FIFO queue just being buffered through a file? If so, that probably wouldn't help me. I would like for the contents to stream "through memory" entirely without utilizing the disk. Maybe this isn't possible across processes?