I have a Windows command-line program that I'm running in a Bash script in Ubuntu via wine. The Bash script basically looks like this:

wine myprogram.exe | while read line
   # Process line

Now, since I've written myprogram.exe I know for a fact that it just spits out data as fast as it can. Can anyone explain to me how the Bash while loop is able to process the data in case my program spits it out faster than the while loop can handle? Is there some sorcery going on behind the scenes where the kernel scheduler will make myprogram.exe sleep if it produces too much data? Anyone? Currently I'm leaning towards it being black magic.


First, the program may do its own output buffering. This is sometimes called “stdio buffering” after the name of the library component that performs this task in C: the functions like putc, fputs, fprintf, etc., declared in stdio.h. If it does, it will produce output in bursts, typically of a few kilobytes (when the output is a terminal, the default is to flush the buffer at each newline character).

At some point, either the programmer or the underlying library function calls write explicitly. This requests that the kernel write the specified data into the pipe. The kernel may decide to write all or part of the data. Since the file is a pipe, the kernel will copy the data into the pipe's buffer area. If the pipe buffer is full, then the write system call blocks until there is room: your program (or more precisely, the thread that called write, in case there are several kernel-level threads) will not resume execution until the call to write returns.

(It is possible, but unlikely in this situtation, that the program has set the pipe's file descriptor as non-blocking. If this is the case, if the kernel determines that it can't copy any data, it will return control to the program; the write system call returns 0. A program that makes such non-blocking system calls would typically call select or poll or epoll in a loop to block until one of the file descriptors it's communicating on is ready for input or output.)

The fact that the program is blocked during a system call is not related to a choice of scheduling algorithm. At its core, any scheduler distinguishes between ready threads, which can be given CPU time, and waiting threads, which cannot. The gist of a scheduler is to choose a ready thread, and let it run until either the thread makes a system call (which puts the thread into a waiting state) or some asynchronous event occurs (in practice, a processor interrupt). During the processing of a system call, it may be that a thread that was until then blocked becomes ready, for example because that thread was in a write call and the kernel has now been able to deliver the data from that call. A few things can make a ready thread blocked from the outside, for example a signal to pause (SIGSTOP). The scheduler maintains some kind of ready list to decide which thread to schedule next: a list of threads that are ready (it is usually a lot more complicated than a simple list in a real-world scheduler).

|improve this answer|||||

It's called a pipe. Run man 7 pipe and you will learn all about how pipes work.

Edited to add: In brief, the wine command's output is collected in a buffer. When the buffer is full and wine attempts to write more, the wine process is stopped by the kernel. When space becomes available in the buffer (ie. when the while loop reads from it), the stopped process is reawoken and it is allowed to continue. This is the classic bounded-buffer algorithm, but in the Unix/Linux world it's usually called a pipe. It's very popular and powerful, and I recommend learning all about it.

|improve this answer|||||
  • 1
    I don't want to be rude, but "RTFM" is the sort of response I'd expect on Slashdot. Granted, RTFM is probably an answer that works on most questions with zero effort required. – Magnus Oct 19 '12 at 18:13
  • @Magnus Oh, I'm sorry. I didn't mean this to come off as a gruff RTFM. The question seeks an explanation and the explanation that is most relevant to the questioner is handily available in that man page, so it seemed to be the optimal answer. – aecolley Oct 19 '12 at 18:23
  • Ok, I've read the man page now, and it seems that the behavior depends on whether the O_NONBLOCK flag was set or not. I guess I will just run the script and see if it blows up. – Magnus Oct 19 '12 at 18:27
  • 1
    If you don't want wine to block without losing data, you can install a program called buffer and run it in the pipeline with any buffer size your system will handle. Obviously, with enough output you'll eventually have to compromise and either lose writes or allow wine to block; but buffer will let you stretch it out and survive larger bursts of output. – aecolley Oct 19 '12 at 18:36
  • It's running right now with I guess the standard buffer size of 64K and seems to work so far... I have 16 GB of memory on this box, so if it fails I will try your suggestion =) – Magnus Oct 19 '12 at 18:43

Nah, no black magic (well not this time at least). It just uses your cache and buffers in the output stream.

|improve this answer|||||
  • And what happens when the buffer is full? – Magnus Oct 19 '12 at 18:14
  • Don't you mean: Yeah, black magic, otherwise known as a buffer. @Magnus, when the receiving buffer is full, attempts to write to stdout in Unix block. I don't know anything about Windows programming, so I can't say how wine passes that on to myprogram.exe. – dubiousjim Oct 19 '12 at 18:20
  • I'm not a Linux programming guru... when you say that the write blocks, does that mean that the kernel notes that the program is trying to write to a full stdout buffer, and will then suspend the thread and only wake it up once the pipe is no longer full? – Magnus Oct 19 '12 at 18:33
  • 1
    @Magnus In *nix a program writing to stdout is the same as writing to a file. When the application prints output, it is making a write() system call to file descriptor 1. The operation will not complete until write() is finished, at that point it is "blocked". – jordanm Oct 19 '12 at 19:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.