0

Given multiple processes, forked from a parent process with shared file descriptors (representing STDOUT/STDERR), if one of the processes writes to STDOUT and exceeds the ~64K buffer it will block (as expected). On close of all of its shared file descriptors on other processes, the process unblocks and continues to write to STDOUT.

How does the act of closing shared file descriptors cause the write-blocked process to unblock? (I assume the buffer is being flushed, but I can not find evidence of that)

To repro, here are 2 scripts which setup the state. My goal is not to solve an issue, but to understand how the act of closing those descriptors causes the blocked process to continue. (ie. the intention is not to solve a Python or subprocess issue)

File: A.py

#!/usr/bin/env python2.6

import subprocess

if __name__ == "__main__":
   subprocess.Popen("./B.sh 70000", shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
   subprocess.Popen("./B.sh 100", shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
   subprocess.Popen("./B.sh 100", shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

File: B.sh

#!/usr/bin/env bash

for i in `seq 1 $1`; do
   echo -n "#"
done

echo ""

while true; do
    echo > /dev/null
done

Python 2.6's subprocess.Popen is used as a means to establish the state. Underneath it pipes and forks (maybe not in that order) the current process which duplicates its file descriptors for each forked process, creating a chain of processes with shared file descriptors.

The shell script B.sh simply outputs data to STDOUT and then loops (intentionally with no sleep so you can distinguish between running and sleeping states in something like htop).

Put both scripts in the same working directory and execute A.py to replicate the behavior (CentOS 6.7, but I would suspect any CentOS 6.X version to repro).

Here is a directory listing of the processes file descriptors to demonstrate the shared state, for reference:

# Process 1: ./B.sh 70000
ls -la /proc/4144/fd
total 0
lrwx------ 1 root root 64 Sep 28 14:18 0 -> /dev/pts/0
l-wx------ 1 root root 64 Sep 28 14:18 1 -> pipe:[53061]
l-wx------ 1 root root 64 Sep 28 14:18 2 -> pipe:[53061]
lr-x------ 1 root root 64 Sep 28 14:18 255 -> /root/B.sh

# Process 2: ./B.sh 100
ls -la /proc/4145/fd
total 0
lrwx------ 1 root root 64 Sep 28 14:18 0 -> /dev/pts/0
l-wx------ 1 root root 64 Sep 28 14:18 1 -> pipe:[53063]
l-wx------ 1 root root 64 Sep 28 14:18 2 -> pipe:[53063]
lr-x------ 1 root root 64 Sep 28 14:18 255 -> /root/B.sh
lr-x------ 1 root root 64 Sep 28 14:18 3 -> pipe:[53061]

# Process 3: ./B.sh 100
ls -la /proc/4146/fd
total 0
lrwx------ 1 root root 64 Sep 28 14:18 0 -> /dev/pts/0
l-wx------ 1 root root 64 Sep 28 14:18 1 -> pipe:[53065]
l-wx------ 1 root root 64 Sep 28 14:24 10 -> pipe:[53065]
l-wx------ 1 root root 64 Sep 28 14:18 2 -> pipe:[53065]
lr-x------ 1 root root 64 Sep 28 14:18 255 -> /root/B.sh
lr-x------ 1 root root 64 Sep 28 14:18 3 -> pipe:[53061]
lr-x------ 1 root root 64 Sep 28 14:18 4 -> pipe:[53063]

The first process spawned (Process 1) outputs ~>64K of data to STDOUT causing it to enter a sleep state (evident via htop and attaching strace to the pid) due to it being blocked on write.

The second and third processes (Process 2 and Process 3 respectively) will remain in a running state, and have duplicate file descriptors which refer to the pipe(s) established as part of Process 1.

Kill one of Process 2 or Process 3 and Process 1 remains sleeping, kill both and Process 1 unlocks (why?) and enters a running state.

Restart the test and use gdb to attach to Process 2 or Process 3 and then close p close(#) the file descriptor shared with Process 1 and Process 1 remains in a sleep state. Attach to the other process and close the shared descriptor and Process 1 unblocks and enters a running state.

So the act of closing all shared descriptors with the blocked process cause it to unblock. What causes this process, which was previously write-blocked, to free up in this circumstance?

0

Once the read end of the pipe is closed, attempting to write is an error. It will kill the process with SIGPIPE. Or if that signal is blocked, the write will return immediately with errno == EPIPE. That should explain your behaviour. It is one of the original features of UNIX pipes.

It happens when the last remaining reference to the read end of the pipe is closed. There can be other references e.g. from dup().

In your case, you have fork()ed a new process, so the child process starts off with all the same file descriptors. For the pipe to be closed, both the parent's and the child's file descriptors must be closed. Note that close() in the parent does not affect the child's file descriptors (or vice versa).

This an example of the general concept of reference counting. The kernel keeps count of how many file descriptors refer to the read end of the pipe. It reduces the count by one for each close() call. If the count falls to zero, the kernel runs the appropriate cleanup function. In the Linux kernel this is a function pointer called .release, since it releases all the associated resources.

The reference counting system is essential to UNIX file descriptors. For example I can find dup() and fork() used in research UNIX V5.


If you want to know why SIGPIPE is blocked in subprocesses started from python2.6, see https://bugs.python.org/issue1652 .

If you were surprised that the pipe FDs leak into P2 and P3, see https://bugs.python.org/issue7213. I.e. to get more sensible behaviour from Popen(), you can pass close_fds=True.

Otherwise, if you want to pass specific extra FDs into P2 and P3, I would really want to make that explicit by using the pass_fds parameter.

I will assume that you do want to, otherwise I don't really see what this example program is supposed to be doing. You are discarding the subprocess objects, and then exiting. So the parent process is closing its pipe FDs, at least when it exits.

We can reproduce this in shell, without relying on details which seem liable to depend on the specific version of python.

$ strace -f sh -c 'cat </dev/zero | { sleep 1& sleep 2& }'
...
[pid 26477] read(0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 131072) = 131072
[pid 26477] write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 131072 <unfinished ...>
...
[pid 26480] nanosleep({tv_sec=2, tv_nsec=0},  <unfinished ...>
...
[pid 26479] nanosleep({tv_sec=1, tv_nsec=0}, NULL) = 0
...
[pid 26479] +++ exited with 0 +++
[pid 26480] <... nanosleep resumed> NULL) = 0
...
[pid 26480] +++ exited with 0 +++

[pid 26477] <... write resumed> )       = 65536
[pid 26477] --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=26477, si_uid=1001} ---
[pid 26477] +++ killed by SIGPIPE +++

I notice one detail here that I had not thought of before. write() is returning that it successfully wrote only 64K into the pipe buffer. What would happen if the caller had disabled the default termination action for SIGPIPE? "short writes" are something that you often have to tolerate on pipes or sockets, by retrying. E.g. this can happen if the process received an unrelated signal, and there was a handler function set up for that signal. So, the caller should continue by retrying write() with the remaining data, and that write() call would immediately return with errno == EPIPE.

  • The exit is intentional just to setup the state of the file descriptors, add an import time and time.sleep(600) as the last line and you'll see the same sleep/running state issue described earlier. – unscannable Sep 28 '18 at 17:24
  • @unscannable I'm still not inclined to debug behaviour that depends on the exact garbage collection behaviour. For all I know you get a GC when you sleep, and it closes the pipes. – sourcejedi Sep 28 '18 at 17:25
  • I'm not expecting troubleshooting of the Python issue. I'm looking for a specific behavior in regards to how a write blocked process will unblock when its shared file descriptor is closed in the shared process. Python is used to establish the state, I am already aware of why Python "leaks" the file descriptors. This isn't about debugging and achieving a working state, I'm interested in what happens during close(FD) that ultimately unblocks the blocked process which is write-blocked due to exceeding the write buffer – unscannable Sep 28 '18 at 17:28
  • 1
    I wish I had reputation to move this to chat :-).We all have differing definitions of simple :-), I do not mean to offend by describing it as simple from my perspective. When P1 enters the sleep state (~65K #'s in the loop), the strace shows a partial write statement write(1, "#", 1 and hangs. The pipe is indeed broken by the parent Python process before this point, as evident in the strace (-EPIPE on write()) having closed it. (More in next comment since it exceeds the limit) – unscannable Sep 28 '18 at 18:17
  • 1
    This would be related more to Linux, I tagged as file-descriptors as that's what the specific question is in regards to, maybe this is kernel related? However the help text for using the linux tag instructed not to use it for general linux and to just include the distro in the body, which I did, CentOS 6.7. This behavior may be kernel specific (details to repro to help achieve the test state, including versions were included in the body) but the kernel is specifically 2.6.32-573.el6.x86_64. ty for the time you've spent already and directing me to source (I'll search in 2.6.32 source) – unscannable Sep 28 '18 at 20:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.