Why is `BrokenPipeError` dependent on the size of the piped stream?

Question

The following script raises BrokenPipeError: [Errno 32] Broken pipe, when piped to a command like head (unless the number of lines to head exceeds the number of lines printed by the Python script).

for i in range(16386):
    print("")

$ python test-pipe.py | head -1

Traceback (most recent call last):
  File "test-pipe.py", line 2, in <module>
    print("")
BrokenPipeError: [Errno 32] Broken pipe

My understanding (from this answer and the answers to this question) is that an error is raised if a pipe is closed before the Python process has finished writing to it.

However, if I decrement the iterated range with one to 16385, the error is not raised (not sure if this threshold is the same on al machines so maybce just try a high and low number to reproduce). I initially thought this might be related to the pipe buffer size, but that is 64K for me (according to M=0; while printf A; do >&2 printf "\r$((++M)) B"; done | sleep 999, so that doesn't seem to be the reason.

Why is the occurrenceBrokenPipeError dependent on the size of what is being piped?

This is with Python 3.8.1 on Linux 5.4.15-arch1-1.

Answer 1

Why is the occurrenceBrokenPipeError dependent on the size of what is being piped?

Because writing more stuff takes more time, and the right side of the pipeline may die before your python has finished writing it. Also, if the python tries to write more than it fits in the pipe buffer, it will block and give the head -1 ample time to exit.

Since the head -1 takes some time to live and die, the python may use that time to write all its stuff --if it fits in the pipe buffer-- and exit successfully. That's hard to predict, since the kernel is free to schedule both sides of the pipeline in any order, and may delay starting the head -1 as long as it sees fit, or it may stop it at any time.

>>> python3 -c 'for i in range(50000): print("")' | sleep .01
Traceback (most recent call last):
  File "<string>", line 1, in <module>
BrokenPipeError: [Errno 32] Broken pipe

>>> python3 -c 'for i in range(50000): print("")' | sleep .1

# OK!

But if python tries to write more stuff than it fits in the pipe, it will inexorably get an EPIPE or SIGPIPE in the end, no matter how much time it has:

>>> python3 -c 'for i in range(100000): print("")' | sleep 20
Traceback (most recent call last):
  File "<string>", line 1, in <module>
BrokenPipeError: [Errno 32] Broken pipe

but that is 64K for me ... so that doesn't seem to be the reason.

Keep in mind that python is using full buffering when the output is not a terminal; it doesn't write line by line or byte by byte, but by chunks of some size:

>>> strace -s3 -e trace=write python3 -c 'for i in range(50000): print("")' | sleep .3
write(1, "\n\n\n"..., 8193)             = 8193
write(1, "\n\n\n"..., 8193)             = 8193
write(1, "\n\n\n"..., 8193)             = 8193
write(1, "\n\n\n"..., 8193)             = 8193
write(1, "\n\n\n"..., 8193)             = 8193
write(1, "\n\n\n"..., 8193)             = 8193
write(1, "\n\n\n"..., 842)              = 842
+++ exited with 0 +++