0

Consider this sample 1MiB file:

$ dd if=/dev/zero of=input.img bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00100595 s, 1.0 GB/s

If I try to cat it and pipe it into dd without using iflag=fullblock I can't read more than 128KiB per block:

$ cat input.img | dd of=output.img bs=128k count=1
1+0 records in
1+0 records out
131072 bytes (131 kB, 128 KiB) copied, 0.000221117 s, 593 MB/s
$ cat input.img | dd of=output.img bs=129k count=1
0+1 records in
0+1 records out
131072 bytes (131 kB, 128 KiB) copied, 0.000495317 s, 265 MB/s
$ cat input.img | dd of=output.img bs=1M count=1
0+1 records in
0+1 records out
131072 bytes (131 kB, 128 KiB) copied, 0.000437209 s, 300 MB/s

Using 2 block counts it even prints a warning:

$ cat input.img | dd of=output.img bs=129k count=2
dd: warning: partial read (131072 bytes); suggest iflag=fullblock
0+2 records in
0+2 records out
262144 bytes (262 kB, 256 KiB) copied, 0.00107657 s, 243 MB/s

I thought it could be due to the pipe buffer size, however I checked its size following https://www.golinuxhub.com/2018/05/how-to-view-and-increase-default-pipe-size-buffer.html and it seems set to 65536 bytes:

$ mkfifo /tmp/testfifo
$ python2
>>> fifo_fd = open('/tmp/testfifo', 'rb+')
>>> import fcntl
>>> fcntl.fcntl(fifo_fd, 1032)
65536

(Note I couldn't make it work with Python3 I guess because of https://bugs.python.org/issue20074)

The main question is (pure curiosity): why do reads get truncated to 128KiB?

(Tested on Arch Linux, kernel 5.4.2, zsh 5.7.1.)

  • 1
    You have creatively interpreted the article you link to: what you're doing in your python snippet is trying to set the pipe size to its minimum value with fcntl(fifo_fd, F_SETPIPE_SZ, 0). If you want to get the default value, you should use fcnt(fifo_fd, F_GETPIPE_SZ). The default value is most certainly 65536, but it could be increased up to the value in /proc/sys/fs/pipe-max-size. – mosvy Dec 7 '19 at 10:40
  • As to why dd reads chunks of 128k bytes when you asked it for 129k, it's because a read(.., 129k) returned 128k. There's no guarantee whatsoever that a read won't return a shorter count than asked, and it's not a reasonable assumption than it will only rarely do that. The dd manpage says about the bs=BYTES argument: "read and write up to BYTES bytes", not "exactly BYTES bytes". – mosvy Dec 7 '19 at 10:46
  • @mosvy your claim is not correct, see what I posted 2 hours before you posted your comment. 128k is read because the program gcat only writes 128k and not more. – schily Dec 7 '19 at 17:14
  • @schily What claim is not correct? I've strace-d the OP's testcase, and it says read(0, ..., 132096) = 131072. And the second part is a literal quote from the dd(1) manpage. – mosvy Dec 7 '19 at 17:22
  • What do you expect from this test? The question was why it is not able to return more than 128k and this is of course a result from the write() size used by gcat. – schily Dec 7 '19 at 17:35
3

Any block size greater then PIPE_BUF (5k on modern UNIX, 4k on Linux) is not granted to pass a pipe without fragementation.

This means, it may be split, but there is no grant for a split.

Whether you are able to read more than PIPE_BUF at all depends on the memory state of the kernel that controls when the write side is stopped by the kernel to prevent it from eating all kernel memory. It also depends on the timing and scheduling of the read side and whether it gets a wakeup after a sufficient amount of data has been accumulated.

BTW: A dd option named iflag=fullblock is a vendor specific extension. Avoid to use it since it is non-standard.

Also note that gcat uses a write block size of 128k, while the UNIX version of cat is mmap() based and uses a write size of 8Mbyte. So if you did run this test on a genetic UNIX, you could be able to read larger blocks than 128k,

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.