In Advanced Programming in Unix Environment 3th, 3.9, I/O Efficiency, I read about this:

The file was read using the program shown in Figure 3.5, with standard output redirected to /dev/null. The file system used for this test was the Linux ext4 file system with 4,096-byte blocks. (The st_blksize value, which we describe in Section 4.12, is 4,096.) This accounts for the minimum in the system time occurring at the few timing measurements starting around a BUFFSIZE of 4,096. Increasing the buffer size beyond this limit has little positive effect.

My question is why "Increasing the buffer size beyond this limit has little positive effect"? I think increasing the buffer size will definitely reduce the user cpu time and system cpu time because of the reduce of number of loops so that the clock time will also be reduced to certain degrees, Isn't that so? And why?

  • Comments are not for extended discussion; this conversation has been moved to chat. – terdon Apr 6 '17 at 16:12

The statement "Increasing the buffer size beyond this limit has little positive effect" has no credibility.

The code as posted gives no indication whatsoever as to how many bytes are actually read then written in each loop iteration - via data read from a pipe - redirected stdin. Given that the Linux PIPE_BUF value is normally 5120 bytes, the code probably reads and writes a handful of kilobytes with each loop iteration.

Once the buffer size grows larger than that, the number of bytes actually moved with each loop iterations doesn't change, so the buffer size is completely irrelevant.

Not only that, the methodology of the test is completely undocumented. How are the files passed to the process? The book pages posted at https://www.dropbox.com/s/r67nacyrqb5ulww/apue_72-73.pdf?dl=0 don't specify - at all. There's no way to duplicate the test because we can't tell what the test was.

Furthermore, perusal of the code at http://www.apuebook.com/src.3e.tar.gz indicates numerous problems - read() and write() are coded as if they return int instead of the correct ssize_t, signal handlers make calls to async-signal-unsafe functions.

In other words, slipshod code and slipshod tests.

  • The use case is ./mycat <file >/dev/null - no pipe is involved. – Satō Katsura Apr 6 '17 at 15:33
  • @SatoKatsura Based on what? That's not documented in the pages you posted. And it doesn't really matter - the code as posted doesn't account for how many bytes are read then written with each loop iteration, so the data is useless anyway. – Andrew Henle Apr 6 '17 at 15:38
  • Based on what? - Based on the fact that that's the only setup that makes sense that involves st_blksize? – Satō Katsura Apr 6 '17 at 15:44
  • @SatoKatsura the only setup that makes sense Again, no. And ext4 uses de facto variable-sized blocks: "To reduce performance difficulties due to fragmentation, the block allocator tries very hard to keep each file's blocks within the same group, thereby reducing seek times... With the default block size of 4KiB, each group will contain 32,768 blocks, for a length of 128MiB." So the "4k" ext4 block size likely has little to nothing to do with the results posted. Still useless. – Andrew Henle Apr 6 '17 at 19:25
  • My point: if it were a pipe the book wouldn't have said anything about st_blksize. In that sense, ./mycat <file >/dev/null seems to be the only setup that makes sense and matches the description in the book. – Satō Katsura Apr 6 '17 at 19:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.