dd: `iflag=nocache` ineffective for the first usage

Question

The flag iflag=nocache of dd seems to take effect only at the second time onwards.
I don't understand why.
I was thinking maybe it is related to readahead cache, but it doesn't explain the results in the table below:

Results - Effect of iflag=nocache: from big SATA SSD file to /dev/null:
Test_Name                dd_flags       Time[s]  time/dd exit status
w/ iflag=nocache         iflag=nocache  6.95     0
again w/ iflag=nocache   iflag=nocache  6.94     0
first w/o iflag=nocache  -              6.94     0
2nd w/o iflag=nocache    -              2.60     0
3nd w/o iflag=nocache    -              2.49     0
first w/ iflag=nocache   iflag=nocache  2.71     0
2nd w/ iflag=nocache     iflag=nocache  6.94     0
3rd w/ iflag=nocache     iflag=nocache  6.94     0

Regarding the bottom 3 tests/rows:

• The first time that iflag=nocache is used, then still the cache is being used (due to the fast time it took).
• The last two attempts with iflag=nocache took a long time (i.e. w/o cache), which means they probably didn't use the readahead cache, so probably the readahead cache isn't the issue here.
• The iflag=nocache flag seemingly works, because that the subsequent attempts to use it are much slower, meaning cache isn't being used.

Note:

• The column dd_flags in the table above shows all flags which were provided to dd, except the two flags if and of. For rows which have - for the dd_flags column, it means that dd didn't have additional flags (except if and of)
• The input file to dd is a big file (3.8GB) which is stored in SATA SSD (max throughput about 500MB).
• The output file for dd is /dev/null.
• The Time[s] column in the table above is the Real time which is a result of the time reserved word of Bash.
• I don't perform sync because there is no need - file was written before PC was booted.
• I don't empty the OS pagecache (with echo 1 >|/proc/sys/vm/drop_caches) because I want to avoid a complete cache invalidation.

I tried dropping the cache for the whole file, as indicated by info dd - seemed to work:

Results - Effect of iflag=nocache: from big SATA SSD file to /dev/null:
Test_Name                  dd_flags               Time[s]  time/dd exit status
w/ iflag=nocache           iflag=nocache          6.95     0
again w/ iflag=nocache     iflag=nocache          6.95     0
first w/o iflag=nocache    -                      6.94     0
2nd w/o iflag=nocache      -                      2.60     0
3nd w/o iflag=nocache      -                      2.49     0
Drop cache for whole file  iflag=nocache,count=0  0.14     0
first w/ iflag=nocache     iflag=nocache          6.94     0
2nd w/ iflag=nocache       iflag=nocache          6.94     0
3rd w/ iflag=nocache       iflag=nocache          6.94     0

Version:

$ dd --version | head -1
dd (coreutils) 8.32

Related readings which didn't solve my issue:

dd "direct" versus "nocache"

https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html#dd-invocation

Answer 1

That's because iflag=nocache drops caches… after it's been read, not before.

You can see this if you use strace:

# strace dd iflag=nocache … |& grep -Pi 'read|dontneed'
[…]
read(0, "\0"..., 65536) = 65536
read(0, "\0"..., 65536) = 65536
fadvise64(0, 393216, 131072, POSIX_FADV_DONTNEED) = 0
read(0, "\0"..., 65536) = 65536
read(0, "\0"..., 65536) = 65536
fadvise64(0, 524288, 131072, POSIX_FADV_DONTNEED) = 0
read(0, "\0"..., 65536) = 65536
read(0, "\0"..., 65536) = 65536
fadvise64(0, 655360, 131072, POSIX_FADV_DONTNEED) = 0
[…]

Each read is followed up with a POSIX_FADV_DONTNEED the range of which covers what had previously been read.

It does not match the reads perfectly as POSIX_FADV_DONTNEED requires different alignment than the blocksize used by dd. But in the end it still drops the caches of the previously read blocks.

So if your intention is to read uncached data, this iflag will not help you.(*)

Instead the purpose is to not cache what had been read, and also not displace other things that had been cached previously. It is a way to copy or stream large files without disturbing existing caches too much.

man 2 posix_fadvise describes it as such:

POSIX_FADV_DONTNEED
The specified data will not be accessed in the near future.

POSIX_FADV_DONTNEED attempts to free cached pages associated with the specified region. This is useful, for example, while streaming large files. A program may periodically request the kernel to free cached data that has already been used, so that more useful cached pages are not discarded instead.

Requests to discard partial pages are ignored. It is preferable to preserve needed data than discard unneeded data. If the application requires that data be considered for discarding, then offset and len must be page-aligned.

So it's for data that has already been used (already been read by dd) but not to be kept in cache so as to not discard more useful things instead.

---

(*)

info dd lists this example as a way to drop caches for the entire file without reading it:

dd if=ifile iflag=nocache count=0

It's also more verbose in the description of the nocache flag than the manpage, so maybe worth a read.