why linux cp command don't consume disk IO?

Question

os: centos7

test file: a.txt 1.2G

monitor command: iostat -xdm 1

The first scene:
cp a.txt b.txt        #b.txt is not exist

The second scene: 
cp a.txt b.txt        #b.txt is exist

Why the first scene don't consume IO, but the second scene consume IO?

Answer 1

It could well be that the data had not been flushed to disk during the first cp operation, but was during the second.

Try setting vm.dirty_background_bytes to something small, like 1048576 (1 MiB) to see if this is the case; run sysctl -w vm.dirty_background_bytes=1048576, and then your first cp scenario should show I/O.

What's going on here?

Except in cases of synchronous and/or direct I/O, writes to disk get buffered in memory until a threshold is hit, at which point they begin to be flushed to disk in the background. This threshold doesn't have an official name, but it's controlled by vm.dirty_background_bytes and vm.dirty_background_ratio, so I'll call it the "Dirty Background Threshold." From the kernel docs:

vm.dirty_background_bytes

Contains the amount of dirty memory at which the background kernel flusher threads will start writeback.

Note: dirty_background_bytes is the counterpart of dirty_background_ratio. Only one of them may be specified at a time. When one sysctl is written it is immediately taken into account to evaluate the dirty memory limits and the other appears as 0 when read.

dirty_background_ratio

Contains, as a percentage of total available memory that contains free pages and reclaimable pages, the number of pages at which the background kernel flusher threads will start writing out dirty data.

The total available memory is not equal to total system memory.

vm.dirty_bytes and vm.dirty_ratio

There's a second threshold, beyond this one. Well, more a limit than a threshold, and it's controlled by vm.dirty_bytes and vm.dirty_ratio. Again, it doesn't have an official name, so we'll call it the "Dirty Limit". Once enough data has been "written", but not committed to the underlying block device, further attempts to write will have to wait for write I/O to complete. (The precise details of what data they'll have to wait on is unclear to me, and may be a function of the I/O scheduler. I don't know.)

Why?

Disks are slow. Spinning rust especially so, so while the R/W head on a disk is moving to satisfy a read request, no write requests can serviced until the read request completes and the write request can be started. (And vice versa)

Efficiency

This is why we buffer write requests in memory and cache data we've read; we move work from the slow disk to faster memory. When we eventually go to commit the data to disk, we've got a good quantity of data to work with, and we can try to write it in a way that minimizes seek time. (If you're using an SSD, replace the concept of disk seek time with reflashing of SSD blocks; reflashing consumes SSD life and is a slow operation, which SSDs attempt--to varying degrees of success--to hide with their own write caching.)

We can tune how much data gets buffered before the kernel attempts to write it to disk using vm.dirty_background_bytes and vm.dirty_background_ratio.

Too much write data buffered!

If the amount of data you're writing is too great for how quickly it's reaching disk, you'll eventually consume all your system memory. First, your read cache will go away, meaning fewer read requests will be serviced from memory and have to be serviced from disk, slowing down your writes even further! If your write pressure still doesn't let up, eventually like memory allocations will have to wait on your write cache getting freed up some, and that'll even more disruptive.

So we have vm.dirty_bytes (and vm.dirty_ratio); it lets us say, "hey, wait up a minute, it's really time we got data to the disk, before this gets any worse."

Still too much data

Putting a hard stop on I/O is very disruptive, though; disk is already slow from the perspective of reading processes, and it can take several seconds to several minutes for that data to flush; consider vm.dirty_bytes's default of 20. If you have a system with 16GiB of RAM and no swap, you might find your I/O blocked while you wait for 3.4GiB of data to get flushed to disk. On a server with 128GiB of RAM? You're going to have services timing out while you wait on 27.5GiB of data!

So it's helpful to keep vm.dirty_bytes (or vm.dirty_ratio, if you prefer) fairly low, so that if you hit this hard threshold, it will only be minimally disruptive to your services.

What are good values?

With these tunables, you're always trading between throughput and latency. Buffer too much, and you'll have great throughput but terrible latency. Buffer too little, and you'll have terrible throughput but great latency.

On workstations and laptops with only single disks, I like to set vm.dirty_background_bytes to around 1MiB, and vm.dirty_bytes to between 8MiB and 16MiB. I very rarely find a throughput benefit beyond 16MiB for single-user systems, but the latency hangups can get pretty bad for any synchronous workloads like web browser data stores.

On anything with a striped parity array, I find some multiple of the array's stripe width to be a good starting value for vm.dirty_background_bytes; it reduces the likelihood of needing to perform a read/update/write sequence while updating parity, improving array throughput.

For vm.dirty_bytes, it depends on how much latency your services can suffer. Myself, I like calculating the theoretical throughput of the block device, use that to calculate how much data it could move in 100ms or so, and setting vm.dirty_bytes accordingly. A 100ms delay is huge, but it's not catastrophic (in my environment.)

All of this depends on your environment, though; these are only a starting point for finding what works well for you.