Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
In Linux, a finished execution of a command such as cp or dd doesn't mean that the data has been written to the device. One has to, for example, call sync, or invoke the "Safely Remove" or "Eject" function on the drive.
What's the philosophy behind such an approach? Why isn't the data written at once? Is there no danger that the write will fail due to an I/O error?
It simply gives an illusion of speed to programs that don't actually have to wait until a write is complete. Mount your filesystems in sync mode (which gives you your instant writes) and see how slow everything is.
Sometimes files exist only temporarily... a program does some bit of work and deletes the file right after the work is done. If you delayed those writes, you might get away with never having written them in the first place.
Is there no danger that the write will fail due to an IO error?
Oh, absolutely. In such a case, usually the entire filesystem goes into read-only mode, and everything is horrible. But that rarely happens, no point in losing out on the performance advantages in general.
What's the philosophy behind such an approach?
Efficiency (better usage of the disk characteristics) and performance (allows the application to continue immediately after a write).
Why isn't the data written at once?
The main advantage is the OS is free to reorder and merge contiguous write operations to improve their bandwidth usage (less operations and less seeks). Hard disks perform better when a small number of large operations are requested while applications tend to need a large number of small operations instead. Another clear optimization is the OS can also remove all but the last write when the same block is written multiple times in a short period of time, or even remove some writes all together if the affected file has been removed in the meantime.
These asynchronous writes are done after the write system call has returned. This is the second and most user visible advantage. Asynchronous writes speeds up the applications as they are free to continue their work without waiting for the data to actually be on disk. The same kind of buffering/caching is also implemented for read operations where recently or often read blocks are retained in memory instead of being read again from the disk.
Is there no danger that the write will fail due to an IO error?
Not necessarily. That depends on the file system used and the redundancy in place. An I/O error might be harmless if the data can be saved elsewhere. Modern file systems like ZFS do self heal bad disk blocks. Note also that I/O errors do not crash modern OSes. If they happen during data access, they are simply reported to the affected application. If they happen during structural metadata access and put the file system at risk, it might be remounted read-only or made inaccessible.
There is also a slight data loss risk in case of an OS crash, a power outage, or an hardware failure. This is the reason why applications that must be 100% sure the data is on disk (e.g. databases/financial apps) are doing less efficient but more secure synchronous writes. To mitigate the performance impact, many applications still use asynchronous writes but eventually sync them when the user saves explicitly a file (e.g. vim, word processors.)
On the other hand, a very large majority of users and applications do not need nor care the safety that synchronous writes do provide. If there is a crash or power outage, the only risk is often to lose at worst the last 30 seconds of data. Unless there is a financial transaction involved or something similar that would imply a cost much larger than 30 seconds of their time, the huge gain in performance (which is not an illusion but very real) asynchronous writes is allowing largely outperforms the risk.
Finally, synchronous writes are not enough to protect the data written anyway. Should your application really need to be sure their data cannot be lost whatever happens, data replication on multiple disks and on multiple geographical locations need to be put in place to resist disasters like fire, flooding, etc.
Asynchronous, buffered I/O was in use before Linux and even before Unix. Unix had it, and so have all its offshoots.
Here is what Ritchie and Thompson wrote in their CACM paper The UNIX Time-Sharing System:
To the user, both reading and writing of files appear to be synchronous and unbuffered. That is immediately after return from a read call the data are available, and conversely after a write the user’s workspace may be reused. In fact the system maintains a rather complicated buffering mechanism which reduces greatly the number of I/O operations required to access a file.
In your question, you also wrote:
Is there no danger that the write will fail due to an IO error?
Yes, the write can fail and the program might not ever know about it. Although never a good thing, the effects of this can be minimized in cases where an I/O error generates a system panic (on some OS'es this is configurable - instead of panicking, the system can continue to run but the affected filesystem is unmounted or mounted read-only). Users can then be notified that the data on that filesystem is suspect. And a disk drive can be proactively monitored to see whether its grown defect list is rapidly increasing, which is an indication that the drive is failing.
BSD added the fsync system call so a program could be certain that its file data had been completely written to disk before proceeding, and subsequent Unix systems have provided options to do synchronous writes. GNU dd has an option conv=fsync to make sure that all the data has been written out before the command exits. It comes in handy when writing to slow removable flash drives, where buffered data can take several minutes to write out.
Another source of file corruption is a sudden system shutdown, for example from loss of power. Virtually all current systems support a clean/dirty flag in their filesystems. The flag is set to clean when there is no more data to be written out and the filesystem is about to be unmounted, typically during system shutdown or by manually calling umount. Systems will usually run fsck upon reboot if they detect that filesystems were not shut down cleanly.
Many good answers, but let me add one other thing... Remember that Unix is a multi-process and multi-users system, so potentially many users would be trying to do file-operations (esp. writes) at (almost) the same time. With old slow hard-disks - perhaps mounted over the network - this would not only take time (for which the programs would basically lock-up and the users have to wait), but cause lots of moving the read/write-head of the disk back and forth.
So instead, the files waiting to be written were kept in the memory for a while, and sorted after where they should end-up on the disk... and when the buffer was full - or the disk-sync daemon had waited for the required number of seconds (I think it usually was about 30 seconds) - the whole buffer was written out to the disk "in order", with the write-head only having to do one continous sweeping motion, writing the files to the disk as it went... instead of jumping all over the place.
Of cource with today's fast disks - not to mention solid-state devices - the gain is a lot less... espeically on a home linux-system, where there is only one user working at a time, and only with a few programs.
Anyway, the combination of anticipating reads by reading in (to the cache/buffer) more than was asked for - and sorting data waiting to be written, so it could be written in "one motion" - was actually a very good idea at the time, especially on systems with lots of reading and writing by many users.
It is not specific to Linux, and it is called the page cache (which Linux does quite well). See also http://linuxatemyram.com/; so if a file is written, then read again a few seconds later, very often no disk I/O is needed.
The main advantage is that on many systems, there is a lot of RAM, and some of it can be used as a cache by the kernel. So some files operations can take profit of this caching. Also, disk I/O time is a lot slower (typically many thousand times for SDD, and nearly a million times slower for mechanical hard disks) than RAM.
Application code can give hints regarding this caching: see e.g. posix_fadvise(2) & madvise(2)
Spinning platters are slower than RAM. We use caching of reads/writes to 'hide' this fact.
The useful thing about write IO is that it doesn't require disk IO to happen immediately - unlike a read, where you can't return data to the user until the read completes on the disk.
Thus writes operate under a soft time constraint - as long as our sustained throughput does not exceed that of our disk, we can hide a lot of the performance penalties in a write cache.
And we do need to write cache - spinning disks are very slow comparatively. But so to do modern RAID types have a significant penalty to operation.
A RAID 6 for example, in order to complete one write IO must:
Thus each write is actually 6 IO operations - and particularly when you've got slow disks like big SATA drives, this gets extremely expensive.
But there's a nice easy solution - write coalescing. If you can build a 'full stripe' write in a buffer, you don't need to read parity from your disk - you can compute it based on what you have in memory.
It's very desirable to do this, because then you don't have write amplification any more. Indeed, you can end up with a lower write penalty than RAID 1+0.
Consider:
RAID 6, 8+2 - 10 spindles.
8 consecutive data blocks to write - compute parity in cache, and write one block to each disk. 10 writes per 8, means a write penalty of 1.25. 10 disks of RAID 1+0 still has a write penalty of 2 (because you have to write to each submirror). So in this scenario, you can actually make RAID 6 perform better than RAID1+0. In real world usage, you get a bit more of a mixed IO profile though.
So write caching makes a huge difference to perceived performance of RAID sets - you get to write at RAM speed and have a low write penalty - improving your sustained throughput if you do it.
And if you don't, you suffer the achey slow performance of SATA, but multiply it by 6 and add some contention in there. Your 10 way SATA RAID-6 without write caching would be a little faster than a single drive without RAID... but not by very much.
You do take a risk though - as you note - power loss means data loss. You can mitigate this by cache flushing cycles, battery backing your cache, or using SSD or other non-volatile caches.
None of the other answers mentioned delayed allocation. XFS, ext4, BTRFS, and ZFS all use it. XFS has been using it since before ext4 existed, so I'll use it as the example:
XFS doesn't even decide where to put data until writeout. Delayed-allocation gives the allocator much more information to base its decisions on. When a file is first being written, there's no way to know if it will be a 4k file or a 1G-and-still-growing file. If there's 10G of contiguous free space somewhere, putting the 4k file at the beginning of it does no good. Putting the large file at the beginning of a big free space reduces fragmentation.
All the other answers here are at a minimum mostly correct for the normal case, and I would recommend reading any of them before mine, but you mentioned dd and dd has a typical use case that May not involve write caching. Write caching is primarily implemented at the filesystem level. Raw devices do not normally do write caching (multiple device drivers such as raid or lvm are another ball of wax). Since dd is often used with raw block devices it provides the bs and related options to allow large writes for better performance on raw devices. This is not as useful when both endpoints are regular files (although large writes uses fewer system calls in this case). The other common place where this is particularly visible is with the mtools package which is a userspace fat file system implementation. using mtools with a floppy drive always feels incredibly sluggish as the tools are completely synchronous and floppy drives are incredibly slow. Mounting the floppy and using the kernel fat file system is much more responsive except for umount which is synchronous (and very important for it to be that way to prevent data loss, especially for removable devices like floppies). There are only a few other programs which I am aware of regularly being used with raw devices like specially configured databases (which implement their own write caching), tar, and specialty device and filesystem tools like chdsk, mkfs and mt.
O_DIRECT if you want to bypass the cache. dd oflag=direct. IIRC, some unices default to direct I/O on block devices. (And require reading / writing of aligned blocks, which Linux doesn't because it's just writing the pagecache anyway.)
Aug 22, 2015 at 23:36
The philosophy is unsafe-by-default.
There are two reasonable and obvious strategies possible: flush writes to disk immediately or delay writing. UNIX historically chose the latter. So get safety, you need to call fsync afterwards.
However, you can specify safety upfront by mounting a device with option sync, or per-file by opening them with O_SYNC.
Remember that UNIX was designed for computer experts. "Safe by default" was not a consideration. Safety means slower I/O, and those early systems really had slow I/O making the price ratehr high. Unfortunately, neither UNIX nor Linux switched to safe-be-default, even though this is a non-breaking change.
It trades a small amount of reliability for a great increase in throughput.
Suppose, for example, a video compressing program. With delayed write ("write back"):
Versus
The second version appears twice as fast because it can use the CPU and disk at the same time, while the first version is always waiting for one or the other.
Generally you want write-back for streaming operations and bulk file operations, and write-through for databases and database-like applications.
In many applications, storage devices will be intermittently busy reading data. If a system is always able to defer writes until a time when the storage device isn't busy reading data, then from an application's point of view the writes will take zero time to complete. The only situations in which writes would not be instantaneous would be when:
Write buffers fill up to the point that no more deferred-write requests can be accepted until writes actually complete.
It is necessary to shut down or remove the device for which writes are pending.
An application specifically requests confirmation that a write is actually completed.
Indeed, it's only because of the above requirements that writes ever need to actually take place at all. On the other hand, there's generally no reason not to perform any pending writes at times when a device would otherwise be idle, so a lot of systems perform them then.
There is also this:
Write "Hi, Joe Moe"
is faster than:
Write "Hi, "
Write "Joe "
Write "Moe "
And also:
Write "Hi, how are you?"
is faster than:
Write "Hi, what's up?"
Delete that
Write "Howdy, how are you?"
Delete that
Write "Hi, how are you?"
It's better for modifications and aggregation to happen in RAM than on disk. Batching disk writes frees application developers from such concerns.
