3

Are there any guarantees as to when and in what order dirty pages are written out to disk? In the specific case of log structured files (files that are only ever appended to), is it possible for dirty pages to accumulate and be written to disk out of order?

For example, imagine a log file with the following dirty pages that have accumulated...

1: file_offset=16kb page_len=4kb <-- dirty
2: file_offset=20kb page_len=4kb <-- dirty
3: file_offset=24kb page_len=4kb <-- dirty

Will Linux write out the above dirty pages in order or is there a chance they they'll get written out of order? If out of order writing is a possibility, how do databases and applications that generate log files handle this? For example, if page 3 gets written out first and the OS crashes before it can write out 1 and 2, there'll be a 8kb filehole.

  • how do databases and applications that generate log files handle this? In general, they don't. Log files like this are merely records of what happened - they're informational, not critical. If you need a guarantee like this, you're using the wrong tool for the job. File systems are data stores, they aren't messaging mediums. – Andrew Henle Nov 6 '18 at 12:34
  • That doesn't sound even remotely correct, nor does it answer the question. My understanding is that write-ahead logs are used for database recover. Definitely critical. Definitely not "purely informational". Nothing to do with it being a "messaging medium". – offbynull Nov 6 '18 at 13:56
  • that write-ahead logs are used for database recover Proper databases bypass the page cache and directly manipulate data, explicitly making sure it's safely on disk. The lack of specifics in your question and your "out of order" statement implies sequentially-written text log files - which a lot of people do try to misuse as a communications channel. Sometimes it even works, if "works most of the time, kinda/maybe/eventually" is an acceptable definition of "works" for you. – Andrew Henle Nov 6 '18 at 14:18
  • Once again, what you're saying doesn't sound correct. The information I'm seeing online says that different databases have different syncing strategies. For example, Cassandra defaults to fsync its WAL every 10 seconds. Postgres and SQLite also fsync. LevelDB leaves it up to Linux. But, the question is about how Linux flushes out pages for append-only scenarios. Forget about the usecases. – offbynull Nov 6 '18 at 15:22
  • And just what does fsync() do? Forces data to storage, taking the page cache out of the equation. – Andrew Henle Nov 6 '18 at 15:43
1

Will Linux write out the above dirty pages in order or is there a chance they they'll get written out of order?

I'm not sure.

If out of order writing is a possibility, how do databases and applications that generate log files handle this?

https://www.sqlite.org/atomiccommit.html#_dealing_with_garbage_written_into_journal_files

SQLite assumes that the underlying filesystem can reorder write requests and that the page count can be burned into oxide first even though its write request occurred last. So as a second line of defense, SQLite also uses a 32-bit checksum on every page of data in the rollback journal. This checksum is evaluated for each page during rollback while rolling back a journal as described in section 4.4. If an incorrect checksum is seen, the rollback is abandoned. Note that the checksum does not guarantee that the page data is correct since there is a small but finite probability that the checksum might be right even if the data is corrupt. But the checksum does at least make such an error unlikely.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.