Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

If this question is too programmer oriented, let me know. I wonder if there are people familiar with the O_DIRECT flag for the open() system call on Linux 2.6? Linus disparages its use, however high performance file writing seems to indicate its use. I would like to know of any real world experience and recommendations.

More info: The application that I am using does maintain its own cache, and in so doing attains an average of 5x or more speed up by doing so. When writing to file, the contents of the cache must be written out to the filesystem cache, which seems redundant and a performance concern.

share|improve this question

4 Answers 4

Ok, you ask for experiences, this makes the question a little subjective and argumentative, but passable.

Linus said that referring to the uses that people usually attribute to O_DIRECT, and for those uses, IMO Linus is mostly correct. Even if you do direct I/O, you cannot transfer data to/from devices directly to your program statements, you need a buffer that is filled (by the program or the device) and transferred through a system call to the other end. Also, to make it efficient, you will not want to reread something you just already read, in case you need it again. So you need some sort of cache... and it is exactly that that the kernel provides without O_DIRECT, a page cache! Why not use that? It also comes with benefits if more processes want to access the same file concurrently, it would be a disaster with O_DIRECT.

Having said that, O_DIRECT has its uses: If for some reason you need to get data directly from the block device. It has nothing to do with performance.

People using O_DIRECT for performance usually come from systems with bad page cache algorithms, or without POSIX advice mechanisms, or even people mindlessly repeating what other people have said. To avoid these problems, O_DIRECT was a solution. Linux, OTOH, has the philosophy that you should fix the real underlying problem, and the underlying problem was OSs that did a bad job with page caching.

I used O_DIRECT for a simple implementation of cat to find a memory error in my machine. This is one valid use for O_DIRECT. There is nothing to do with performance.

share|improve this answer
    
Thanks for the info, it is appreciated. I have updated my question with the specific conditions of the app that prompted this question. If you have more details on the POSIX advice mechanisms for writing files, that would be appreciated, too. –  casualunixer Jan 28 '11 at 5:42
    
o_direct might also be useful in a system where the developer wants to provide a caching mechanism at the application layer (think databases). –  Jmoney38 Feb 3 at 17:14

Note that using O_DIRECT is liable to fail in newer kernels with newer file systems. See this bug report for example. So not only is the use often dubious, it will likely not work at all in the coming generation of Linux distributions. So I would not bet the performance of my code on it, even if you happen to be able to prove that it might have a benefit.

share|improve this answer
    
The bug report actually discusses the use of filesystems with the journal=data option on. This option is directly opposite in effect to the O_DIRECT flag. Most ext3 and ext4 filesystems do not have this flag set and if they do, turning it off will permit opening the file with O_DIRECT. –  casualunixer Jan 29 '11 at 17:57

Relating to what @Juliano has already said.

Checkout posix_fadvise if the real problem is misbehaviour of underlying filesystem's cache algorithm, you can try give it advice, how are you going to use filesystem. For nicely implemented fs, it should give performance boost. (Here is link to another topic touching similar considerations http://stackoverflow.com/a/3755818/544721 )

share|improve this answer

Actually, O_DIRECT is needed to avoid either of

  • cache pollution — sometimes you know that there's no sense in overhead with caching, for e. g. when dealing with really large files, say 64 GiB when there's only 2 GiB of RAM. Torrent file of 32 GiB which a user decided to verify doesn't seem to be a good candidate for cache. It's just extra activity with its own overhead. And it can cause some really useful data to be pruned from cache.
  • double caching — for e. g. some RDBMSes (MySQL to mention) allows for defining its own cache. They supposedly know better how to cache and what, then kernel's Virtual Memory which knows not a thing about SQL planning and so on

— which is no good, as it seen. And O_DIRECT doesn't mean to be faster, often it is not.

share|improve this answer
2  
posix_fadvise can take care of the cache pollution problem. –  psusi Jun 14 '12 at 14:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.