Ok, you ask for experiences, this makes the question a little subjective and argumentative, but passable.
Linus said that referring to the uses that people usually attribute to O_DIRECT, and for those uses, IMO Linus is mostly correct. Even if you do direct I/O, you cannot transfer data to/from devices directly to your program statements, you need a buffer that is filled (by the program or the device) and transferred through a system call to the other end. Also, to make it efficient, you will not want to reread something you just already read, in case you need it again. So you need some sort of cache... and it is exactly that that the kernel provides without O_DIRECT, a page cache! Why not use that? It also comes with benefits if more processes want to access the same file concurrently, it would be a disaster with O_DIRECT.
Having said that, O_DIRECT has its uses: If for some reason you need to get data directly from the block device. It has nothing to do with performance.
People using O_DIRECT for performance usually come from systems with bad page cache algorithms, or without POSIX advice mechanisms, or even people mindlessly repeating what other people have said. To avoid these problems, O_DIRECT was a solution. Linux, OTOH, has the philosophy that you should fix the real underlying problem, and the underlying problem was OSs that did a bad job with page caching.
I used O_DIRECT for a simple implementation of cat to find a memory error in my machine. This is one valid use for O_DIRECT. That had nothing to do with performance.