Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

We discovered an I/O related performance problem when using the default SLES11 SP2 kernel. Our same application on the same hardware had no issue with (the admittedly ancient SLES9 SP3).

We had a feeling it has something to do with the kernel and have done the following experiment so far:

  1. Downloaded and unpacked the kernel.org 3.0.13 source tree.
  2. Installed Novell's kernel-source-3.0.13-0.27.1 package. This is the 3.0.13 source with all of Novell's patches/tweaks already applied.
  3. In the 3.0.13 tree did a make x86_64_defconfig to generate a default config file, then used make menuconfig to turn on the drive controller and network card drivers our hardware needed and built the kernel. This kernel had no performance problem.
  4. In the 3.0.13-0.27.1 tree we got the config file that Novell ships with their compiled kernel, built with it, and had the same performance problem we had with the Novell-compiled kernel.
  5. In the 3.0.13-0.27.1 tree, took the config file used in (3), built with it (accepting the additional config defaults that Novell's version of the configurator wanted to add). There was no performance problem.
  6. In the 3.0.13 tree, took the config file used in (4), built with it (losing the Novell-only config options in the file). There was no performance problem.
  7. In the 3.0.13-0.27.1 tree, took the config file used in (4), and made the device drivers (but nothing else) match the config file in (3) (i.e. turned off lots and lots of device drivers that weren't being used anyways). There was no performance problem.

So the kernel.org 3.0.13 tree had no performance problem either way. Novell's 3.0.13-0.27.1 tree only had the problem when we built with Novell's config file or a file that was identical to Novell's but with most of the device drivers de-configured.

Given that, it would seem to be that there's either some Novell-only config option (and implicitly the Novell code it activates) causing the problem, or some bad interaction between a Novell patch and some standard option (given that the as-close-as-possible-to-Novell's-config did not have the problem when built against the virgin kernel.org source).

While we continue to investigate, since I am no kernel hacker (I know how to build the kernel but that's about it) I was wondering what "families" of config options would be the best ones to focus attention on. The idea would be to change some options from the "Novell" setting to the "kernel.org" setting and see if the problem goes away, then set them back and try another set, and so on.

But I'd like to narrow that down -- hence the question about which config options would be good bets to play with first.

share|improve this question
3  
Run diff on the generated .config files to see what the differences are. –  Michael Hampton Nov 13 '12 at 1:50
    
SuSE is not part of Novell any longer. Apart from that - what IO (network, disk, ram, ...) are you talking about and how did you measure that problem? –  Nils Nov 13 '12 at 21:48
add comment

1 Answer

We solved the problem! Doing a quasi binary search on the config options differences between "good" and "bad" kernels, we found out that the problem is due to SLES11 turning on ext3 write barriers by default. A kernel built with write barriers off by default did not have the problem. And when we used a kernel with the problem but put barrier=0 in the mount options in /etc/fstab, the problem went away.

share|improve this answer
    
Just make sure you know what that change means. I seem to remember some scary silent data corruption issues around that timeframe. No newer kernels available either? –  vonbrand Jan 24 '13 at 17:58
    
barriers off means that fsync (and others) are satisfied by flushing buffer cache to the volatile on-disk cache. Losing power means that you could lose data in that disk cache that the client process was told had been flushed to stable storage. –  Tim B Feb 14 '13 at 21:08
    
We understand the risks in having them off. However, for our use case the performance with them on is unacceptable. –  QuantumMechanic Feb 16 '13 at 1:12
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.