This is a very generic question, but it has been coming up more and more often for me. I would like to try and determine an appropriate course of action for this generic case, as I think it could benefit me and other sysadmins in the future. The commonality in all cases though is java. The reason I am asking this question here rather than stack overflow is that this is a sysadmin question. I think it is very appropriate here.

A user comes up to me and says "My application is very slow. The server you gave me has poor performance." This server could be bare metal, OpenVZ, or KVM. In all cases, OS is CentOS 6. When the user tells me this, they are referring to page load time (JIRA) or benchmark numbers. (Selenium or proprietary app)

Now, I have a sneaking suspicion that the server itself is fine. I ssh onto the server, and notice that the load is less than 1. Memory usage is fine as well, ranging anywhere from 2-20 gigs of ram free. Finally, I check iowait with vmstat, and notice that it is 0. cpu idle is 99.

Looking at Java with top, I can see that CPU percentage is around 200% or so for that process. This is on systems with anywhere from 4 to 24 cores.

Of course, openvz does processor allocation different from KVM or bare metal.) In the cases that this occurs on an openvz orkvm guest, I hop onto the host as well, and see similar things. cpu idle is 99%. iowait is 0. memory is sometimes hundreds of gigs free.

I have looked at these java processes and threads with strace, and strace -f. There are tons of futex, clock, and epoll. If I reverse grep those, I'm left with mostly ioctl, stat, sendto.

Applications where this has happened: Selenium, JIRA, and misc proprietary java applications. Always java though.

I have considered tuning the niceness of the java processes, but AFAIK niceness will only affect how processes are scheduled. If the load is low, it won't help to speed things up.

Sometimes the application uses mysql, but looking at mysql in top shows almost no cpu usage there. I'm guessing that if that was the bottleneck, we would see higher cpu usage.

Bottom line: How do I prove to the user that the server works just fine, but their application sucks? Theoretically, the application could be slow because it encountered a while loop that will never exit. Which leads me to believe that this task is impossible, analogous to solving the halting problem. o_O But perhaps I can tune java to behave a bit better, or perhaps there is a way to profile the running application, point to a chunk of code and say "There's your problem".



Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. --Arthur Conan Doyle

First you may want to check out this talk, which is a discussion about the sources of performance problems and how to diagnose them. It compares Linux to SmartOS, which is kind of separate from the point of your post, but it also helps elucidate where performance problems come from.

Usually when you debug performance problems with software, you should try and replicate the exact same software configuration on a separate physical machine, with the same version of Java, the same kernel, the same userland, etc. If the performance problem is non-existent, then you can start looking at physical hardware, hypervisor/container layer, or installation problems on your box.

If the user has never run this software in this particular OS/version configuration before, then they can't immediately claim that it's your server. I would rather put the burden of proof on them to demonstrate that, running CentOS 6 (with the same minor version and the same hypervisor and the same host OS, if it's not running on bare metal), they are able to get better performance.

If the better performance is then due to them having much superior system specs (e.g. they ran it on a 1 GB KVM on a low-end processor on your infrastructure, but on their own 24-core Ivy Bridge E5 system with 8 PCIe SSDs in hardware RAID0, it's 1000% faster), then you can start to talk about physical hardware. If the physical hardware is the same or very very similar, you can start talking about OS/hypervisor configuration.

You may also consider the possibility that, based on the performance of the application and the number of users and the data workload, the hardware the user has been allocated is simply insufficient for the task. If this is a very important customer and keeping them happy is critical to your business, you could try backing off some of the restrictions you might be imposing on their virtual container or VM instance; for example, temporarily give them more RAM or CPU time, network bandwidth, etc. and see if that helps. If it does, you may have to tell the user that they will need to pay for a bigger VM to get the performance they want on that workload.

Escaping The Burn (when it's Not Your Problem)

If for some reason the user flat-out refuses to try the same software configuration on separate hardware, you will have to resort to either helping them troubleshoot their performance problem in their application (which can be difficult to impossible on Linux, as Brendan Gregg describes in his talk; sometimes you may just have to "wear" that performance cost), or simply insist that the user is incorrect. This is an unfortunate side effect of supporting developers on your systems.

At this point you may need to ask yourself what your business relationship is with your users (if any). If you are providing unmanaged hosting in the private sector, you may have to insist that you are not responsible for ensuring the performance of the user's code.

If you are providing managed hosting, or have otherwise agreed to help the software folks with their woes regardless of whether the problem is their fault, you may have to roll up your sleeves, get your hands dirty, and start running some more sophisticated performance profiling tools, to see if you can spot an actual problem. Is it just a really chatty protocol blocking on a network socket? Is it memory I/O-bound? Or maybe the BIOS doesn't have VT-x enabled? There are way too many possibilities to even list.

To sum up this section: if you can find some business or political reason to excuse yourself from being responsible for resolving this problem, it may be the easiest and most efficient way out. Unfortunately, this also means the user may never fix their issue, and could resort to looking at other business partners as an alternative.

Fanning the Flames (when it's Your Problem)

Since the common theme "appears" to be Java, you could start by at least determining whether the Java process itself is running some code that's extremely slow. For this, ideally, the developer/user/customer will be able to provide you the source code of their Java program (and all dependent libraries).

Try running the program under a good profiler. There are free and non-free profilers; an example of a free one is the one that comes with NetBeans. Getting the profiler to successfully launch and instrument your application can be a challenge, depending on exactly what that application is, but the results are almost always telling in some way.

You can of course resort to using htop and/or iotop to detect whether the Java VM (or the associated RDBMS) is doing an inordinate amount of CPU, memory I/O, or disk I/O, beyond the amount that you'd consider "reasonable" (given the hardware and the workload). These are extremely broad-stroke tools, and don't always give you the full picture, because sometimes performance problems are about what the program isn't doing while it's waiting around for something else, which can be unrelated to a resource bottleneck. This is about as generic a troubleshooting step as attempting to start your car to determine whether the engine is running correctly or not. If it doesn't start, well, what do you do? That's why a proper Java profiler can come in handy if you really need to deep-dive.

Lastly, I'll just echo what Brendan Gregg said in his talk, and mention DTrace. The full functionality of DTrace has not yet been replicated on Linux, neither by the dtrace-on-linux reimplementation, nor any of the competitors, like SystemTap. That said, you can attempt to use one of these tools, and see if it helps. Some insight can be better than none.

You will, of course, need specific expertise with using these very "swiss-army-knife"-ish type tools. You will gain most of that expertise through a lot of experience (read: trial and error, and googling). It's not a matter of just firing up the tool and letting it point you to the problem; you'll need to noodle out the problem based on the available data. It's not easy. Performance rarely is.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.