CPU idle is high, but processes are struggling to keep up as if the cpu is under heavy load

Question

My centos is barely usable. Often times the terminal becomes totally unresponsive for several seconds, but that may just be lag from ssh, which would be strange because my server is connected to a very powerful internet connection. When I ping it I get about 100ms.

Processes are struggling to keep up, such as my vanilla minecraft server, which spits out these messages periodically, even when it is idling and nobody is connected.

[09:52:47 WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 2425ms behind, skipping 48 tick(s)
[09:54:27 WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 2484ms behind, skipping 49 tick(s)
[09:55:24 WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 9341ms behind, skipping 186 tick(s)
[09:55:45 WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 9635ms behind, skipping 192 tick(s)
[09:55:55 WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 3539ms behind, skipping 70 tick(s)
[10:00:47 WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 3239ms behind, skipping 64 tick(s)

When I take a look at the cpu usage via top, I see this:

top - 13:07:09 up 13:07,  2 users,  load average: 0.28, 0.58, 0.66
Tasks:  55 total,   1 running,  54 sleeping,   0 stopped,   0 zombie
Cpu(s): 10.1%us,  0.8%sy,  0.0%ni, 89.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4194304k total,  1246272k used,  2948032k free,        0k buffers
Swap:   131072k total,        0k used,   131072k free,   479528k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3548 Minecraf  20   0 2947m 658m  12m S 31.6 16.1  10:29.84 java
  755 mysql     20   0  563m  21m 6464 S  0.3  0.5   2:37.13 mysqld
 3762 Minecraf  20   0 14900 1240  984 R  0.3  0.0   0:01.71 top
    1 root      20   0 19236 1520 1224 S  0.0  0.0   0:00.15 init
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd/27052
    3 root      20   0     0    0    0 S  0.0  0.0   0:00.00 khelper/27052
    4 root      20   0     0    0    0 S  0.0  0.0   0:00.00 rpciod/27052/0
    5 root      20   0     0    0    0 S  0.0  0.0   0:00.00 rpciod/27052/1
    6 root      20   0     0    0    0 S  0.0  0.0   0:00.00 rpciod/27052/2
    7 root      20   0     0    0    0 S  0.0  0.0   0:00.00 rpciod/27052/3
    8 root      20   0     0    0    0 S  0.0  0.0   0:00.00 rpciod/27052/4
    9 root      20   0     0    0    0 S  0.0  0.0   0:00.00 rpciod/27052/5
   10 root      20   0     0    0    0 S  0.0  0.0   0:00.00 rpciod/27052/6
   11 root      20   0     0    0    0 S  0.0  0.0   0:00.00 rpciod/27052/7
   12 root      20   0     0    0    0 S  0.0  0.0   0:00.00 nfsiod/27052
...

I copied this view of top within the second the server spit out one of the 'can't keep up!' messages.

What is going on? I'm pretty sure the server should have plenty of resources to run just fine, and top tells me that the cpu is mostly idle.

Other info which may be useful is that this is a vps server with 3 cpu cores. I've tried to tell minecraft to use all available cpu cores, but that didn't fix the problem.

Also if this is the wrong place to ask this question, where should I go? Everybody I've asked so far has only told me that I am in the wrong place to be asking this question.

Answer 1

This doesn't seem to be an OS issue so much as a minecraft issue, so I would read through the minecraft forums for answers to this. There are tons of pages dedicated to this issue like this one:

http://minecraftserverhq.com/blog/can%27t-keep-up/

Snippet:

"Other info which may be useful is that this is a vps server with 3 cpu cores. I've tried to tell minecraft to use all available cpu cores, but that didn't fix the problem."

MORE POWER Ok, so be cautions here. This is a bit of a confusing topic. Something you have to realize is that Minecraft Servers can only run on ONE CORE of a processor. Upgrading to a 6-core monster won't do you any good. However, if you get a processor that is faster per core, then you can really up your player capacity without adding any lag. The absolute best processor you can get at the time of this writing is the Intel Core i7-2700K as determined by this website. You'll see there that there are several other processors that performed better, but closer inspection will reveal that they are all 6-cores. That means that even though the 3960X has a passmark of 14,900, your server will only be able to access 2,480 of that power, compared with 2,575 with the 2700K.