4

System:

Linux v22017032713145956 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1+deb8u2 (2017-03-07) x86_64 GNU/Linux
This is a virtualized server running on a node with KVM virtualization.

What I have done:

  • I wanted to run a factorio game server. So I downloaded and run it. (This was in March)
  • After a few days the server simply crashed. I have no records of this beside an email asking my support if a kernel message about rcu_sched detected stalls on cpu had something to do with the node the server is running on.
  • Support said I should try setting the I/O scheduler to noop
  • I set the scheduler accordingly (but only temporary by echoing noop to the sys-file)
  • Everything worked fine for about a month
  • I did regular updates from the Debian repos (only jessie and jessie-updates, no backports or something experimental of any kind)
  • I did regular updates from Froxlor and GitLab repositories.
  • Server crashed again about 4 AM on the 29th April for no apparent reason.
  • I rebooted the server from the node's control panel on May 1st.
  • It crashed again on the same day. This time I did not start the factorio server nor did I change the I/O scheduler.

Additional Info

Ping responses

Monitoring reported the server not responding to pings between:

  • 04-29-2017 04:07:30 -> 04-30-2017 09:55:46
  • 05-01-2017 11:08:52 -> 05-01-2017 11:16:54

Kernel log

/var/log/kern.log in these time frames:

Question Time

What is the problem? I don't remember installing anything.
How can I debug the rcu_sched detected stalls message?

Update from May 7th

I just got a text from my buddy, that the server is behaving oddly. So I checked the logs and again there are stalls. I uploaded the latest log.

Update from May 8th

I just ran memtest86+ and found nothing. But I checked the CPU graph of the last 31 days and found something interesting: CPU Load When the server first became unresponsive to pings, the CPU load on core 2 went haywire, while all other cores were idle. The spike on CPU0 was memtest.

Update from June 7th

Uptime reports:
10:05:05 up 27 days, 20:50, 1 user, load average: 0.23, 0.25, 0.18
But I shut down GitLab. Has anyone experience with GitLab causing troubles on Debian?

  • As far as I know NMI are triggered from either a hardware problem (in which case there's not much you can do but change it) or a watchdog timeout. See this article if it helps. – user86969 May 3 '17 at 9:54
  • 1
    Can hardware errors even exist on a VM? And what do I do if it is the watchdog thing? – Martin B. May 3 '17 at 9:58
  • A memory error could well affect a virtual machine that would venture in that particular faulty space. The fact that you get random crashes doesn't exclude that. But I'm no expert. Running a virtual machine doesn't rule out manifestations of hardware failures, just be aware of how you can spot them. Again, this is just a hint. I also recall there have been bugs in the 3.x kernel series which would affect disk performance and other. Don't take my word for it however. Have you tried running your VM on a different hardware and/or hypervisor? – user86969 May 3 '17 at 10:11
  • I see. I have this server rented from a hosting provider, so my options are limited concerning the memory location. And I recently upgraded my package. On my old package I did not have this problem. I'll consult support on this matter. I have the 3.16 kernel a while now, my old package was running Debian 8 as well. Maybe it is a hardware fault. – Martin B. May 3 '17 at 10:15
  • 1
    Indeed your logs are before that. It makes sense. – Rui F Ribeiro May 3 '17 at 12:06
3

As seen in your logs, I would suppose your problems were probably due to having VirtualBox Guest additions installed in a KVM VM machine, and having some sort of a conflict.

Somewhat the vboxdrv kernel module appeared to be uninstalled and superseded by the kvm/virtio drivers in the old package, I think, and somewhat that not appeared to be happening in the new one for some reason.

As you said, after the logs you are giving us, you uninstalled Virtual Box components.

IMO, you took the right action. Now give it a few days to see whether this happens again.

  • Something is happening again right now. I uploaded the new logs and edited my question. – Martin B. May 7 '17 at 16:54
  • @MartinB. There must be something else. Why the tainted there? I noticed it last time. What their support says? – Rui F Ribeiro May 7 '17 at 17:46
  • cat /proc/sys/kernel/tainted returns 0, that probably was virtualbox. Support says "Try rescue mode, see if the problem persists". But I can't get my mail server working in rescue mode so I'm not really happy with this kind of troubleshooting. They also say that there seems to be nothing wrong with the host system. – Martin B. May 8 '17 at 11:27
0

From the logs, there were some NMI,Refer: https://en.wikipedia.org/wiki/Non-maskable_interrupt

suggest you to check your hardware as well.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.