Linux v22017032713145956 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1+deb8u2 (2017-03-07) x86_64 GNU/Linux
This is a virtualized server running on a node with KVM virtualization.
What I have done:
- I wanted to run a factorio game server. So I downloaded and run it. (This was in March)
- After a few days the server simply crashed. I have no records of this beside an email asking my support if a kernel message about
rcu_sched detected stalls on cpuhad something to do with the node the server is running on.
- Support said I should try setting the I/O scheduler to
- I set the scheduler accordingly (but only temporary by echoing noop to the sys-file)
- Everything worked fine for about a month
- I did regular updates from the Debian repos (only jessie and jessie-updates, no backports or something experimental of any kind)
- I did regular updates from Froxlor and GitLab repositories.
- Server crashed again about 4 AM on the 29th April for no apparent reason.
- I rebooted the server from the node's control panel on May 1st.
- It crashed again on the same day. This time I did not start the factorio server nor did I change the I/O scheduler.
Monitoring reported the server not responding to pings between:
- 04-29-2017 04:07:30 -> 04-30-2017 09:55:46
- 05-01-2017 11:08:52 -> 05-01-2017 11:16:54
/var/log/kern.log in these time frames:
What is the problem? I don't remember installing anything.
How can I debug the
rcu_sched detected stalls message?
Update from May 7th
I just got a text from my buddy, that the server is behaving oddly. So I checked the logs and again there are stalls. I uploaded the latest log.
Update from May 8th
I just ran memtest86+ and found nothing. But I checked the CPU graph of the last 31 days and found something interesting: When the server first became unresponsive to pings, the CPU load on core 2 went haywire, while all other cores were idle. The spike on CPU0 was memtest.
Update from June 7th
10:05:05 up 27 days, 20:50, 1 user, load average: 0.23, 0.25, 0.18
But I shut down GitLab. Has anyone experience with GitLab causing troubles on Debian?