0

Continuing from Server crashing for not so obvious reasons.

Short overview of what happened

My server (running Debian) has crashed very often.
Everything would work fine until noticable lag sets in. There are oops messages in the dmesg logs. They do not appear to be fatal but sometimes the server simply becomes unresponsive. The management interface from the node the server runs on (it's virtualized) says that during this time [when it's unresponsive] the CPU load goes haywire.

So...

I have talked with friend of mine (who maintains multiple servers) and he said that he is experiencing this problem as well. But only on the server running GitLab CE.
I have GitLab CE (installed from deb https://packages.gitlab.com/gitlab/gitlab-ce/debian/ jessie main). So I tried stopping the GitLab services.

Since I stopped all GitLab processes the problem is gone. Everything running perfectly fine.

What do I do wrong? Are there some configuration traps that can crash my server without leaving a trace of information? The rcu_shed stall messages still appear but the server is not crashing anymore.

  • 1
    Probably I/O related? When deploying? Have you done more specific tests? – Rui F Ribeiro Jun 10 '17 at 20:49
  • No specific tests. I don't know which... Once I changed the I/O scheduler to noop but only temporary and it crashed either way. – Martin B. Jun 10 '17 at 21:03
  • With a recent kernel in vmware, I think the default is noop nowadays. – Rui F Ribeiro Jun 10 '17 at 21:05
  • With Debian 8 cfg is the default. At least on my maching. Do you think it is a problem with the scheduler? What can I try to test this? – Martin B. Jun 10 '17 at 21:11
  • The scheduler wont be my first suspicion. I would monitor I/O vs CPU usage over time. – Rui F Ribeiro Jun 10 '17 at 21:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.