1

I have a Linode running Ubuntu 14.04, with a stack managed by ServerPilot.io.

Only real modification is that I'm running HHVM in a limited fashion.

Tonight we had an odd occurrence. I received notification the websites were down. Upon investigation, I couldn't connect to the server at all over SSH and pings were not returned.

I logged in via the console Linode supplies and everything looked normal. All expected services were up, however I couldn't connect to any outside host via ping, telnet, etc.

Basically, it seems networking was shutdown.

It's been quite a few years since I've done *nix system administration, so after hunting around in all of the logs I could find, I gave up and rebooted the server and all is well again.

I'm curious if someone can point me to the logs I should be looking at next time this happens so I can determine why it happened, fix it, and prevent it from happening again

P.S. The only other thing of note was that the CPU and network traffic spiked considerably before the outage. Graphs are here: http://cl.ly/image/2L0n1Y3A2v3h

Improve this question
1
  • I'm also seeing a problem on Linode that looks exactly like yours (Although on Debian 6). The server looses all network conectivity, but logging in via shows everything up and ok (but with no connectivity). This has happened multiple times to us on multiple machines. Did you get to the bottom of this?
    – UpTheCreek
    Commented Dec 10, 2014 at 12:00

1 Answer 1

Reset to default
1

In your case I would have tried to get the network backup by issuing:

ifdown eth0
ifup eth0

and only reboot after that would not work.

As I cannot see if the network traffic was going out or in, nor that you indicate what is actually running on the servers, it is difficult to say what happened.

Can you see from the logs if some expensive (CPU wise) http requests were being done?

It could be that the system was put "off the hook" by your provider because of the excessive amount of traffic, and that the network did not try to reconnect after its internet connection was re-established.

There is of course a chance that the system is compromised. I hope you have kept it up-to-date with security patches.

Improve this answer
3
  • Yeah, these are all good points. The server is only a few days old and ServerPilot automagically handles security patches. I will check with Linode to see if there is any chance that they disconnected it. Just never seen this type of thing happen before.
    – clifgriffin
    Commented Nov 7, 2014 at 14:08
  • Had Linode look into this and they have seen this with the latest Ubuntu kernel. Suggested I switch to the previous kernel, so that may be the answer to the mystery :)
    – clifgriffin
    Commented Nov 7, 2014 at 15:41
  • 1
    And if it happens again in spite of that you can still try the ifdown/ifup.
    – Anthon
    Commented Nov 7, 2014 at 16:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .