1

During weekend my Debian based proxy crashed due to insufficient free space. After reboot it was fine again, so on monday I went hunting for logs and/or explanations for saturday hang but I couldn't find anything.

Grepped any kind of stuff in /var/log/*, checked crontab, mail queue... The only thing I found was monitoring system echos in syslog, with free space getting critical minute after minute (filled up about 80GB in 30mins). No daemons error or so.

I would easily understand what is going on in realtime, but have no idea on how to further debug this kind of problem in the past. Any advice?

This is the first time this has happened in a year or so. Uptime was low, problem did not reoccur in next days.

Thank you

|improve this question
  • This is not a fact but if your server was being used as a proxy server, and somehow, some miscreant found out about it, this person might have tried to brute force his way into your proxy server to use it for his nefarious purposes. Of course every unsuccessful access will get logged and if you are talking about a machine attack, 80 GB filling up with your proxy server's log, in 30 minutes, is not out of the ordinary. – MelBurslan Mar 15 '16 at 16:31
  • @MelBurslan If that was the case, the giant log should still be there (unless the log rotation frequency is absurdly high). The OP makes it sound like the problem mysteriously went away though. – Chris Down Mar 15 '16 at 16:34
  • I have seen proxy servers deleting logs at the startup time for a clean slate. Not condoning the practice but if the server is used mostly by non-hostile people, there really is not much of a need for the logs. And especially if the OP's modus operandi for using a proxy server of his own, was "plausible deniability" :) why not ? – MelBurslan Mar 15 '16 at 16:45
2

You can use atop to debug stuff like this. atop can run in realtime, but more critically for your case, it can show and analyse snapshots from the past captured in logs. It logs a lot of different metrics, so you likely won't be left thinking "damn, I wish I logged that" after the fact. :-)

On Debian, you can install it with apt-get install atop. You can then start and enable it on boot using your init manager. On systemd for example, it would be systemctl enable atop && systemctl start atop. Now atop will start logging -- typically this is to /var/log/atop/<date>.

You can view historic logs by using atop -r <log file>, going forward in time with t, and backwards with T. You can find more commands by pressing the ? key.

You should look for an app writing a lot to the disk. You can see this in the WRDISK column. You can also sort by disk usage by pressing D.

Obviously this can't go back to before atop even started logging, but you can have it running and logging in the background for next time, when you can investigate properly.

|improve this answer
  • Nice shot, I'll give a try! – realpclaudio Mar 15 '16 at 16:25
0

Also add some checks on disk usage, in particular to look for deleted-but-still-open log files, along the lines of lsof $partition_var_lives_on | grep deleted >> somefile. (Also look at how logs are rotated, if it's something custom, there easily could be a bug.)

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.