find out the culprit process

Question

This is rather a basic linux administration question. We have a CentOS linux machine running a production application. There are 10 application specific processes running on that machine. Once in every 3/4 days, the linux machine freezes and the only way to get it back is to hard reboot it from Amazon AWS console.

We have Amazon Cloudwatch enabled which captures the CPU usage every 5 minutes. We see that the CPU reaches 100% (8 cores) within in 10-15 seconds just before it freezes. And unfortunately we could not figure out anything from the process log files.

How do we really pin point which process out of those 10 processes is causing the linux server to freeze?

Are there any simple CPU/memory monitoring applications that can record the top cpu/memory hoggers to disk, say every 2 seconds? Appreciate any other ideas to figure out the culprit process.

Answer 1

You could simply run top in batch mode and save the output to a file:

$ top -b -d 2 > /your/log/file &

-d 2 is the sampling period. Be warned that this will generate quite a bit of data.

You might want to use the -u option to only list processes for a given user, or even the -p option to explicitly list your application's processes.

Answer 2

This link contains several (20) tools for monitoring processes. top of course is the most popular, but other tools such as iotop are useful too. iotop will show which processes are using the most file system IO.

http://www.tecmint.com/command-line-tools-to-monitor-linux-performance/