0

We run a cluster of Java Spring server apps on AWS EC2 instances running Centos 7. We have health monitors on them, and occasionally an alarm will go off and we'll find that the Java process has quietly just disappeared. We can find nothing in any of the logs...either our own, or system logs. We have an outer "catch Throwable" around our own code that logs what it catches, but we run Tomcat, which has may of its own threads. We've added extra logging to try to capture the moment when it disappears, but so far, that has yielded no information.

I've looked over this question: How to find out why a Java process died without a trace in Linux. I see nothing helpful there.

We currently can't involve the launcher of these processes in a solution. It's a long story. Trust me that we've tried to go down that road.

Any suggestions? I'm wondering if maybe I should wrap the Java process in an outer parent process that carefully monitors and logs all signals from the Java child process. I'm wondering if there's such an off-the-shelf solution that I haven't found yet. Any ideas would be greatly appreciated.

1 Answer 1

0

I've used monit to monitor a service. It basically runs as its own service on the system, and has the ability to watch for the existence of a certain named process. You can have it restart the service or just generate an alert when it is gone. It has logic to detect failure to restart (and number of tries) as well. You can also have it do stuff like open an http connection to check if the service is up, as well.

I believe that monit is in EPEL7 for CentOS7.

2
  • Wow. Fabulous! I'll check this out right away.
    – Rapt0r
    Commented Aug 23, 2021 at 18:40
  • I checked this out. Unfortunately, I don't think this will give us anything more than our current monitoring. We know when the process dies due to monitoring, and we don't care about restarting it automatically. What we need is more information as to WHY it died. Monit seems great for what it does, but I don't see that it can know anything more than the operator can about why the process disappeared.
    – Rapt0r
    Commented Aug 23, 2021 at 19:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .