I have running processes on my server that get killed every night at midnight. It's at work, I'm not around when it happens and I don't have remote access.
The kill occurs very predicably at 23:59 every night. I know this because when I arrive the next day:
- Processes are up until 23:59
- Logs of the process show last modified time of 23:59 (and new dated log is started right after).
Since the killing occurs at the same hour, I strongly suspected a batch job. I went through the
crontabs of all our machines and couldn't find anything. Clearly I'm missing something.
I am thinking of laying out a surveillance script that would report the output of
ps intermittently, would be launched with
at a few minutes before and would loop for a little while. This idea seems weak and highly error prone, so I'm wondering if anyone has a better idea.
The universe is a very large and very old legacy system; no one in my team seems aware of such a process (if anyone did, she'd be in our team) although the larger organization consists of thousands of employees, a lot of them would theoretically have access to this (I don't see why they would). In other words, security isn't very tight.
Environment consists of multiple machines running Solaris 10.
It's not a production environment, so timeout or down time isn't critical.
I'm not excluding the possiblity that the killing might not be due to a batch job, although unlikely because of how accurate the timing is.
Clearly, there are defficiencies in our bookkeeping, so anything imagineable is possible.
My question is what's the best strategy to adopt? It falls under the greater umbrella of "the joy of working on legacy systems". I'm starting to work on my script that I'll post here shortly for feedback. In the meanwhile if anyone has a better idea, please say so.