What's the best strategy to catch mystery process?

Question

I have running processes on my server that get killed every night at midnight. It's at work, I'm not around when it happens and I don't have remote access.

The kill occurs very predicably at 23:59 every night. I know this because when I arrive the next day:

• Processes are up until 23:59
• Logs of the process show last modified time of 23:59 (and new dated log is started right after).

Since the killing occurs at the same hour, I strongly suspected a batch job. I went through the crontabs of all our machines and couldn't find anything. Clearly I'm missing something.

I am thinking of laying out a surveillance script that would report the output of ps intermittently, would be launched with at a few minutes before and would loop for a little while. This idea seems weak and highly error prone, so I'm wondering if anyone has a better idea.

More details:

• The universe is a very large and very old legacy system; no one in my team seems aware of such a process (if anyone did, she'd be in our team) although the larger organization consists of thousands of employees, a lot of them would theoretically have access to this (I don't see why they would). In other words, security isn't very tight.

• Environment consists of multiple machines running Solaris 10.

• It's not a production environment, so timeout or down time isn't critical.

• I'm not excluding the possiblity that the killing might not be due to a batch job, although unlikely because of how accurate the timing is.

• Clearly, there are defficiencies in our bookkeeping, so anything imagineable is possible.

My question is what's the best strategy to adopt? It falls under the greater umbrella of "the joy of working on legacy systems". I'm starting to work on my script that I'll post here shortly for feedback. In the meanwhile if anyone has a better idea, please say so.

Answer 1

It is common to rotate logs periodically, rotating them at midnight is common. Many applications will do this automatically.

For those that don't there are tools like logrotate that will do the rotation. Many programs are configured to reopen their logs when sent a HUP signal, and this is one of the techniques used by logrotate.

Things to check:

• Do all the PIDs change. If not, then the programs may be rotating their own log, or responding appropriately to having their logs rotated.
• For programs which change PIDs, were they restarted at midnight? If not check their parent to see what it does.
• Check the crontab for root to see what processes run at the end of the day.
• Check the crontab for the process userid to see what processes run at the end of the day.
• Check to see if the log files are being written directly, or are being written by a log-writer which rotates the logs.