0

Every evening, a database gets backed up on a server. This is a good thing.

The strange thing is that I can't find the cron job that does this. I've checked in /etc/cron*, I've checked in /var/spool/cron/. I've checked crontabs for all other users on this system. I can't seem to find this job. So my next theory is that the job is executing remotely, connecting to the server and then backing up the database to a file. No one currently involved seems to know (or remember) anything about this process.

I need to catch this job in the act! I'd like to find out:

Is it really executing remotely? If so, how can I find where the connection is coming in from? I assume it must be from some user that has login credentials, but I'd like to know who.

If it's executing locally, where is the file, and how is it so hard to find it?

How can I find these things? So far, the only evidence I have of this occurring is database dump files created at the exact same time every evening.

6
  • check /etc/cron*. both files and directories Jan 25, 2017 at 17:03
  • @IporSircer: I did, sorry for the typo. Jan 25, 2017 at 17:03
  • Check the cron logfile to see if the job is even being run there. If not, check the db logs to see how it's being run, and the time, and the source of the connection. For example, if your db is mysql, and mysqldump is being used for the backup, maybe the mysqldump client is running from a remote host. Jan 25, 2017 at 17:16
  • If this is a modern system, this also may be done with systemd timers.
    – Chris Down
    Jan 25, 2017 at 18:24
  • 1
    You can also use BCC's opensnoop tool to find out the process which is creating the file: github.com/iovisor/bcc/blob/master/tools/opensnoop.py
    – Chris Down
    Jan 25, 2017 at 18:25

1 Answer 1

1

As a first step I would use inotifywait.

Assuming a Debian system, if not you can port the equivalent commands to your system.

Install inotify-tools:

apt-get install inotify-tools -y

Run inotifywait on / and monitor all accesses overnight. Here's what I typically use:

echo 10000 > /proc/sys/fs/inotify/max_user_watches
inotifywait -mr / -e access -e create -e modify -e delete -e moved_to -e moved_from --format %w:%f:%e:%T --timefmt %F:%T >> /root/system.inotify.log &

Verify the logs the next day to catch what happened.

Even if a remote system copies a script locally, executes it and then deletes it, you will still see it there. Whatever you see there will hint to you at what happens, as it will catch everything at the underlying level.

It's not vulnerable to race conditions or if the script deletes the source file very quickly as it hooks to the Linux inotify interface so no actions will be missed.

Let us know what you find, I'm interested to know.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .