How to Find a Fugitive Crontab

Question

A few years ago I setup a cron job to automatically ping a URL every minute as part of a monitoring system (that's oversimplification, but it'll do for this question). Because I'm a horrible person, I didn't document this anywhere.

Today, years later, I started having trouble with the application on the other end of the URL that's being pinged. I fixed it, but then realized, I have no idea where this cron job is coming from.

Is there a way to quickly search through or cat out all the crontabs on a particular system? I have root access so permissions aren't a problem. I've only ever been a user of cron, I've never looked too deeply at its implementation, but my *nix instincts say there has to be a group of text files somewhere that hold all the crontabs. I just don't know where they'd be, and if I dug into it I'd be afraid of finding some, but not all of them, or missing some weird nuance of the system

Also, I realize with root access I could

1. Get a list of all the users in the system
2. su as a user
3. crontab -l
4. Repeat with all the users

but I'm looking for something a little less manual (and looking to learn something about cron's implementation)

Answer 1

There are only a few places crontabs can hide:

• /etc/crontab
• /etc/cron.d/*
• /etc/crond.{hourly,daily,weekly,monthly}/*
  these are called from /etc/crontab, so maybe an asterisk on this
• /var/spool/cron/* (sometimes /var/spool/cron/crontabs/*)

Be sure to check at as well, which keeps its jobs in /var/spool/at/ or /var/spool/cron/at*/

Also, instead of

su <user>
crontab -l

Just do this:

crontab -lu <user>

Answer 2

Crontabs live in /etc/crontab and (with many implementations) its components in /etc/cron.*/* (all edited by root), and in /var/spool/cron/* (users' crontabs).

If you have trouble locating the offending job, another approach is to investigate one while it's going on. For example, you can add a firewall rule to log the user id of the process opening connections to example.com on port 80:

iptables -A OUTPUT -p tcp --syn -d example.com --dport 80 -j LOG --log-prefix "[->example.com] " --log-uid

If the job is using an application such as ping or curl, shadow the usual binary by a wrapper that logs information about what is using it, with a script like this one in /usr/local/bin:

#!/bin/sh
{
  date
  echo "$0" "$@"
  ps -p $PPID
  echo
} >>"/tmp/$(basename "$0").log"
exec "/usr/bin/$(basename "$0")" "$@"

Answer 3

The quick and dirty way:

grep -r ping /var/spool/cron/crontabs

The exact location where crontabs are stored may vary from system to system, but it's generally in /var/spool and has crontab somewhere in the name.

Also note that many systems have system crontabs (like in /etc/crontab, /etc/cron.d) some of which may call more scripts like /etc/cron.hourly, .daily...

Answer 4

That sounds like a cronjob that was created by crontab. Not all crontabs have a -u switch, but for GNU/Linux it is available. This is a handy line to list all cronjobs created by crontab.

for user in $(awk -F':' '{ print $1}' /etc/passwd); do crontab -u $user -l; done

(Run as root.)

Answer 5

If all else fails, you can make a honeypot out of URL it's requesting -- i.e. serve a large file or something -- and look for the process that's waiting to receive the data, and then look up its PPID.

Answer 6

Any decent system should explain the precise location of the crontabs in the manpage (usually in the FILES section near the end, and also for other dæmons).

On my system for instance, cron(8) contains the following:

 FILES
      /etc/crontab          system crontab file
      /var/cron/atjobs      directory containing at(1) jobs
      /var/cron/log         cron's log file
      /var/cron/tabs        directory containing individual crontab files
      /var/cron/tabs/.sock  used by crontab(1) to tell cron to check for
                            crontab changes immediately

And I second tylerl's advice to also check for at jobs while you're at it.

Answer 7

Approaching it the other way ’round: cron keeps a log of what it is doing, precisely to avoid this kind of problems in the first place. On my system, the log is kept in /var/cron/log and looks like this:

==> /var/cron/log <==
Oct 25 00:21:01 fortress cron[20232]: (vucar) CMD (/home/vucar/lighttpd-watchdog)

This line tells me that the cron instance with PID 20232 on machine fortress is executing /home/vucar/lighttpd-watchdog on behalf of user vucar. A well-behaved system only has a single cron running, so this will be straightforward.

This also works for at jobs, as they’re usually just handed to cron anyway:

==> /var/cron/log <==
Oct 25 00:28:01 fortress cron[31282]: (vucar) ATJOB (1414189680.c)

The snippets are from a BSD system, but the general concept is very likely the same everywhere else.