How to make a scheduled task based on a file creation time?

The supercomputer I run my Analysis uses SGE to do the job management and requires each job duration not longer than 24 hours. And this feature is causing me quite a headache.

Sadly, I was asked to use checkpoint mechanism to output results and resubmit the script qsub myJobScript. On program's part, it's not difficult for me to have it output a temporary file when the clock time hits 23:30:00 (I give the IO operation half of hour), let's say the output file name is called 1.00137/U and 1.00137/p, etc.
Note: when you submit a job, you never know when it is going to start. but it does have a log file under my folder when it is started.

So my question is how to use crontab or at or bash scripts to submit my job automatically when "new files" are created and detected? Any ideas?

-
What unix variant is it? For Linux, use an inotify-based mechanism, see Can a bash script be hooked to a file? –  Gilles Dec 11 '12 at 23:48

1 Answer

Run a simple script with cron every 5 minutes. Your crontab could look e.g. like this:

*/5 * * * *  /path/to/your/script


Script pseudocode:

IF (lockfile exists) { exit }
create lockfile
IF (old job has not finished yet) {
remove lockfile
exit
}
IF (new job has already been queued) {
remove lockfile
exit
}
schedule new task
remove lockfile


See Correct locking in shell scripts? for how to lock correctly (as race-condition-free as possible).

-