One problem with cron is that if there are more jobs that can reasonably be run at a certain time, cron doesn't care: it will happily spawn the jobs anyway resulting in poor performance for all of them.

Suppose I want to write a cron-like scheduler, but one that limits spawning to only what the machine can actually handle. Jobs that currently can't be handled are deferred and retried.

For example, suppose there are, say, 10 jobs that run every 5 minutes. The machine can actually only handle, say, 5 at a time. At, say, 11:05, rather than spawning all 10 jobs, it would spawn only 5. However, every few seconds, it would see if the CPU load has come down (because some of the original 5 jobs have completed). If so, it would then spawn some of the previously deferred jobs. It would repeat this until either all the jobs that were supposed to run at 11:05 have run; or, for jobs that couldn't be run before the next instance of them will run (in this case, at 11:10) will simply not be run (skipped).

If I want to constrain all such jobs to use no more than, say, 50% of the CPU, it would seem that using a cgroup would be a way to accomplish this.

The question: is there a way to inquire how much of the current cgroup CPU capacity is being used? For example, if the scheduler spawned 5 jobs and those 5 jobs in total are using only, say, 80% of the 50%, then the scheduler could probably spawn at least one more job since 20% of the 50% is currently being wasted.

  • 2
    FWIW and for the interest of people coming across this question later in search of scheduling solutions, at/batch do a related thing: batch tasks are run when load is below 0.8, and they offer queues with increased niceness, i.e. the jobs will cede to higher-priority jobs. This would not cause jobs to be skipped entirely, but it can use 100% of the machine if it is otherwise idle. – Ulrich Schwarz Aug 10 '16 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.