I'm trying to chain some commands to periodically check/launch two processes via cron (I'm on a shared host, can't change things around). After a lot of googling around, all the things I've done don't work properly:

Trying to launch them separately in two cron jobs cascades and crushes the server (maybe because the grep command doesn't manage to catch the search words because they are parameters, I don't understand):

ps -fu $USER | grep celeryd >/dev/null || $HOME/python27/bin/python $HOME/utilities/manage.py celeryd -E -B --concurrency=1
ps -fu $USER | grep celerycam >/dev/null || $HOME/python27/bin/python $HOME/utilities/manage.py celerycam

And these other variants where I try to launch the processes together on the same cron, launch only the first process:

ps -u $USER | grep python >/dev/null || $HOME/python27/bin/python $HOME/utilities/manage.py celeryd -E -B --concurrency=1 & $HOME/python27/bin/python $HOME/utilities/manage.py celerycam &

ps -u $USER | grep python >/dev/null || ( $HOME/python27/bin/python $HOME/utilities/manage.py celeryd -E -B --concurrency=1 & $HOME/python27/bin/python $HOME/utilities/manage.py celerycam; )

ps -u $USER | grep python >/dev/null || ( $HOME/python27/bin/python $HOME/utilities/manage.py celeryd -E -B --concurrency=1; ) sleep 30s; ( $HOME/python27/bin/python $HOME/utilities/manage.py celerycam; )

I gave up trying to use grep and took advantage of the --pidfile option of the celery worker; celerycam doesn't allow multiple instances so I don't need any grep there to check for it:

$HOME/python27/bin/python $HOME/utilities/manage.py celery worker -E -B --concurrency=1 --pidfile=worker.pid
$HOME/python27/bin/python $HOME/utilities/manage.py celerycam
  • Have you try to run your celery processes in screen/tmux sessions, that might be more easier to manage than cron jobs – number5 Aug 6 '13 at 23:51
  • @number5 yes screen was the solution I used before, but they either get automatically axed by my host after an amount of time or the server restarts too often (2,3 times during the day) – symbiotech Aug 7 '13 at 0:21
  • Yep I understand the pain to work in shared hosting env (Dream****), the ultimate solution might be to switch to some real VPS... (I know sometime it not an option either) – number5 Aug 7 '13 at 1:07

$USER is not set in most crons. Luckily ps -u without a username defaults to the present user (which is the user from which cron is running for). However your grep has quite a high chance to match the grep itself as well as the celery processes. You can clean that issue up with a sneaky grep '[c]elery'. This will match processes with names/arguments that contain celery only (and not the grep which has argument [c]elery). The other issue I can see is that because you run the program with a giant path to manage.py you may be missing the "celery" part in your ps output. That can be remedied by adding more verbosity to the output of ps with the -w flag (the more of them there are the more verbose it becomes). So The suggested fix is:

ps fwwwu | grep '[c]eleryd' >/dev/null || $HOME/python27/bin/python $HOME/utilities/manage.py celeryd -E -B --concurrency=1
ps fwwwu  | grep '[c]elerycam' >/dev/null || $HOME/python27/bin/python $HOME/utilities/manage.py celerycam

EDIT: Removed the extraneous dash from the ps arguments above and added pgrep as a viable alternative.

Alternatively you can also make use of two other commands pidof and pgrep to get the process ids of running processes. I think in this instance (if your username is fred) you can replace the ps command with:

pgrep -fu fred celeryd

to achieve similar results.

EDIT 2: Add single quotes around '[c]elery' to stop shell globbing (the directory where the cron is ran from (the users $HOME) may contain a file that is called celery.

  • Thanks for your great explanation, it's much clear for me now what happens there. Unfortunately ps returns this error: "Warning: bad syntax, perhaps a bogus '-'?". Could be some version related issues? – symbiotech Aug 6 '13 at 22:01
  • Ah sorry that's a habbit I've never managed to shake, just use ps fwwwu (other 'flavors' of unix start ps arguments with a dash and I started on those - I just put the dash in by habbit :) – Drav Sloan Aug 6 '13 at 23:10
  • both "ps fwwwu" and "ps -fwwwu $USER" (with fwwu or fwwwu) work in the shell, but neither work with cron. I think cron might be limited in some way. Anyway thank you very much :) – symbiotech Aug 6 '13 at 23:40
  • Either that could be the $PATH does not include the directory where ps is or you could add the user to filter for with 'ps -fwwwwu fred' if the username is fred. I'll update the answer to give some alternative solutions. – Drav Sloan Aug 6 '13 at 23:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.