0

I've written 2 php scripts one worker and one master.

If I run master manually it checks a rabbitmq system and spawns 1 worker for each queue it detects. Then if I run master again it checks see if the each worker process is still running if it is then it does nothing if not then it restarts it.

Fairly simple concept I'd have thought. I achieved it using nohup and & in the master for each child it spawns, now as I'm sure you can guess i want to run master every 60 seconds to check the queues are alive and respawn them if they're not.

This gives me the problem of I can't use nohub in cron jobs and just using & on its own doesn't seem to work. As far as I can tell the workers aren't even execing with the call

I've tried creating a separate shell script that then calls master.php also didn't work. I tried creating it as an upstart task then remembered that in ubuntu 17+ upstart is removed.

I'm open to any suggestions of different ways of doing this but whatever route I take it must allow my master.php to spawn the work.php files as headless background processes.

  • 1
    Would it be acceptable for your master script to be running constantly, with a sleep 60 between each check? – JigglyNaga Aug 16 '18 at 14:17
  • yeah that would be fine although i'd worry then if the master died what would there be to restart the master. in my head having the master run and terminate then rerun from scratch every 60 seconds avoids that issue – Dave Aug 16 '18 at 14:32
  • in the master.php i could just put a while (true) in there and have it run forever but then again if it dies no one is around to restart it. have i explained that bit well its clear in my head :) – Dave Aug 16 '18 at 14:33
  • What's causing the master script to die? If processes are dying uncontrollably (eg. OOM killer) then whatever solution you install to rerun master may also be vulnerable. – JigglyNaga Aug 16 '18 at 15:48
  • Alternatively, please expand on "doesn't seem to work", "didn't work". You should include enough of your script for someone else to reproduce the problem, and the output you see (including any error messages). – JigglyNaga Aug 16 '18 at 15:57
1
#!/bin/bash
#I am just like a system daemon... 
#I do not need nohup for work, and I should not be called from cron.

#My name is main_safe
#I will become a separeted process even if my father dies... 
#i will check if main is still alive, and if dies i will restart it
#nohup is not needed inside shell script. 
#IMPORTANT: to die is very different from to freeze 
main_safe(){
trap "" HUP
while sleep 120; do
   main&
   wait 
done

}

#My name is main I like to keep restarting php master.php
#everytime it go away... remove wait and I will keep starting master.php with absolutely no reason.
#If you are paranoid you can program me to restart main_safe,
#But what will happen if you try to stop me? Bad things.. so... 
#IMPORTANT: to die is very different from to freeze 
main(){
trap "" HUP
while sleep 60; do

     php master.php & 
     #do whatever you want here

     #uncomment this to prevent two instances of master.php from going up maybe it is necessary:
     wait

done
}


#nohup is not needed in shell script
main_safe& 
pstree -p | grep $! 

Is that acceptable?

  • more i've thought about it the more thats not really a solution as if main dies for whatever reason there's nothing to restart it, i could achieve the same by just putting while(true) in the php and then nohup & that on terminal – Dave Aug 16 '18 at 14:49
  • Well, and what will restart nothing if "nothing" dies ... ? Do you still want this "nothing" considering this? If yes, please leave a comment, and i will do program this "nothing"... So the script will be restarted... – Luciano Andress Martini Aug 16 '18 at 14:52
  • If you are talking about the php script dying so uncommenting #wait should be enough. When the php scripts dies, or terminate, the loop will continue and the php script will be called again... – Luciano Andress Martini Aug 16 '18 at 14:55
  • no i'm talking about the shell script dying as i'd have to manually run the shell script once and then it just loops infinitely running master every 60 seconds which is fine but if the shell script dies there's then no watchdog to re-run the shell script. the result is the same as me putting the loop in the php script and not bothering with the shell script at all. if the parent process dies there's nothing to restart it. – Dave Aug 16 '18 at 15:15
  • 1
    I just can't imagine why the shell script will die. And why the watchdog do not. I have a script like this running for 4 years. Considering how simple is this script.... But it is not impossible to make a "hard to die" script or multiple scripts, or even a program, the problem is similar, the watchdog will be in fact a loop too. Maybe you can have two scripts, one checking each other, is that acceptable? If yes, please let me know. – Luciano Andress Martini Aug 16 '18 at 16:26
1

How about using systemd. I did some searching, and as systemd replaces upstart for ubuntu's newer versions, this may be useful. The link below may be helpful.

Digital Ocean: How To Configure a Linux Service to Start Automatically After a Crash or Reboot

  • I tried that using a timer service systemd has the same issue as cron it would appear a master running script cannot spawn and background child scripts so putting master.php in systemd every 60 seconds and then that launching children with & on the end to background results in the children not execing because of the & – Dave Aug 16 '18 at 15:14
  • can't use docker as the server is already an lxc container. i'm not doing container inception on a live system handling 1 mill transactions an hour. – Dave Aug 16 '18 at 22:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.