1

I'm running a resque worker service on Ubuntu, and I want to manage it using upstart (the default process manager for Ubuntu).

A proper shutdown sequence for a resque worker is to send the QUIT signal first, and wait for the process to finish any running job first. It should then stop itself. If the worker doesn't shutdown in a timely manner - because the job is still running (possibly stuck), I need to send a TERM signal which will cause the worker process to abort the job and exit. And obviously, if that doesn't work, a KILL signal would have to be sent.

The way I thought to implement this is to have a pre-start script to send the QUIT signal and then sleep until either the process exists or a timeout expires.

But in effect what happens is that if the worker responds to QUIT and exits, respawn kicks in and restarts the process which apparently causes the stopping process to not happen.

Is there a way to get upstart to not respawn if we are stopping the process anyway?

2

There are two ways that I found to tackle the problem, neither is perfect:

  1. Upstart has the kill signal stanza. If you set kill signal QUIT and also kill timeout 600, Upstart will send SIGQUIT to the process, then wait 600 seconds before sending SIGKILL. If the process terminates before SIGKILL is sent, then we're good. There are issues with that approach: (a) it sends SIGKILL and not SIGTERM after the timeout which doesn't allow the resque worker to close nicely, but that's a no biggie. (b) A worse problem that I found is that upstart sends the "kill signal" (in our case SIGQUIT) to all processes in the session, which will likely interfere with the operation of child processes of the resque worker (SIGQUIT default action is to terminate with a core dump).
  2. Implement your own wrapper that traps whatever signal your configured upstart to send (or just assume the default SIGTERM) and handle all the graceful shutdown yourself. In that case its important to set the Upstart kill timeout to a sufficient time to let your wrapper do a graceful shutdown and either force the abort with the time is up (in which case make sure to have the timeout slightly less than Upstart's) or just rely on Upstart's SIGKILL for the cleanup (which may or may not be a good idea depending on your requirements). Cons: its rather complicated to get this right, and the reason Upstart exists is so you won't have to write a process manager yourself for every service.

If you don't want to go either way, there are other process managers out there that may implement a more complex signal regime than Upstart support, and these are usually easy to implement under Upstart to manage just the process you are having a problem with. Unfortunately, its hard to find one that works correctly in all cases. For example I've tried Bluepill, which looks great on paper but at the time of writing has a glaring bug where if you try a complex signal regime with multiple signals and multiple timeouts, it just sends all the signals at once (not necessarily in the correct order) and crashes the child process.

If someone else has more information to add, feel free to add more answers and I may even mark it as "answered" (instead of my self answer) if its good.

0

From Upstart cookbook:

Since the job's goal will already be 'stop' when a pre-stop is run, you can shutdown the process through any means, and the process won't be re-spawned (even with the respawn stanza).

You could try sending QUIT in the pre-stop section of your script.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.