I know daemons have to be sent a HUP for config changes to take effect. But I'm wondering why this is, and if it is possible to create a daemon responsive to such changes.

  • 12
    Different programs act differently, depending on their description in their manuals. This feels like too broad of a question. – Jeff Schaller May 18 '20 at 13:30
  • The first point in Steven's answer is the most important IMO. There are a number of configuration files which make no sense on their own. They "include" other snippets from elsewhere. Now if you reload the daemon if the "main" file changes before it can be validated in-place (and faulty changes rolled back), you'll run into situations where you just stopped your daemon and it can't start up again due to the faulty configuration which was about to be validated (and would have failed and been rolled back). So you'd literally cause your own denial of service as a result. – 0xC0000022L May 18 '20 at 15:06
  • 2
    @0xC0000022L: Bonus points if the daemon whose files you're currently editing is SSHd … – Jörg W Mittag May 19 '20 at 3:13
  • 1
    @JörgWMittag hehe, yep. Been there, done that. It's a lesson you learn quickly, once you make that sort of mistake. But for sshd the remedy is easy (as the upgrade installer for Ubuntu proves). – 0xC0000022L May 19 '20 at 6:14
  • 2
    Some daemons actually do reread the config at some interval, like the samba daemon. See comment in serverfault.com/questions/759552/… – tetra May 19 '20 at 10:42

There are multiple reasons. One major reason is that many daemons have multiple configuration files, and any single file change might not be usable on its own — so having a daemon attempt to reload its configuration whenever one of its configuration file changes might cause more problems than it solves.

From a purely implementation-related standpoint, having to watch for changes to configuration files adds more complexity to the daemon. Daemons have a central loop of some sort, checking for work to be done corresponding to the daemon’s core purpose; checking for changes to configuration files doesn’t necessarily fit nicely into that core purpose.

Handling a separate signal solves both of these problems: it indicates that the user thinks the configuration is coherent, and is safe to reload, and it can be implemented asynchronously in a signal handler (typically as a basic flag change), while minimising the impact on the main loop (it reacts to the flag change).

There are daemons which react to configuration changes on their own; cron for example checks its configuration files for changes every time it goes round its main loop.

  • I suspect it possibly checks for a change in size or timestamp on each cycle, but only reads files that appear to have changed. On some OS, the crontab editor automatically sends a SIGHUP to the daemon after it writes a changed (and validated) file. – Paul_Pedant May 18 '20 at 14:13
  • Indeed, thanks @Paul_Pedant, Vixie cron at least only loads files if it believes they might have changed. – Stephen Kitt May 18 '20 at 14:35
  • Checking if a configuration file needs to reloaded is checking for work to be done. I surely wouldn't want to re-read a config file directly from a signal handler, since there'd be no telling what else is going on at the time. – ilkkachu May 18 '20 at 14:47
  • 6
    The problem you describe for multiple config files can also happen when there's only one configuration file. Because locking is rarely used, the daemon could notice the file changed before it's been completely saved by the editor, leading to the daemon loading a partial config file. – ikegami May 18 '20 at 22:23
  • 1
    Locking isn't the real problem. The problem is in assuming that people always save a file exactly once, after they've done all necessary edits and checked for typos. That's not what happens. – user1686 May 19 '20 at 11:14

Apart from all the other reasons mentioned in the other answers, there is a more deep, philosophical reason, and that is one of the fundamental Unix programming design principles: Do One Thing And Do It Well.

In Unix, programs generally do one thing, and one thing only. More complex operations are achieved by combining multiple programs.

Now, serving web sites (for example) that is one thing. Watching files for changes, that is another thing. So, according to the Unix motto, those should be two different programs, because a web server that e.g. serves web sites and watches files for changes wouldn't be doing One Thing And Do It Well, it would be doing two things. And programs that do two things very often do at least one of those two things not very well. (For example, the person who wrote the web server might be an expert in HTTP, but not an expert in file watching.)

For network-facing daemons specifically, there is also another reason to keep to the Do One Thing mantra: security. Every single line of code is a potential bug. The most secure code is no code. By removing responsibilities from the daemon that do not belong there, you reduce the amount of code, and thus the attack surface.

Apart from the fact that programs that do multiple things often do at least one of those multiple things badly, there is another reason for splitting the responsibilities: code reuse. If file watching were part of the web server, then your SSH server also needs to implement file watching. And your file server. And your chat server. And your telephony server. And your database server. And your media streaming server. And so on, and so forth.

Whereas if file watching is the responsibility of a separate program, then this program only needs to be implemented once, tested once, optimized once, documented once, etc. You also only need to train people once how to use it, and then they can apply it to every single daemon ever written, and in fact even to every single daemon that will be written in the future.

So, if you wanted to do what you ask in your question, you would have two daemons: one that, e.g. serves web sites, and one that watches files for changes and executes an action based on those changes.

And, it probably shouldn't come as a surprise, such daemons already exist.

  • "If file watching were part of the web server, then your SSH server also needs to implement file watching." - no, it just needs to use a common library. Like many other common libraries it already uses. "Combing multiple programs" vs "combining multiple libraries" is just a matter of taste. – user11153 May 19 '20 at 15:38
  • 1
    @user11153 It is not just a matter of taste. "Send a HUP signal when you want to reload configuration" is a totally different level of complexity than "recompile the entire program to fix a bug in its statically linked library" or "stop the program, update its dynamically linked library, and restart the program" which can also affect other programs on the same system. – kbolino May 19 '20 at 16:55
  • If only this terrific Unix principle was still followed when developing daemons ... (cough)systemd(cough) – davidbak May 19 '20 at 17:55
  • 2
    @davidbak: Well, we can argue that Unix itself abandoned the Unix principles when they made network sockets "something close to a file" instead of "a file". See Plan9 and Inferno for an alternative approach where sockets are just files, the network stack is just a filesystem, and you do not need NAT, because the internal client can just mount the gateway router's network stack as a filesystem and make TCP connections directly from the router. – Jörg W Mittag May 19 '20 at 18:16

Adding to other answers: One of the biggest obstacles would be in requiring each configuration value to have an interpretation that's aware of, at minimum, the current state of the application, the previous value, and the other values changed. In general the config file instructs the daemon on how to go from non-running state to its initial state. For the daemon to reload its config on changes means also being able to interpret the config values as modifications to the daemon's current state. For example, let's consider some kind of task daemon. In it's config could be worker_threads: 2. If the daemon is just starting up, the interpretation is simple: create two threads for executing tasks. If you need to handle a modification to that value, now you need to do multiple things - if the thread count was increased, add more threads. If it was decreased, remove some threads.

Not only is the interpretation of the change dependent on the previous value, it's also dependent on the current state of the daemon. If you update the config to worker_threads: 1, then what should the daemon do if both threads are in use? Should one be killed to immediately match the state described in the config? Should the tasks be allowed to run to completion and then one thread is killed?

Past that, the interpretation of the config value change is also dependent on which other config values changed. Let's say there's an additional config value to determine what to do when we decrease the worker thread count while all workers are in use: kill_threads_on_worker_reduction: true/false. Well if that's set we can determine whether to kill the active worker threads or not. But what if that config value was also changed? Does it only apply going forward to new tasks being executed or does it apply retroactively to tasks started before the change?

This is just drilling into one possible config value. You would have to figure this out for every config value, for every possible application state, and for every possible combination of changed values.

While there are certainly cases where the interpretation of config changes are much more straightforward, this helps explain why live-reloading config changes isn't something that generally can be easily done.

  • Indeed, figuring out how to go from an arbitrary state A to an arbitrary state B, subject to some non-trivial constraints (e.g. the daemon should be working the whole time) is a complex operation. Complex both in the sense "I wouldn't even know how to approach the problem" and in "I wouldn't be surprised if you told me it is NP-hard". Better to just drop and reload. – Jörg W Mittag May 19 '20 at 19:40

Not the answer you're looking for? Browse other questions tagged or ask your own question.