The best way to learn how a system works is to break it in all possible ways.
— either anonymous or the younger myself, I don't really remember.
So, your experiment is perfectly fine, and indeed has a significant didactic value. Many people treat systemd as a piece of technology indistinguishable from magic. It's wrong: it's a not very consistent, but very well documented system service manager. It's tricky, and requires reading the manuals. They are exquisitely complete.
The answer to your question is, as it's common with systemd, it depends. systemd is a complex and flexible system service manager. And the main thing it depends on is the type of the service, defined by Type= and the absence/presence of BusName=. Without both, as is your case, the service is simple: systemd just runs it, assuming that startup has been successful, even it it dies 10ms after starting. It simply has no way of knowing that it will, and its main design goal is to bring up the system to a working state on modern 56-core CPUs as quickly as possible. Who'd waste 10ms...
There are many different protocols for starting the service, because systemd wants to know as precisely as possible that the service has in fact successfully started. They are described under the Type= setting in the systemd.service(5) manual. Of these, only oneshot type is different: the service is considered "active" by systemd, while it executed quickly and terminated with exit code 0. This makes sense if you want to do something when a unit is started, and when something else it's stopped, but not in between. If successful, it's active and has a green button if you check its status. When you stop it, it executes its ExecStop= thing. For example, Linux NFS4 server is a kernel facility, and its service systemd unit only enables when starting, and disables when stopping, a few ns to call the kernel and terminate. There is no process than needs to be running all the time the NFS server is serving. Its unit is active but has no running userspace process.
For all other types, absent ExecStop=, the behavior of initiating service shutdown is the same: the service is sent a signal. The signal is defined by KillSignal=, (or RestartKillSignal= for restart), or, if not set, the standard service termination signal SIGTERM (see systemd.kill(5)). This is how all "traditional" Unix daemons had been told to terminate since raptors roamed the Earth. If the service fails to stop, protocols also differ: for example, a service communicating with systemd over dbus may request more time to shutdown. Without that, the setting TimeoutStopSec= is used, or the default set in system.conf in DefaultTimeoutStopSec=(system.conf(5) in the manual), or a compile-time default, commonly 90s unless changed by the system supplier. After that, systemd rudely kills the process with SIGKILL, which is impossible to mask or ignore: the process just dies, it's memory that can be freed is freed, it is considered terminated and becomes a zombie; then, as soon as all kernel modules lose interest in it (e.g., if it was killed during disk or GPU DMA transfer, inside a read(3), the kernel would notify the driver module to interrupt or complete the transfer ASAP), is erased from all system tables. In other words, it's terminated as any other process. There are subtleties related (a) to its child processes and (b) its control group, but you can read the manuals for these gory details.
Going back to your self-inflicted problem:
- Without
Type= and BusName=, the service is simple.
- Absent
ExecStop=, a simple service is sent the a SIGTERM signal, and given (likely) 90s to terminate itself and its child processes.
- If it fails to do so, systemd goes full samurai on it, and kills it merciless with SIGKILL.
Your example is a very good one indeed! Let's parse it line-by-line.
Feb 28 19:56:00 esm-dev systemd[1]: Stopping server integrity checker...
Feb 28 19:57:30 esm-dev systemd[1]: serviceDaemon.service: State 'stop-sigterm' timed out. Killing.
Note that the time passed between the two messages is exactly 90s. The service is apparently not designed to stop upon receiving SIGTERM. The second line suggests an internal systemd state of having had sent SIGTERM, waiting, and having timed out. "Killing" means sending the SIGKILL signal, which your process does not even notice: it's a signal to the kernel to get rid of that thing as fast as possible.
Feb 28 19:57:30 esm-dev systemd[1]: serviceDaemon.service: Killing process 1235476 (intchecker) with signal SIGKILL.
Feb 28 19:57:30 esm-dev systemd[1]: serviceDaemon.service: Killing process 1235478 (intchecker) with signal SIGKILL.
Feb 28 19:57:30 esm-dev systemd[1]: serviceDaemon.service: Killing process 1235479 (intchecker) with signal SIGKILL.
systemd is smart enough to make do with the whole process' spawn. All its child processes are also unconditionally killed at the same time (only the main PID talks to systemd, child processes are not systemd's business, unless misbehaving).
Feb 28 19:57:30 esm-dev systemd[1]: serviceDaemon.service: Main process exited, code=killed, status=9/KILL
As soon as the process and everything it has started under its group lead has been killed with SIGKILL (9 is the numeric value of SIGKILL; kill -SIGKILL <pid>, kill -KILL <pid> and kill -9 <pid> are totally identical commands), zombified or not (that's the kernel's problem, not systemd's), systemd internally sets the service as failed:
Feb 28 19:57:30 esm-dev systemd[1]: serviceDaemon.service: Failed with result 'timeout'.
which now allows it to proceed with its dependency graph.
Note that, as soon as the carnage started, all events are timestamped 19:57:30. systemd's problem is now to get rid of the rebelling process as soon as possible. And it's very good at it, so you need to establish a handler for SIGTERM early when starting service. Better cooperate.
As a service author, you have multiple options. A simple service is the fastest but the least informative: systemd posts it as started as soon as it forks off an internal unit of work, or job, to start the binary. Even if the ExecStart= file is missing, your service starts successfully. It will show a red button, and the overall "degraded" status, but won't hold start-up of other services.
The next more certain but slower one is the exec type: systemd will ensure that the process has started before posting a success.
From this point, decide whether you want to talk to systemd via dbus. It is beneficial to do so, if you know that the system is managed by systemd. You commonly link to libsystemd.so.1, which has high-level API for that, but it's possible to do without it—it's just a UNIX socket with a well-known name. I've seen code that doesn't want to depend on this library, but talks to dbus if its present, on its own. This is very rare, but allows the same binary to be started by either systemd or traditional init without recompiling.
With dbus, the next level up the certainty is the dbus type. systemd considers the service started when it establishes connection to dbus (so don't linger, do it early: it affects whole system boot-up).
The next up is the notify service. systemd posts success only after the service posted READY=1 to its dbus connection. This is the most informative to systemd, as it has a positive confirmation of service startup.
Neither type needs to fork(2).
Otherwise, you may implement the service as a "traditional" self-daemonising process. You perform all initialization after starting, and then, if everything is ready to go, fork(2) the process into two clones. The original one exits and returns successful status to systemd. The second remains running, and is the main long-running daemon. Traditionally, these write a <servicename>.pid file under the /run filesystem, but systemd normally finds the forked process without that. If you save one, tell systemd its location with the PIDFile= setting. systemd will remove it if you leave it behind. This file is needed only for the "traditional" init/rc system. The Type= of such a service is forking. It also provides positive startup indication to systemd, like the notify type. No other long-running type does (oneshot does too, however, and its startup commands should complete as quickly as only possible).
Note that this ordering is w.r.t. systemd's certainty that the service has started. From the boot-up speed perspective, the scale is nearly reversed, and the simple type is the best for performance. So, choose to be fast if you know that other services don't depend on your service having completed its initialization. Otherwise, go with the most informative, forking or notify, keeping in mind that this will impact the system boot up time the most.
