20

I'm writing systemd unit files for OSSEC HIDS. The problem is that when systemd starts the services it immediately stops them.

When I use the following ExecStart directive everything is working fine.

ExecStart=/var/ossec/bin/ossec-control start

But when I make ths following small improvement, I find in OSSEC logs that it receives SIG 15 after start.

ExecStart=/bin/sh -c '${DIRECTORY}/bin/ossec-control start'

If I make another small change, the service will receive SIG 15 after 20 seconds.

ExecStart=/bin/sh -c '${DIRECTORY}/bin/ossec-control start && sleep 20'

So, I guess, that systemd kills /bin/sh process after service start, and /bin/sh then kills OSSEC.

How can I solve this problem?

2
  • 1
    What's the Type of the service?
    – Wieland
    May 4, 2015 at 13:01
  • @Wieland, I was try simple and forking, but result is still the same. May 4, 2015 at 16:04

4 Answers 4

44

readiness protocol mismatch

As Wieland implied, the Type of the service is important. That setting denotes what readiness protocol systemd expects the service to speak. A simple service is assumed to be immediately ready. A forking service is taken to be ready after its initial process forks a child and then exits. A dbus service is taken to be ready when a server appears on the Desktop Bus. And so forth.

If you don't get the readiness protocol declared in the service unit to match what the service does, then things go awry. Readiness protocol mismatches cause services not to start correctly, or (more usually) to be (mis-)diagnosed by systemd as failing. When a service is seen as failing to start systemd ensures that every orphaned additional process of the service that might have been left running as part of the failure (from its point of view) is killed in order to bring the service properly back to the inactive state.

You're doing exactly this.

First of all, the simple stuff: sh -c doesn't match Type=simple or Type=forking.

In the simple protocol, the initial process is taken to be the service process. But in fact a sh -c wrapper runs the actual service program as a child process. So MAINPID goes wrong and ExecReload stops working, for starters. When using Type=simple, one must either use sh -c 'exec …' or not use sh -c in the first place. The latter is more often the correct course than some people think.

sh -c doesn't match Type=forking either. The readiness protocol for a forking service is quite specific. The initial process has to fork a child, and then exit. systemd applies a timeout to this protocol. If the initial process doesn't fork within the allotted time, it's a failure to become ready. If the initial process doesn't exit within the allotted time, that too is a failure.

the unnecessary horror that is ossec-control

Which brings us to the complex stuff: that ossec-control script.

It turns out that it's a System 5 rc script that forks off between 4 and 10 processes, which themselves in their turn fork and exit too. It's one of those System 5 rc scripts that attempts to manage a whole set of server processes in one single script, with for loops, race conditions, arbitrary sleeps to try to avoid them, failure modes that can choke the system in a half-started state, and all of the other horrors that got people inventing things like the AIX System Resource Controller and daemontools two decades ago. And let's not forget the hidden shell script in a binary directory that it rewrites on the fly, to implement idiosyncratic enable and disable verbs.

So when you /bin/sh -c '/var/ossec/bin/ossec-control start' what happens is that:

  1. systemd forks what it expects to be the service process.
  2. That's the shell, which forks ossec-control.
  3. That in turn forks between 4 and 10 grandchildren.
  4. The grandchildren all fork and exit in turn.
  5. The great-grandchildren all fork and exit in parallel.
  6. ossec-control exits.
  7. The first shell exits.
  8. The service processes were the great-great-grandchildren, but because this way of working matches neither the forking nor the simple readiness protocol, systemd considers the service as a whole to have failed and shuts it back down.

None of this horror is actually necessary under systemd at all. None of it.

a systemd template service unit

Instead, one writes a very simple template unit:

[Unit]
Description=The OSSEC HIDS %i server
After=network.target 

[Service]
Type=simple
ExecStartPre=/usr/bin/env /var/ossec/bin/%p-%i -t
ExecStart=/usr/bin/env /var/ossec/bin/%p-%i -f

[Install]
WantedBy=multi-user.target

Save this this as /etc/systemd/system/[email protected].

The various actual services are instantiations of this template, named:

Then enable and disable function comes straight from the service management system (with RedHat bug 752774 fixed), with no need for hidden shell scripts.

 systemctl enable ossec@dbd ossec@agentlessd ossec@csyslogd ossec@maild ossec@execd ossec@analysisd ossec@logcollector ossec@remoted ossec@syscheckd ossec@monitord

Moreover, systemd gets to know about, and to track, each actual service directly. It can filter their logs with journalctl -u. It can know when an individual service has failed. It knows what services are supposed to be enabled and running.

By the way: Type=simple and the -f option are as right here as they are in many other cases. Very few services in the wild actually signal their readiness by dint of the exit, and these here are not such cases either. But that's what the forking type means. Services in the wild in the main just fork and exit because of some mistaken received wisdom notion that that's what dæmons are supposed to do. In fact, it's not. It hasn't been since the 1990s. It's time to catch up.

Further reading

7
  • 3
    Very detailed answer! I'd also suggest to create a "grouping" target, say, ossec.target, which Requires= all the needed instances, and then set PartOf=ossec.target in [email protected]. This will allow to start and stop ossec by starting and stopping ossec.target.
    – intelfx
    May 4, 2015 at 18:45
  • @JdeBP, wow! Thank a lot for such kind of detailed answer. Hope I'll make this unit and write here about results. I was though, that I'll be easier. But you are right, ossec-controll is a init hell. May 4, 2015 at 22:14
  • 1
    What's the reason for using /usr/bin/env as a wrapper? Mar 30, 2019 at 16:18
  • Got this issue after installing a fresh newer version of Ubuntu. A service with a config that worked normally would be deactivaded. After inserting Type=forking it started working normally.
    – culebrón
    Oct 16, 2021 at 17:21
  • If the initial process doesn't fork within the allotted time, it's a failure to become ready. If the initial process doesn't exit within the allotted time, that too is a failure. – what are these timeouts' lengths? Jan 5, 2022 at 11:17
1

Keep Type=forking and give a PID file location if start service/app is maintaining any pid.

[Unit]
Description="Run app on boot"
After=network.target syslog.target auditd.service

[Service]
Type=forking
PIDFile=/var/run/apache2/apache2.pid
ExecStart=/etc/init.d/apache2 start
ExecStop=/etc/init.d/apache2 stop
StandardOutput=syslog
StandardError=syslog
Restart=on-failure
SyslogIdentifier=webappslog

[Install]
WantedBy=multi-user.target
Alias=webapps
0

Somewhat related, I had a systemd service that it appeared that systemd would "kill" it after 30s.

systemctl status service-name would show main process exited, code=exited, status=1/FAILURE after 30s had elapsed.

It would run fine "in isolation" (like manually in the terminal with the same environment).

Turns out it was

Type=forking
...
Environment=ABC="TRUE"
ExecStart=/path/to/my_script_to_spawn_process.sh

within my_script_to_spawn_process.sh it was doing

/bin/something > /dev/null 2>&1 &

which works but was discarding output log information (normally it goes to a file, or, if not that, possibly journalctl).

Changing it to log to somewhere else like /bin/something > /tmp/my_file

then tailing the /tmp/my_file revealed the actual cause. Which was (tangentially) that you can't use the syntax Environment=ABC="true" like you can in bash, it has to be no quotes or the key value all within quotes like Environment="ABC=true" which was causing my process to exit "in its setup phase" after about 30s.

-4

Note that systemd's daemon model is simplistic and incompatible with many existing daemons which do multiple forking, exec'ing and setuid'ing. Most common are the daemons which start as root to set things up and then switch to a less privileged UID for routine operation. e.g. Pid file initialization is one thing which fails under systemd due to privilege problems. There are workarounds (not fixes) but it's badly documented.

JdeBP's explanation is welcome but incomplete and his claim that it's all ossec-control's fault is simply not true. Even quite trivial stuff is problematic e.g. getting untruncated log lines to debug problems or meaningful error messages from systemd itself when it kills processes.

1
  • 2
    What use are PID files, anyway? If one exists for a given service, there may or may not be an actual process with that PID, and when a process with the right PID does exist, it may or may not actually be the expected service.
    – JoostM
    Nov 11, 2018 at 10:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .