3

I am working on migrating some old System V type services over to "real" systemd services on RHEL 7/8. I have things running for Redhat 7, SLES 12 and SLES 15 just fine. But when I try to get a service running on RHEL 8, I found the system now was requiring my application's pidfile to be in the /run directory...or at least it wants it there. (Our applications typically were writing the pidfile in the application's installation directories.)

I found I was able to accomplish this by changing the application startup scripts and writing to the /run directory. So Success! But now there is one problem that I cannot seem to get over. I need my running services to be running in the context of a particular user (my login id), and not as root. All my research has told me I simply need to specify User= in the .service file in order to accomplish this. But whenever I add that one line to the file, my service start fails. It seems it is failing at the point where the pidfile is written to the /run directory. Pidfile can't be written, the process exits.

My service file:

Description={removed}
After=remote-fs.target
After=network-online.target
Wants=remote-fs.target
Wants=network-online.target

[Service]
Type=forking
Restart=no
User=myid
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=none
GuessMainPID=no
RemainAfterExit=no
SuccessExitStatus=5 6 255
PIDFile=/run/adidmn.pid
ExecStart=<fullPathToScript> start
ExecStop=<fullPathToScript> stop

[Install]
WantedBy=multi-user.target

My start script defines how the daemon is to be run on various platforms, sets some environment variables and in turn calls a shell script which starts the actual process. The called daemon script is runnable directly when called with sudo privilege. My userid is in the sudoers list, but of course, again the owner of the running process is root.

Without the "User=" property in the .service file, I have no problems starting the service. I do always run the systemctl commands as sudo under my own userid. My end goal is to see the actual running process as running under my userid and not as root. I thought adding User=<myid> to the .service file would accomplish this, but that one line makes the service start fail.

The user exists (my own userid) and is also in the sudoers list. /run is owned by root and when the pid file is written there using sudo, the owner of the pid file is root. I tried using su <userid> for the command in my start script, but that resulted in the daemon not starting at all.

Basically, I have different behavior between SuSE and Redhat. My daemon has (for years) come up with a pid file in the installation path of the program. The process is started using su <userid> -c command that starts the process in the context of user "a". The pid file is placed in the application's install path and owned by user "a". SuSE Linux: no problem at all. However, on Redhat, while the process runs, systemd complains with:

New main PID 26979 does not belong to service, and PID file is not owned by root. Refusing.

Why the difference? Is what I am trying to accomplish "doable?" or is it necessary that root now owns all systemd service processes?

2

The complication was the "su -c" command in the underlying script that started the daemon. System did not like that. But since the script has to work on other platforms as well, I modified it with a case statement for Linux. Since the systemctl is run under sudo anyway, it is not a problem. Now pid file can be written wherever I need it to be and the system seems happy.

  • Since systemd already starts it as another user, there's no need for su. Your case statement should check the current user and not the current system. – muru Aug 9 at 1:53
  • Thanks muru. I should have been clearer...the reason my script uses a case statement for the system is that the same script has to work on multiple OS paltforms, i.e. AIX/HP/Solaris. Since those use System V and not systemd for services, there are several of these case statements for how the service is run. In the case of the older System V, the su command was necessary -- or at least that's how the original authors implemented it and I am not at this point going to fool with what works ;-). – BobP Aug 9 at 15:51
2

The problem you hit is that System V init scripts expect to run as root, that's part of the "specification" on how they work and they'll often have steps that need root to complete.

In your case, it was about running su <userid> -c ... to actually start running as the non-root user, but that part actually fails if you're already running under that user. System V init scripts will often use tools such as su or runas or similar to switch to a non-root user, but these tools are often not perfectly suited for that purpose (su is originally meant to run from an interactive shell and will integrate with PAM which doesn't make much sense here.)

Even worse, some System V init scripts will not deal with changing user and will end up unnecessarily running a daemon as root, since that's what feels more "natural" in a System V init script. This is, in my opinion, one of the worst issues of System V init scripts, they make it easy to do the wrong thing here, and hard to do the right one.

If you want to keep compatibility with a System V init script, you could just run them as root from systemd, since that's the "protocol" when invoking such scripts. In fact, if you wish to keep compatibility, you don't even need to ship a systemd unit, since systemd will be able to generate one by itself through systemd-sysv-generator. The generated unit will look a lot like the one you presented, except that it will be run by root instead.

If you do want to ship a systemd service unit (which I'd recommend that you do), then you should seriously consider shipping a unit that uses Type=simple, rather than forking.

The only pre-requisite for that is that you're able to start the daemon in foreground, which many daemons are able to do by passing an extra command-line flag or through some configuration. (It actually takes considerably less effort to do so, so if your daemon currently doesn't support that and you have control of the sources, consider adding or requesting that feature.)

At that point, all you need is to call your daemon in foreground from an ExecStart= directive. You don't need an ExecStop=, as long as your daemon properly terminates upon receiving a signal to kill it.

You don't need a pidfile anymore! Since systemd is launching the daemon process in foreground, it knows what the main PID of the daemon process is. This is huge, because pidfiles are often/usually implemented incorrectly (they're supposed to be created only once the daemon is ready to serve), so getting rid of that requirement is quite a big deal.

If you need to export specific environment variables before starting the main process, you can use systemd's Environment= or perhaps EnvironmentFile= to set those variables. (This will work fine, as long as the variables are set to fixed values rather than dynamically generated values.) If you need to execute steps before the daemon starts, you can use ExecStartPre=.

If you need more flexibility (for example, conditionally setting variables or running commands, or setting variables to dynamic values, etc.) then you should wrap the startup in a shell script (or Python, Perl, etc.) and call that script in ExecStart=. The script will set and export any variables needed, run any commands that need to be run, before executing the main daemon.

The important part when using a shell script to start a daemon is to use the exec command, in order to replace the shell with the daemon program. That means the shell will no longer be around, and the daemon will be running under the same PID that was in use by the shell, so systemd will still reliably know the main PID of the daemon. Of course, the daemon exec'd by the shell script should still run in foreground.

Using a Type=simple service allows you to configure User= in the systemd unit itself. Furthermore, you typically can apply more security measures through systemd configuration, that might have tripped a System V init script and prevented their usage. Also, using simple rather than forking makes this setup much more reliable, it's also more efficient on the system.

You can easily ship both a systemd service unit with Type=simple and a System V init script on your package. As long as they have the same name, systemd will prefer the native service unit (so the legacy code of systemd-sysv-generator will not trigger for the init script in that case.) That way you keep compatibility with non-Linux and other non-systemd setups, while at the same time you're able to make the most out of modern Linux systems with systemd.

  • 1
    Thank you for the comments and guidelines, filbranden. A lot here to consider moving forward. First step was to get past the errors I was seeing. This was all started because on newer distributions of both Redhat and SuSE, the systemd-sysv-generator seemed not to be quite working, so I decided to go ahead and bite the bullet and get our daemons running natively under systemd. Question about "type=forking," though. The process that runs as the daemon is called by a script, or, more accurately, several scripts are called to actualy get it running. Isn't this the definition of a forked process? – BobP Aug 12 at 20:04
  • The Type=forking services support "forking daemons", which typically launch a background process (fork, sometimes fork twice) and the main process doesn't stick around, it exits once the daemon is running in background... If you have scripts that fork a daemon, if the script sticks around and "waits" for the daemon to exit, that's more of a Type=simple service... But in that case systemd will interact with the script and not the daemon, so when you stop the service it will kill the script (which should trap signals and possibly forward them to the daemon process.) – filbranden Aug 12 at 20:52
  • It's fine if you have one or more scripts involved in starting the daemon process, but you want each of them to run the next step using exec, so that at the end of the chain you end up with the daemon program running under the initial PID, and no shell script is left around. That's the ideal scenario. Does that make sense? – filbranden Aug 12 at 20:54
  • 1
    Yes, makes sense. Thank you again. – BobP Aug 13 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.