0

We have a forever running shell script handled by systemd which runs fine until some days ago, our server storage was 100% used because of some gigantic log files. I had to truncate those files in order to free some spaces.

Today, I got a report that the script is stopped running. But when I checked the status, it says :

● ImportantService.service - Important daemon
   Loaded: loaded (/etc/systemd/system/ImportantService.service; enabled; vendor preset: disabled)
   Active: active (exited) since Wed 2020-04-29 16:46:48 WIB; 5 days ago
  Process: 48877 ExecStop=/usr/local/bin/importantScript stop --instance XYZ (code=exited, status=0/SUCCESS)
  Process: 48889 ExecStart=/usr/local/bin/importantScript start --instance XYZ (code=exited, status=0/SUCCESS)
 Main PID: 48889 (code=exited, status=0/SUCCESS)
    Tasks: 0
   Memory: 48.0K
   CGroup: /system.slice/ImportantService.service

I noticed that it has 0 task, hence I restarted it manually. Now it runs normally. I suspected that the problem I mentioned earlier caused this.

The question is, how to make systemd respawn the processes if this kind of problem occur in the future?


Here's the .service file:

[Unit]
Description= Important daemon

[Service]
Type=oneshot
ExecStart=/usr/local/bin/importantScript start --instance XYZ
RemainAfterExit=true
ExecStop=/usr/local/bin/importantScript stop --instance XYZ

[Install]
WantedBy=multi-user.target

Update : Some explanation regarding my script

The main functionality of ImportantScript is to enter a forever loop to do something if some event occurs in a given (arg) directory. And I should be able to launch multiple instances for different directories. Before I could start the script, I have to add a set of specifications of an instance, like :

ImportantScript add --name XYZ --dir /path/to/dir ..etc..

Now, I'm able to start instance of XYZ.

Every time I start an instance, it will store the PID to a file contains all predefined instances (using previous add command). PID that isn't listed means that corresponding instance is in idle state.

To stop an instance, I just have to call ImportantScript stop --instance name. It will terminate the process and delete the corresponding entry in the file, clean up its mess.

I hope this is not a systemd House of Horror entry.

  • rc scripts are only responsible for starting services and not maintaining it running. You should look forward onto something like supervisor daemon with proper config. – DevilaN May 5 at 6:12
  • 1
    It's a good bet, given the start and stop verbs, that "importantScript" is far from important and is more likely a Poor Man's Dæmon Supervisor and Bad Logger that is in fact getting in the way. For best results, you should tell people how your real dæmon is actually run, in the depths of that script. Only with that information can one construct a service unit that isn't a systemd House of Horror entry. – JdeBP May 5 at 9:49
  • @JdeBP I was looking up on google about systemd House of Horror and I found your articles. I'm not so familiar with systemd yet, I hope you can give me insights. Please see my edit. – annahri May 5 at 19:05
  • @JdeBP My initial thought of using systemd is simply to automatically starts the script in case of system reboots. After reading this, I think it's true that the "ImportantScript" is a Poor Man's Daemon. Feels bad haha – annahri May 5 at 19:24
2

If you want systemd to restart your service, you want the Restart= service option. Your service likely exited of its own accord, as shown by Active: active (exited) in the systemctl output, and the fact it had a 0 exit code.

A couple of the most common possible configurationss, as documented in man systemd.service, are:

  • Restart=always: Unless the service was explicitly stopped (eg. systemctl stop, or directly calling StopUnit over DBus), restart it.
  • Restart=on-failure: If the service terminated with a >0 exit code, restart it.
  • Restart=no: The default. Don't restart the service.

These are probably the most commonly used settings -- you can find more options in the above-linked table.

| improve this answer | |
  • My configuration always say that it is Active: active (exited) in most occasions. Might try this, anyway. – annahri May 5 at 19:27
  • @annahri Then that's because /usr/local/bin/importantScript start --instance XYZ forks away, but you are using Type=oneshot, which only exists for the main process' lifetime. You want to either stop forking, or use Type=forking. – Chris Down May 5 at 20:46
  • That's intended, because the main script executes the forever loop (something like while watch /this/dir; do something; done & ) and then it leaves a background process. I tried using Type=simple, and it ended up spawning the background processes endlessly. I don't know how to say it properly, but I hope you get the idea. – annahri May 5 at 20:51
  • 2
    Well, you can't have it both ways. If you want systemd to manage your processes, you need to give it the right process to manage... – Chris Down May 5 at 21:06
  • Ok, I got the point. So I have to rewrite my script. Thanks for your replies! – annahri May 5 at 21:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.