2

I have a small C-program which I'm running already running as a daemon with systemd. Currently restart is set to always. I'd like to adjust the unit so that it restarts periodically, say each day, in case my code has some memory leaks.

I know systemd offers various restart options, such as WatchdogSec or RuntimeMaxSec, but these appear to result in a SIGABRT signal, which I believe I can't catch in in my code, so can't shut down gracefully.

Is there anyway for a systemd unit to periodically shutdown and restart its service? Or, is it possible to listen for the SIGABRT signal in my code and handle it gracefully?

  • 1
    Since this is your code, why not have it have an option to shut down gracefully after a certain amount of time? You could either then have systemd restart it ... or it could just exec itself as the final action.... – mattdm Jul 23 '17 at 15:57
  • 1
    @Dom why not just use a cronjob to restart it gracefully? – Charles Addis Jul 23 '17 at 22:38
  • 1
    0 2 * * * /path/to/script-that-gracefully-restarts-program or 0 2 * * * /bin/bash command-that-restarts-program will execute the script or command at 2am each day. – Charles Addis Jul 23 '17 at 22:44
2

No, you don't need to modify anything in your code. just as simple as, You can use crontab for restarting your service periodically.

  1. check your process is alive yet.
  2. if it was dead, restart it.
  3. if it took too many memory, kill it and restart it.

    PID_YOURS = `ps -a | grep "YOUR_PROCESS" | awk -F" " '{print $1}'`
    MEM_USES = `ps -eo pid,rss | grep "$PID_YOURS" | awk -F" " '{print $2}'`
    

However, A best way is improving your code for reducing memory leaks. I hope you can solve it as simple as possible.

P.S. you can't catch a critical exceptions internally. especially, Errors due to memory leaks can't be handled internally as they typically involve heap corruptions. Therefore, it is inevitable that the situation is forcibly terminated by kernel. also, there is a way to approach it in a different way;

while ( condition for keeping a service running ) {
    system("your_program");
    // blocked in above code. if the flow reached this point,
    // it means that your program is dead.
}

OR, You can register a SYSABRT handler for catching it. but, you can't recover the flow for working well. you should only perform emergency operations to safeguard your data in SYSABRT handler. (but SYSABRT handler is not guaranteed that will be executed in that situation)

_set_abort_behavior(0, _WRITE_ABORT_MSG); // suppress warnings.
signal(SIGABRT, abrtHandler); // register handler

...

void abrtHandler(int signo) {
    if(signo == SIGABORT) {
        // do something for storing your datas.
        signal(SIGABRT, SIG_DFL); // restore an original handler.
    }
}

don't spend much time in SYSABRT handler. not, kernel kill it immediately. Also, doing memory allocation operations in SYSABRT handler can cause critical exceptions. so you need to construct emergency memory for allocating memory(temporarily).

static uint8_t g_emergency[64 * 1024];
static uint8_t g_situation = 0;
static uint8_t* g_allocs = g_emergency;

void* emergency_alloc(size_t sz) {
    ...
    if(g_situation) {
        g_allocs += sz;
        return g_allocs - sz;
    }
    ...
}

Finally, in the first place, you can replace your service restart with your own implementation. You can send just a message to your program via a Unix socket or files, or other methods... It is not a complicated problem.

Real Solutions for your daemon.

Improve your daemon for interaction with Systemd. Systemd supports Start, Restart, Stop mechanisms and you specify them as your wants.

i.e:

ExecStart=/your/daemon/path/and/binary start
ExecStop=/your/daemon/path/and/binary stop
Restart=always
WatchdogSec=10000
MemoryMax=2048M

even more, it supports pre- actions and post- actions and you can specify them too. you can handle it with programmatical methods. if restart mechanism was not specified, it executes ExecStop before executing ExecStart.

Also, In recent version of systemd, it supports to make a limit for memory. MemoryLimit was deprecated, but MemoryMax is still available. you can setup a limit with restarting options. you can read this document from freedesktop.org.

if you implement both sides of systemd and your daemon's interactions, you can handle it perfectly. you can leave your pid to file and grab it when you need to handle stop. even more, you can reuse it for checking your daemon is already running. if your system shutdown in unexpected state, you must check its pid is correct or existing.

also, if you implement this method, you can reload configurations for your daemon with it. just open a unix socket like mysqld, put actions as data. it allows you that your daemon can stop itself or reload itself with an operation for storing an important datas that must be stored.

crontab or system() method can go your system ahead into resource lack. even SYSABRT signal maybe not executed correctly(because it represent as critical situation). and polling your daemon will drive you into same situation with crontab or system() method.

all methods for killing it forcily can make your datas to be leaving it unsaved. you must handle it with command line arguments and interact systemd's management cycle. systemctl kill and kill for killing your daemon, they cause that also. so, i don't recommend them.

finally, you need to implement interaction routines with command line arguments and include SYSABRT handler for handling it gracefully.

Further more, you can monitor your daemon with forking a flow in your codes and open some IPC channel such as shared memory or Unix Socket.

pid_t pid = fork();
if(pid == 0) {
    // children.
} else {
    pid = fork();
    if(pid == 0) {
        // parent. you can monitor a child with IPC channel and,
        // you can check your child process is alive yet.
        // if your child doesn't response with IPC, you can kill it in here.
    } else exit(0);
}

If you want to control only with systemd, you can read this document.

See "Advanced unit files" part in that document, and you can read this document too. it case is similar with your scenario like if a daemon doesn't response anymore that is not dead.

  • Surely by using cron to check that a process is alive and restarting it if its not, you creating a sort of daemon. Isn't this one of the reasons systemd was created in the first place, to stop people needing to write their own daemons and all the problems that came with them? – Dom Jul 23 '17 at 21:04
  • 1
    Not really, as tools to do that were invented in the 1990s, decades before systemd. That said, a lot of this answer is entirely wrongheaded. The scheduled task does not need to have an aliveness check, grepping the process table is a bad idea, forking off the actual daemon as a child breaks service management paradigms, and you do not need to re-invent SIGTERM badly. This answer is wrong on quite a lot of points I and recommend not following any of its advice if one is writing daemons to operate under service management. – JdeBP Jul 23 '17 at 21:16
  • @Dom oh, you must read to systemd service file's restart method and you must replace its implementation for notifying a event to your daemon. it was meaning in last finally summary. – jay k Jul 23 '17 at 21:40
  • @JdeBP thanks a lot, big bro. i notice your advice. i will modify after few hours... – jay k Jul 23 '17 at 21:42
  • 1
    I suggest an answer that talks about systemctl kill and, perhaps more importantly, that talks about using service management mechanisms to impose process resource limits on the daemon. – JdeBP Jul 23 '17 at 22:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.