6

I have a .NET Core service running on a Debian 9, let's call it MyService. At some point this service is running a bash script update.sh using Process.Start() with ShellExecute=true.

This script basically runs apt-get update; apt-get upgrade.

During package upgrade, MyService process is terminated: update script is terminated as well and apt-get upgrade is killed as well, leaving inconsistent packages that must be fixed manually.

What I want is that update.sh is NOT terminated when MyService is terminated.

I tried splitting update.sh in 2 parts, the first running the second in different ways; I tried starting update2.sh with setsid and nohup but I always get same result. I tried to execute update2.sh script in a new bash shell with /bin/bash /c "update2.sh", same result.

How do I run a script started from a binary and completely detach from binary process, so I can kill binary while script keeps running?

Here's my environment. MyService is a binary running as a service. update.sh is started by MyService.

.NET Core code to start shell script, inside MyService binary:

var process = new Process();
process.EnableRaisingEvents = true; // to avoid [defunct] sh processes
process.StartInfo.FileName = "/opt/myservice/update.sh";
process.StartInfo.Arguments = "";
process.StartInfo.UseShellExecute = true;
process.StartInfo.CreateNoWindow = true;
process.Start();
process.WaitForExit(10000);
if (process.HasExited)
{
  Console.WriteLine("Exit code: " + process.ExitCode);
}
else
{
  Console.WriteLine("Child process still running after 10 seconds");
}

update.sh:

nohup /opt/myservice/update2.sh > /opt/myservice/update.log &
systemctl stop MyService

update2.sh:

apt-get update >> /opt/myservice/update.log
apt-get -y install --only-upgrade myservice-1.0 >> /opt/myservice/update.log

update2.sh is never executed because it's terminated when MyService is terminated by update.sh.

update.sh returns code 143, it seems it has been killed.

2018-08-16 14:46:14.5215|Running update script: /opt/myservice/update.sh
2018-08-16 14:46:14.5883|Update script /opt/myservice/update.sh returned: 143

UPDATE

I tried following approaches, thanks for suggestions:

  • setsid
  • disown
  • nohup
  • screen
  • tmux
  • unshare

Every approach has same result, termination of all spawned processes. I suspect this is a .NET Core "feature".

UPDATE 2

I discovered that systemctl stop MyService by default explicitly kills all spawned processes by a service.

https://stackoverflow.com/questions/40898077/systemd-systemctl-stop-aggressively-kills-subprocesses

If I add KillMode=process to service descriptor, update script is not terminated when service is terminated.

There is NO WAY to escape from PID space for a service started by systemctl. Every technique used, included the one in accepted answer, does not generate a separate process. Every spawned process is always killed by systemctl stop MyService unless KillMode=process is specified.

I ended up creating a separate service MyServiceUpdater: this service runs the plain updater script without any forking. Since PID space is different, everything works as expected. That was a long ride.

MyServiceUpdater example:

[Unit]
Description=Your Service Updater
After=network.target

[Service]
ExecStart=/path/to/update/script/updatescript.sh
ExecStopPost=
TimeoutStopSec=30
StandardOutput=null
WorkingDirectory=/path/to/service/directory/
KillMode=process

[Install]
WantedBy=multi-user.target
5
  • unix.stackexchange.com/questions/452262/… is one way to "completely detach from" a process
    – thrig
    Aug 16, 2018 at 13:56
  • Thanks for suggestion. I tried perl script, same result. Could be ".NET Core runtime uses a PID namespace that the process cannot escape no matter how it forks itself" as described in page you suggested.
    – Ghigo
    Aug 16, 2018 at 14:51
  • It seems that .NET Core application kills all spawned processes on termination, regardless any strategy I take.
    – Ghigo
    Aug 17, 2018 at 12:37
  • Fighting the same problem with my upgrader script. Possible to post the unit file for MyServiceUpdater ?
    – Danny
    Oct 28, 2020 at 1:41
  • @Danny I added the updater service file. Hope it helps!
    – Ghigo
    Oct 29, 2020 at 8:31

3 Answers 3

2

Use crontab (or at) (not mono/.net) to schedule tasks.

Normal options;

  • nohup my.sh &
  • screen -dm -S my my.sh
  • tmux new -d -s my my.sh
  • service my start / systemctl start my
  • Ctrl+Z, bg, disown
6
  • Crontab: it's not designed to run one task once. nohup: tried and it's terminated. I would like to use base installation without additional tools/daemons. I know I could use at.
    – Ghigo
    Aug 16, 2018 at 12:14
  • at has another issue, it doesn't run as root. If I schedule a task as root, it's not executed as root. So it doesn't work in my scenario.
    – Ghigo
    Aug 16, 2018 at 12:20
  • update.sh MUST be executed by .NET Core service because it's application feature autoupdate, triggered from remote command. Inside update.sh I can do whatever I want to perform update.
    – Ghigo
    Aug 16, 2018 at 12:28
  • Your tmux command line is wrong. Tried 1,2,3, no luck, they all behave the same. systemctl is unrelated because update script is not a service, can't run like this. Last one is interactive and unrelated in my context.
    – Ghigo
    Aug 17, 2018 at 12:34
  • @Ghigo i fixed the tmux example, you can make your script a service by copying it to /etc/init.d . All interactive things can be scripted. Aug 17, 2018 at 12:43
1

On a Centos 7 test system via

$ sudo rpm -Uvh https://packages.microsoft.com/config/rhel/7/packages-microsoft-prod.rpm
$ sudo yum install dotnet-sdk-2.1

which results in dotnet-sdk-2.1-2.1.400-1.x86_64 being installed then with the test code

using System;
using System.Diagnostics;
using System.ComponentModel;
namespace myApp {
    class Program {
        static void Main(string[] args) {
            var process = new Process();
            process.EnableRaisingEvents = true; // to avoid [defunct] sh processes
            process.StartInfo.FileName = "/var/tmp/foo";
            process.StartInfo.Arguments = "";
            process.StartInfo.UseShellExecute = true;
            process.StartInfo.CreateNoWindow = true;
            process.Start();
            process.WaitForExit(10000);
            if (process.HasExited) {
                Console.WriteLine("Exit code: " + process.ExitCode);
            } else {
                Console.WriteLine("Child process still running after 10 seconds");
            }
        }
    }
}

and a shell script as /var/tmp/foo a strace stalls out and shows that /var/tmp/foo is run through xdg-open which on my system does...I'm not sure what, it seems a needless complication.

$ strace -o foo -f dotnet run
Child process still running after 10 seconds
^C
$ grep /var/tmp/foo foo
25907 execve("/usr/bin/xdg-open", ["/usr/bin/xdg-open", "/var/tmp/foo"], [/* 37 vars */] <unfinished ...>
...

a simpler solution is to simply exec a program that in turn can be a shell script that does what you want, which for .NET requires not using the shell:

            process.StartInfo.UseShellExecute = false;

with this set the strace shows that /var/tmp/foo is being run via a (much simpler) execve(2) call:

26268 stat("/var/tmp/foo", {st_mode=S_IFREG|0755, st_size=37, ...}) = 0
26268 access("/var/tmp/foo", X_OK)      = 0
26275 execve("/var/tmp/foo", ["/var/tmp/foo"], [/* 37 vars */] <unfinished ...>

and that .NET refuses to exit:

$ strace -o foo -f dotnet run
Child process still running after 10 seconds
^C^C^C^C^C^C^C^C

because foo replaces itself with something that ignores most signals (notably not USR2, or there is always KILL (but avoid using that!)):

$ cat /var/tmp/foo
#!/bin/sh
exec /var/tmp/stayin-alive
$ cat /var/tmp/stayin-alive
#!/usr/bin/perl
use Sys::Syslog;
for my $s (qw(HUP INT QUIT PIPE ALRM TERM CHLD USR1)) {
   $SIG{$s} = \&shandle;
}
openlog( 'stayin-alive', 'ndelay,pid', LOG_USER );
while (1) {
    syslog LOG_NOTICE, "oh oh oh oh oh stayin alive";
    sleep 7;
}
sub shandle {
    syslog LOG_NOTICE, "nice try - @_";
}

daemonize

With a process that disassociates itself from the parent and a shell script that runs a few commands (hopefully equivalent to your intended apt-get update; apt-get upgrade)

$ cat /var/tmp/a-few-things
#!/bin/sh
sleep 17 ; echo a >/var/tmp/output ; echo b >/var/tmp/output

we can modify the .NET program to run /var/tmp/solitary /var/tmp/a-few-things

            process.StartInfo.FileName = "/var/tmp/solitary";
            process.StartInfo.Arguments = "/var/tmp/a-few-things";
            process.StartInfo.UseShellExecute = false;

which when run causes the .NET program to exit fairly quickly

$ dotnet run
Exit code: 0

and, eventually, the /var/tmp/output file does contain two lines written by a process that was not killed when the .NET program when away.

You probably should save the output from the APT commands somewhere, and may also need something so that two (or more!) updates are not trying to be run at the same time, etc. This version does not stop for questions and ignores any TERM signals (INT may also need to be ignored).

#!/bin/sh
trap '' TERM
set -e
apt-get --yes update
apt-get --yes upgrade
7
  • Thanks for your effort. May I ask you a test? Try to kill .NET Core foo binary inside a-few-things script. Check if a-few-things script continue execution. Consider that in my scenario, foo is a service running in background, killed by apt-get upgrade during package upgrade.
    – Ghigo
    Aug 20, 2018 at 15:19
  • Instead of 17 seconds, just wait 5 seconds then kill foo. Then do something else that produces output. You should also add a Console.ReadLine() after process execution in C# code. What happens?
    – Ghigo
    Aug 20, 2018 at 15:24
  • kill foo how? various signals, if necessary, could be ignored or handled in various ways via calls detailed in signal(2)
    – thrig
    Aug 20, 2018 at 15:47
  • I don't know exactly. In your opinion, how apt-get upgrade kills a previously running daemon when updating to the new version? Inside .deb package I have prerm script running systemctl stop MyService. Does this perform a KILL?
    – Ghigo
    Aug 20, 2018 at 16:08
  • does it? use sysdig and see what signals are sent to what, then use that information to block or change the signal handling as necessary
    – thrig
    Aug 20, 2018 at 17:49
0

Been battling the exact same thing myself.

One solution I've found is to use systemd-run, which essentially creates a transient one-time service to execute an arbitary command. However for it to work it seems to require that you specify the -r (remain-after-exit) argument. See https://www.freedesktop.org/software/systemd/man/systemd-run.html for more documentation.

For example:

var process = new Process()
{
    StartInfo = new ProcessStartInfo
    {
        FileName = "systemd-run",
        Arguments = "-r sleep 90",
        RedirectStandardOutput = false,
        RedirectStandardError = false,
        UseShellExecute = false,
        CreateNoWindow = true,
    },
};
process.EnableRaisingEvents = true;
process.Start();

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .