5

I have the following service configured:

[Unit]
Description=SCollector
After=NetworkManager.service

[Service]  
Type=simple
ExecStart=/bin/sh -c "/opt/scollector/scollector /opt/scollector/collectors || (echo '' | /usr/bin/mail -s 'scollector died' jharvey@stackoverflow.com && exit -1)"
Restart=on-failure

[Install]
WantedBy=multi-user.target    

For some reason, that mail command never sends any mail when the scollector process exits with non-0. This works AOK when run on the command line, /bin/sh call and all. I've captured STDOUT and STDERR of mail, and it is throwing no errors. There is nothing in maillog.

What gives? Why won't it send mail?

  • your ExecStart runs exit -1 when scollector exits with 0. Is it ok? – Evgeny Vereshchagin Sep 22 '15 at 21:31
  • Yeah, you're correct. I ended up not using any of the above, so I never ran into that exit. I'll fix it in the question regardless so it doesn't red-herring anyone. – alienth Sep 22 '15 at 21:36
  • try ExecStart=/bin/sh -c "false || (echo '' | /usr/bin/mail -s 'scollector died' jharvey@stackoverflow.com && exit -1)". Does it work? Which version of systemd are you using? – Evgeny Vereshchagin Sep 22 '15 at 22:06
  • Nope. As indicated in my answer, the issue is caused by systemd nuking the double-forked sendmail call from mail. I'm on cent7, which uses systemd 208. – alienth Sep 22 '15 at 22:08
  • To reiterate: simply having ExecStart=/bin/sh -c "/usr/bin/mail -s test me@gmail.com </dev/null" will never send mail, unless you change the KillMode as indicated in my answer. – alienth Sep 22 '15 at 22:18
7

/usr/bin/mail performs a double fork to daemonize sendmail for sending the email. This sendmail proc gets reowned to init, so normally it wouldn't be affected by anything that happens with the original parent - except in the systemd case that reowned grandchild is still in the same cgroup as the original service. When systemd tears things down, it kills all processes within the cgroup, including the reowned sendmail process.

The mail command itself ran fine, but sendmail was getting killed by systemd before it had a chance to do its thing.

You can get around this by setting KillMode in the Unit section to process (the default is control-group). That will cause systemd to only kill the process which it directly fired.

Interestingly the way I stumbled upon this was through the use of strace. A normal strace revealed nothing, but the mail suddenly started working when using strace -f. strace -f was causing the main process to stick around until all of the children and orphaned grandchildren had wrapped up.

  • 4
    -S sendwait as in cat /tmp/mailtext | /usr/bin/mailx -S sendwait -r from@example.com -s "My Working Mail" to@example.com works for me – mnagel Feb 18 '16 at 14:08
2

The questioner has identified the problem; but xyr solution is a bodge, and xyr description of the mechanics is incorrect.

The mail command does not perform a double fork. It forks just once, and the sendmail shim process is its immediate child that is not reparented to anything. It simply chooses whether to waitpid() for that child or not, before it exits.

The same is true of the sendmail shim itself. It does not double fork. On some MTSes it doesn't even fork at all. On others it forks just the once and chooses whether to wait or not dependent from some configurable "delivery mode" option.

The correct way to get around the problem is twofold:

  1. Set mailx's documented and standardized sendwait option. That specifically addresses the problems of asynchronous enqueueing, by making mailx wait for the sendmail shim child process to finish. (Sadly, even though this option has been around since at least 1986 and is documented for mailx in the SVID, bsd-mailx does not have it. heirloom-mailx has it, though.)
  2. Set whatever MTS is in use to use a synchronous queueing/delivery mode if it isn't using one already.
    • If using netqmail, do nothing. netqmail's sendmail shim is always queued and synchronous, directly chain loading through qmail-inject to qmail-queue without forking at all.
    • If using Postfix, do nothing. Postfix's sendmail shim is always queued and synchronous, forking once and waiting for postdrop to finish before exiting itself.
    • exim has the -odf command line option.

Further reading

  • Note: to set the mailx internal variable, add 'set sendwait' to /etc/mail.rc (at least that's where it is on CentOS 7) – siliconrockstar Jun 3 '16 at 1:28
  • I was able to add -S sendwait when calling mailx since I didn't want to change the system configuration, but needed to make my script work when called from a systemd service unit. – JFlo May 26 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.