46

I am a computational scientist, and I run a lot of lengthy calculations on Linux. Specifically, I run molecular dynamics (MD) simulations using the GROMACS package. These simulations can take days or weeks, running on (for example) 8 to 24 cores. I have access to several nodes of a cluster, which means that at any given time, I am running approximately 4 or 5 jobs (each on a different node, and each on 8-24 cores).

The problem is that the simulation take a variable amount of time. I like to keep all nodes working on simulations around the clock, but to start a new simulation, I need to do log in with a terminal and do some manual work. But I always forget how much time is left in a simulation, so I always end up constantly checking them.

Is there any way that I can receive an e-mail when a Linux process finishes? Could there be a Linux program that does this? That way I would know when to log in with a terminal and prepare the next simulation.

I am using Ubuntu Linux. Thanks for your time.

  • 3
    That was a rather long and irrelevant description for a simple question which was already covered in the title. Do you always do this? – Acumenus Nov 9 '14 at 5:58
  • 5
    @A-B-B As a scientist I consider there to be useful information in the "long and irrelevant" description. For example, we know that manual terminal work is required and hence there is a reason for asking the question. In addition we have some idea about what the OP is doing so we aren't thinking totally in the dark and may therefore be able to provide additional info which is not requested in the question but may be of use or interest. – user3728501 Jul 9 '16 at 0:39
30

Jobs submitted to the at daemon will send any output to you from stderr and stdout upon completion. It can also be configured to send mail even if the job has no output. It also has the benefit of running without a controlling terminal, so you don't have to worry about the effect that closing your terminal will have on the job.

example:

echo "/opt/product/bin/job.sh data123"|at -m NOW

When this job completes, the user who submitted the job will receive an email, and if there is any output at all you will receive it. You can change the email recipient by changing the LOGNAME environment variable.

at has a batch mode where you can queue jobs to run when the system is not busy. This is not a very good queueing system when multiple users are competing for resources, but nonetheless, if you wanted to run jobs with it:

echo "/opt/product/bin/job.sh dataA"|batch
echo "/opt/product/bin/job.sh dataB"|batch
echo "/opt/product/bin/job.sh dataC"|batch

By default the jobs will not start unless the system load is under 1.5, but that load figure can be adjusted (and with 24 cores I'd say you should). They can run in parallel if they don't bump the loadavg over the load limit (1.5 default again), or if they individually bump the loadavg over 1.5, they will run in serial.

You can view the job queue with atq, and delete jobs with atrm

Answer dependencies:

  1. Running atd daemon ( ps -ef|grep atd )
  2. You are allowed to submit jobs to atd (not denied by /etc/at.deny//etc/at.allow configurations)
  3. Functional sendmail MTA

Most systems have no problem with these requirements, but it's worthwhile to check.

49

Yeah, there is

command; echo "Process done" | mail -s "Process done" mail@domain.tld

Where -s "text" is the subject, the echo gives mail some Text to send to you.

  • 4
    The mail command does require the local MTA to be configured properly. This is the easy and simple solution. – jordanm Nov 10 '12 at 20:57
  • 6
    you can also use mail to send you the output of the command: { command; echo Done; } | mail -s "subject" recipient@example.com – glenn jackman Nov 11 '12 at 12:41
  • 4
    @Andrew Do be aware that for this to work the terminal/shell you ran the original command in has to stay open. This might not be hard if you use something like screen or tmux, but it might be touchy to have a terminal open somewhere for days/weeks without it. The solution with at for job management would also take care of that problem. – Caleb Nov 14 '12 at 13:13
  • @Caleb Cant you add a & at the end of the command to get it to run in the back ground. sleep 30 && echo "Process done" | mail -s "Process done" mail@domain.tld & – Theo Kouzelis Aug 26 '15 at 9:02
  • @Theo no you can't. Even in the background the process is still a child of your shell and if the shell dies the child process will receive a kill signal. You could use nohup to unlink them, but you also lose the ability to continue controlling it from the shell. This is partially what tmux is for. It divorces the running processes from your actively connected and viewed terminal session while still letting you interact with them normally. – Caleb Aug 26 '15 at 9:20
7

I would recommend setting a python script to send an email, they are very easy to write and configure the correct mailing servers for whatever service you use. They look something like this:

#!/usr/bin/python

import smtplib

sender = 'from@fromdomain.com'
receivers = ['to@todomain.com']

message = """From: From Person <from@fromdomain.com>
To: To Person <to@todomain.com>
Subject: SMTP e-mail test

This is a test e-mail message.
"""

try:
   smtpObj = smtplib.SMTP('localhost')
   smtpObj.sendmail(sender, receivers, message)         
   print "Successfully sent email"
except SMTPException:
   print "Error: unable to send email"

Use this in conjunction with the pipe operator suggested in the other answers.

Another great solution I have found to this problem is using Pushover. Pushover - "Pushover makes it easy to send real-time notifications to your Android and iOS devices." I setup a simple script which ultised the easy API to send a message to my phone when my builds are done.

curl -s \
  -F "token=APP_TOKEN" \
  -F "user=USER_KEY" \
  -F "message=The build is done." \
  https://api.pushover.net/1/messages.json
6

You can also use trap command for sending mail on exit of process ,or at the interupt,hangup or at termination of process .Here is the code that you have to place at the top of your script.

    trap 'mail -s "Process done" mail@domain.tld' exit # It will mail on normal exit
    trap 'mail -s "Process interrupted" mail@domain.tld' INT HUP# it will mail on interrupt or hangup  of the process

Arun

5

I wrote process_watcher.py

process_watcher --pid 1234 --to me@gmail.com

Currently, email body looks like:

PID 18851: /usr/lib/libreoffice/program/soffice.bin --writer --splash-pipe=5
Started: Thu, Mar 10 18:33:37 Ended: Thu, Mar 10 18:34:26 (duration 0:00:49)
Memory (current/peak) - Resident: 155,280 / 155,304 kB Virtual: 1,166,968 / 1,188,216 kB

[+] indicates the argument may be specified multiple times, for example:
 process-watcher -p 1234 -p 4258 -c myapp -c "exec\d+" --to person1@domain.com --to person2@someplace.com

optional arguments:
  -h, --help            show this help message and exit
  -p PID, --pid PID     process ID(s) to watch [+]
  -c COMMAND_PATTERN, --command COMMAND_PATTERN
                        watch all processes matching the command name. (RegEx pattern) [+]
  -w, --watch-new       watch for new processes that match --command. (run forever)
  --to EMAIL_ADDRESS    email address to send to [+]
  -n, --notify          send DBUS Desktop notification
  -i SECONDS, --interval SECONDS
                        how often to check on processes. (default: 15.0 seconds)
  -q, --quiet           don't print anything to stdout

Ceate a GitHub issue if you need any improvements to it.

  • 2
    Winner for me because can attach by PID to an already-running process. – joseph_morris Sep 9 '16 at 23:49
  • 1
    @joseph_morris I like this script too — but for completeness, you can easily attach to a running (suspended, via Ctrl-Z) process without this helper by issuing the command fg ‹PID›; mail -s Completed to@example.com <<< 'Message here'. – Konrad Rudolph Oct 10 '16 at 11:14
1

I have a home-grown script that works for me. Usage:

mail-after <your command>

It will send an e-mail with meaningful information in the subject, and with the process's output in the body.

Installation instructions are on the main page of the GitHub repo -- currently for all scripts a symlink is created in ~/bin.

1

Assuming your program is already running, use bash+pgrep to watch it.

# Check that "pgrep $PROGRAM" returns the rid PID.
PROGRAM=foo
RECIPIENTS=you@yours.com
echo Watching $PROGRAM; while pgrep $PROGRAM &>/dev/null ; do sleep 1; echo -ne "."; done ; echo "$PROGRAM process done" | /bin/mail -s "$PROGRAM process done" $RECIPIENTS;
  • I used something similar: while [[ $(pgrep myprog) ]]; do sleep 60; done; echo "myprog is done" | mail -s "process done" mail@example.com – marlar Nov 11 '18 at 10:59
0

Ages ago, I created this alias (in csh, I now use it in tcsh):

alias shunt "( \!* | Mail -s 'Output of: \!*' $user) >& /dev/null &"

It requires that Mail be installed and working on your system, but has the advantage that your command runs exactly as it would from the command line (including aliases, tilde expansions, etc)-- when you run a job using at or cron it runs under a default environment that may be sufficiently different to cause your script to fail or provide unusual output (I've been bitten by that from cron many times)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.