Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am a computational scientist, and I run a lot of lengthy calculations on Linux. Specifically, I run molecular dynamics (MD) simulations using the GROMACS package. These simulations can take days or weeks, running on (for example) 8 to 24 cores. I have access to several nodes of a cluster, which means that at any given time, I am running approximately 4 or 5 jobs (each on a different node, and each on 8-24 cores).

The problem is that the simulation take a variable amount of time. I like to keep all nodes working on simulations around the clock, but to start a new simulation, I need to do log in with a terminal and do some manual work. But I always forget how much time is left in a simulation, so I always end up constantly checking them.

Is there any way that I can receive an e-mail when a Linux process finishes? Could there be a Linux program that does this? That way I would know when to log in with a terminal and prepare the next simulation.

I am using Ubuntu Linux. Thanks for your time.

share|improve this question

4 Answers 4

up vote 18 down vote accepted

Jobs submitted to the at daemon will send any output to you from stderr and stdout upon completion. It can also be configured to send mail even if the job has no output. It also has the benefit of running without a controlling terminal, so you don't have to worry about the effect that closing your terminal will have on the job.

example:

echo "/opt/product/bin/job.sh data123"|at -m NOW

When this job completes, the user who submitted the job will receive an email, and if there is any output at all you will receive it. You can change the email recipient by changing the LOGNAME environment variable.

at has a batch mode where you can queue jobs to run when the system is not busy. This is not a very good queueing system when multiple users are competing for resources, but nonetheless, if you wanted to run jobs with it:

echo "/opt/product/bin/job.sh dataA"|batch
echo "/opt/product/bin/job.sh dataB"|batch
echo "/opt/product/bin/job.sh dataC"|batch

By default the jobs will not start unless the system load is under 1.5, but that load figure can be adjusted (and with 24 cores I'd say you should). They can run in parallel if they don't bump the loadavg over the load limit (1.5 default again), or if they individually bump the loadavg over 1.5, they will run in serial.

You can view the job queue with atq, and delete jobs with atrm

Answer dependencies:

  1. Running atd daemon ( ps -ef|grep atd )
  2. You are allowed to submit jobs to atd (not denied by /etc/at.deny//etc/at.allow configurations)
  3. Functional sendmail MTA

Most systems have no problem with these requirements, but it's worthwhile to check.

share|improve this answer

You can also use trap command for sending mail on exit of process ,or at the interupt,hangup or at termination of process .Here is the code that you have to place at the top of your script.

    trap 'mail -s "Process done" mail@domain.tld' exit # It will mail on normal exit
    trap 'mail -s "Process interrupted" mail@domain.tld' INT HUP# it will mail on interrupt or hangup  of the process

Arun

share|improve this answer

I would recommend setting a python script to send an email, they are very easy to write and configure the correct mailing servers for whatever service you use. They look something like this:

#!/usr/bin/python

import smtplib

sender = 'from@fromdomain.com'
receivers = ['to@todomain.com']

message = """From: From Person <from@fromdomain.com>
To: To Person <to@todomain.com>
Subject: SMTP e-mail test

This is a test e-mail message.
"""

try:
   smtpObj = smtplib.SMTP('localhost')
   smtpObj.sendmail(sender, receivers, message)         
   print "Successfully sent email"
except SMTPException:
   print "Error: unable to send email"

Use this in conjunction with the pipe operator suggested in the other answers.

Another great solution I have found to this problem is using Pushover. Pushover - "Pushover makes it easy to send real-time notifications to your Android and iOS devices." I setup a simple script which ultised the easy API to send a message to my phone when my builds are done.

curl -s \
  -F "token=APP_TOKEN" \
  -F "user=USER_KEY" \
  -F "message=The build is done." \
  https://api.pushover.net/1/messages.json
share|improve this answer

Yeah, there is

command; echo "Process done" | mail -s "Process done" mail@domain.tld

Where -s "text" is the subject, the echo gives mail some Text to send to you.

share|improve this answer
3  
The mail command does require the local MTA to be configured properly. This is the easy and simple solution. –  jordanm Nov 10 '12 at 20:57
4  
you can also use mail to send you the output of the command: { command; echo Done; } | mail -s "subject" recipient@example.com –  glenn jackman Nov 11 '12 at 12:41
    
And wow, thanks for those upvotes :) –  Bonsi Scott Nov 13 '12 at 22:21
4  
@Andrew Do be aware that for this to work the terminal/shell you ran the original command in has to stay open. This might not be hard if you use something like screen or tmux, but it might be touchy to have a terminal open somewhere for days/weeks without it. The solution with at for job management would also take care of that problem. –  Caleb Nov 14 '12 at 13:13

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.