0

I need to process at least three thousand incoming emails an hour that are received in a postfix mailbox. Processing involves extracting email attachments and sending the content of an email attachment to an external email address. For each email attachment I send one email. I think I could achieve this by writing a script to cronjob to sleep every second but I am not sure that this is good practice(doing this will easily give me approximate 3600 mails processed an hour). Please advise on the best way to approach this. I already have a script that extract emails and sends email, the only problem with the scripts is that it can only handle one email at a time. If I use per minute cronjob I can only process 60 mails per hour while the target is really at least 3000 mails per hour.

1
  • Better to use postfix pipe to pipe to your script, example perlmonks.org/?node_id=1030830 then you can insert the mail into db or send it to the other end of the earth in your script
    – number5
    Jul 24, 2013 at 7:08

1 Answer 1

1

Procmail trivially solves this, provided your server has the CPU and bandwidth to sustain the flow. If you already have a script which takes care of this, simply pipe each incoming message to your script. Put the following in your $HOME/.procmailrc:

:0
| /path/to/your/script

The script receives the message as its standard input, and is responsible for delivering or otherwise handling the message from here on. (In other words, Procmail will not deliver this message into your inbox at all. See below for how to modify this behavior.)

(Procmail is not strictly necessary for this, but it adds a nice safety net so that your script doesn't need to cope with all the possible error conditions. You could simply tuck the pipeline in your .forward if your script is robust enough. This is basically what @number5's comment tells you, too, except it does this in Postfix' config file, instead of using the .forward facility.)

If two instances of the script cannot run concurrently (for example, because it needs exclusive access to a back-end database), add a lock file:

:0:yourscript.lock
| /path/to/your/script

This will cause Procmail to look for the file yourscript.lock and, if it exists, wait until it goes away; then create the file, run the recipe, and remove the lock file.

Using a lockfile forces deliveries to be serialized. This will reduce performance, though. If at all possible, it would be better to make the script robust under parallel execution.

On the other hand, if your script incurs a heavy load on the server, you might not want to run multiple concurrent instances; in this scenario, performance might actually improve if you force serialized delivery.

If you want a copy in your inbox as well, clone a copy when delivering to your script:

:0c # or :0c:yourscript.lock
| /path/to/your/script

You can also add a condition, so that e.g. only messages with a particular subject line are piped to your script. The conditions are specified with an asterisk as the first character, followed by a regular expression which needs to match the message's headers.

:0
* ^Subject: xyzzy$
| /path/to/your/script

If the above is not suitable, the following recipe will extract all attachments to a directory and send an email for each incoming message. Looping over the attachments is probably best done from an external script like above, but this should at least give you a whiff of what it would look like to do something a bit more involved in Procmail itself.

METAMAIL_TMPDIR=`mktemp -d /tmp/extracted.XXXXXXXXX`

# Crude attachment extraction ... how are you currently doing this?
:0c
| metamail -w -d

COUNT=`find "$METAMAIL_TMPDIR" -printf "%i\n" | wc -l`

:0
| ( echo Subject: $COUNT attachments extracted into $METAMAIL_TMPDIR; echo; echo ) \
  | sendmail -oi [email protected]
4
  • where exactly would these two lines go? :0 | /path/to/your/script my current path for new mail looks like this /home/user/Maildir/new I am using Postfix Mail. Please let me know if I should be concerned about mails coming in milliseconds apart from each other. Is there any way to let each mail get completely processed and only upon completion the next mail gets processed? An answer to any of the questions above will be very helpful.@tripleee.
    – user40952
    Jul 24, 2013 at 16:57
  • Updated the answer. Hope this helps.
    – tripleee
    Jul 24, 2013 at 17:30
  • thanks very helpful. My hope is that no mails are lost as a result of the wait. I will test the scripts accordingly though. Thanks a lot @tripleee
    – user40952
    Jul 24, 2013 at 17:52
  • thanks I saw you had actually answered the question in your previous response. Thanks a lot. I have enough information to make it work.
    – user40952
    Jul 24, 2013 at 19:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .