Processing multiple mails in a Postfix mailbox by making cronjob run per second-is that considered good practice?

Question

I need to process at least three thousand incoming emails an hour that are received in a postfix mailbox. Processing involves extracting email attachments and sending the content of an email attachment to an external email address. For each email attachment I send one email. I think I could achieve this by writing a script to cronjob to sleep every second but I am not sure that this is good practice(doing this will easily give me approximate 3600 mails processed an hour). Please advise on the best way to approach this. I already have a script that extract emails and sends email, the only problem with the scripts is that it can only handle one email at a time. If I use per minute cronjob I can only process 60 mails per hour while the target is really at least 3000 mails per hour.

Answer 1

Procmail trivially solves this, provided your server has the CPU and bandwidth to sustain the flow. If you already have a script which takes care of this, simply pipe each incoming message to your script. Put the following in your $HOME/.procmailrc:

:0
| /path/to/your/script

The script receives the message as its standard input, and is responsible for delivering or otherwise handling the message from here on. (In other words, Procmail will not deliver this message into your inbox at all. See below for how to modify this behavior.)

(Procmail is not strictly necessary for this, but it adds a nice safety net so that your script doesn't need to cope with all the possible error conditions. You could simply tuck the pipeline in your .forward if your script is robust enough. This is basically what @number5's comment tells you, too, except it does this in Postfix' config file, instead of using the .forward facility.)

If two instances of the script cannot run concurrently (for example, because it needs exclusive access to a back-end database), add a lock file:

:0:yourscript.lock
| /path/to/your/script

This will cause Procmail to look for the file yourscript.lock and, if it exists, wait until it goes away; then create the file, run the recipe, and remove the lock file.

Using a lockfile forces deliveries to be serialized. This will reduce performance, though. If at all possible, it would be better to make the script robust under parallel execution.

On the other hand, if your script incurs a heavy load on the server, you might not want to run multiple concurrent instances; in this scenario, performance might actually improve if you force serialized delivery.

If you want a copy in your inbox as well, clone a copy when delivering to your script:

:0c # or :0c:yourscript.lock
| /path/to/your/script

You can also add a condition, so that e.g. only messages with a particular subject line are piped to your script. The conditions are specified with an asterisk as the first character, followed by a regular expression which needs to match the message's headers.

:0
* ^Subject: xyzzy$
| /path/to/your/script

If the above is not suitable, the following recipe will extract all attachments to a directory and send an email for each incoming message. Looping over the attachments is probably best done from an external script like above, but this should at least give you a whiff of what it would look like to do something a bit more involved in Procmail itself.

METAMAIL_TMPDIR=`mktemp -d /tmp/extracted.XXXXXXXXX`

# Crude attachment extraction ... how are you currently doing this?
:0c
| metamail -w -d

COUNT=`find "$METAMAIL_TMPDIR" -printf "%i\n" | wc -l`

:0
| ( echo Subject: $COUNT attachments extracted into $METAMAIL_TMPDIR; echo; echo ) \
  | sendmail -oi [email protected]