I have huge mailboxes I would like to manage based on the age of their content: backup, archival or removal. I would like to split these mailboxes along the date of their contained messages.

For a mailbox:


Here is the scheme of a new directory structure I would like to span it on:


Is there a way to achieve this splitting without too many forked commands?

migrated from serverfault.com Mar 18 '15 at 19:06

This question came from our site for system and network administrators.


First off, a couple of clarifications. The Date: header is not very reliable, as the sender could set it to pretty much anything. However, the alternatives are not very attractive, either; some mail clients actually try to parse the Received: headers, but because the contents of the header are not properly standardized, that can fail. If you have your messages in Berkeley mbox format, the From_ pseudo-header usually contains the local delivery timestamp, but I imagine there are delivery programs which omit that, too (and it seems like you are not using mbox anyway).

So, let's just look at parsing the RFC822 Date: header without forking.

An additional conundrum is that not all senders use a valid RFC822 date in this header. Assuming you are not sorting spam, you can probably identify an alternative pattern for the correspondent(s) who do this; I will simply ignore that can of worms here. (I only bumped into this when handling spam, but it seemed to be fairly prevalent in email from Japan, IIRC.)

I see you don't care about the day of month, but I'm including that here as well just in case somebody else would find this useful.

# Poor man's associative array...  Remove leading zeros if desired
# (but then also change the MONTH regex below, as per comment there)
MONTHS='Jan 01
Feb 02
Mar 03
Apr 04
May 05
Jun 06
Jul 07
Aug 08
Sep 09
Oct 10
Nov 11
Dec 12'

* ^Date:[   ]*[A-Z][a-z][a-z], \/[ 0-3][0-9] [A-Z][a-z][a-z] [12][0-9][0-9][0-9]
    * RDATE ?? ^ *\/[1-9][0-9]?
    { DAY=$MATCH }
    * RDATE ?? () \/[12][0-9][0-9][0-9]
    { YEAR=$MATCH }
    * RDATE ?? () \/[A-Z][a-z][a-z]
    { MON=$MATCH }
    :0  # Adjust if you don't want leading zeros -- $MON \/[1-9][0-9]?
    * $ MONTHS ?? $MON \/[01][0-9]
    { MONTH=$MATCH }

    :0  # Assume TOPIC is set by caller, or somehow determined above

Just to recap the constructs used above, the \/ special token causes whatever matches after it to be captured in MATCH (Procmail doesn't have proper back references) and VAR ?? as the prefix of a condition causes the condition to be applied against the value of the variable VAR instead of against the message's headers. Finally, $ is required if you want the condition to contain the expanded value of a variable (normally, $ just means a newline). Oh, and as always, the whitespace in [ ]* is supposed to be a tab and a space.

The recipe above is mainly useful for parsing individual messages. If you want to split a mailbox per date, I don't really see how you could avoid parsing each message separately and then filing each where it belongs.

If you really want to squeeze optimize, and have messages from a constrained period of time, I suppose you could enumerate regexes for each of the months you want to cover, something like

for month in 2014/11 2014/12 2015/01 2015/02 2015/03; do
    printf ':0\n* ^Date:[ \t]*[A-Z][a-z][a-z], [ 0-3][0-9] %s\n%s/$TOPIC\n' \
        "$(date -d "${month%/*}-${month#*/}-01" +'%b %Y')" "$month"
done >months.rc

but I'm not entirely convinced it will actually spare you any cycles (measure if you must! :-)

  • FWIW, the use of date -d *date* to obtain information for a date in the past is not supported on OSX. Try this if you are on OSX / BSD. – tripleee Mar 26 '15 at 16:06

To achieve this task I used a shell script and a procmailrc file.

The core of the shell script is just a way to split a huge mailbox and feed it to procmail with a tuned procmailrc file:

$ cat <<eof >splitter.sh


formail -s <${orig_mailbox} procmail ${HOME}/src/splitter_procmailrc

And the procmailrc file is based on a single rule which will use date to extract the correct fields from the Date: header (which hopefully was a system generated one on a clean OS). date will also output the required fields YYYY/MM with the format "+%Y/%m".

cat <<eof >${HOME}/src/splitter_procmailrc


:0 w
* ^Subject: TOPIC
* ^Date: \/.*$
|    (    ;\
          ORIG_MONTH_DIR=`date -j -f "%a, %d %b %Y" "${MATCH}" "+%Y/%m"` ;\
          [ -d ${ORIG_MONTH_DIR} ] || mkdir -p ${ORIG_MONTH_DIR} ;\
          cat >>${ORIG_MONTH_DIR}/TOPIC ;\
:0 E

The second rule is here just in case I catch any error on the first one so as to be able to sort the problem and tune the procmailrc file.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.