0

I am looking for a simple procedure to extract email from my inbox into a file for further processing, using e.g. awk. With "extract" I mean use e.g. regex to get emails with a particular subject.

My Inbox is currently over 1Gb, so I use the tail command to get the last part. Here is an example of two messages:

From aaa.aaa@uni.no Tue Nov 26 17:27:33 2013
Return-path: <aaa.aaa@uni.no>
Envelope-to: fcihh@uib.no
Delivery-date: Tue, 26 Nov 2013 17:27:33 +0100
Received: from alfons.uib.no [2001:700:200:30::141] 
    by alfred.uib.no  with esmtp  (Exim 4.77)
    id 1VlLU5-0003Gg-8R; Tue, 26 Nov 2013 17:27:33 +0100
Received: from thoralf.uib.no (smtp.uib.no) [2001:700:200:13:129:177:13:16] 
    by alfons.uib.no  with esmtps (TLSv1:AES256-SHA:256) (Exim 4.77)
    id 1VlLU5-0002Po-4p; Tue, 26 Nov 2013 17:27:33 +0100
Received: from cipr064089.klientdrift.uib.no [129.177.64.89]:35068 
    by smtp.uib.no  with esmtpsa (Exim 4.69)
    id 1VlLU5-0006dY-35; Tue, 26 Nov 2013 17:27:33 +0100
Message-ID: <5294CBFA.3030408@uni.no>
Date: Tue, 26 Nov 2013 17:27:38 +0100
From: bb b <bb.bb@uni.no>
Organization: Uni Research AS
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.1.1
MIME-Version: 1.0
To: b b <bbb.bbb@uni.no>
CC: Haakon Haegland <hakon.hagland@uni.no>
Subject: Re: ddd
References: <5294C625.9010403@uni.no>
In-Reply-To: <5294C625.9010403@uni.no>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
X-checked-clean: by exiscan on alfons.uib.no
X-Scanner: 2db488888a1c80ee768332c03a147d00 http://tjinfo.uib.no/virus.html
X-UiB-SpamFlag: NO UIB: -18 hits, 8.0 required
X-UiB-SpamReport: spamassassin found;
  -15 From is listed in 'whitelist_SA'
 -3.0 Message received from Norway
X-sender_host_address: 2001:700:200:13:129:177:13:16
X-sender_host_name: thoralf.uib.no
X-UID: 28258                                                 
Status: RO

On 11/26/2013 05:02 PM, b b wrote:
> ggg
> hhh

qqq


From bbb.bbbk@uni.no Tue Nov 26 17:33:10 2013
Return-path: <bbb.bbb@uni.no>
Envelope-to: fcihh@uib.no
Delivery-date: Tue, 26 Nov 2013 17:33:10 +0100
Received: from alfons.uib.no [2001:700:200:30::141] 
    by alfred.uib.no  with esmtp  (Exim 4.77)
    id 1VlLZW-0004jv-7u; Tue, 26 Nov 2013 17:33:10 +0100
Received: from thoralf.uib.no (smtp.uib.no) [2001:700:200:13:129:177:13:16] 
    by alfons.uib.no  with esmtps (TLSv1:AES256-SHA:256) (Exim 4.77)
    id 1VlLZW-0002wL-3T; Tue, 26 Nov 2013 17:33:10 +0100
Received: from cipr064186.klientdrift.uib.no [129.177.64.186]:57257 
    by smtp.uib.no  with esmtpsa (Exim 4.69)
    id 1VlLZW-0006iZ-29; Tue, 26 Nov 2013 17:33:10 +0100
Message-ID: <5294CD3D.9020407@uni.no>
Date: Tue, 26 Nov 2013 17:33:01 +0100
From: b b <bbb.bbb@uni.no>
Organization: Uni CIPR
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20131028 Thunderbird/17.0.10
MIME-Version: 1.0
To: a a <aaa.aaa@uni.no>
CC: Haakon Haegland <hakon.hagland@uni.no>
Subject: Re: ddd
References: <5294C625.9010403@uni.no> <5294CBFA.3030408@uni.no>
In-Reply-To: <5294CBFA.3030408@uni.no>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
X-checked-clean: by exiscan on alfons.uib.no
X-Scanner: 467bf36c80a8700bd89776b495ccd761 http://tjinfo.uib.no/virus.html
X-UiB-SpamFlag: NO UIB: -17.9 hits, 8.0 required
X-UiB-SpamReport: spamassassin found;
  -15 From is listed in 'whitelist_SA'
 -3.0 Message received from Norway
  0.0 BODY: ReplaceTags: Contact
X-sender_host_address: 2001:700:200:13:129:177:13:16
X-sender_host_name: thoralf.uib.no
X-UID: 28259                                                 
Status: RO

txtxt

I guess the format of this email is SMTP http://en.wikipedia.org/wiki/SMTP_protocol ? (I have also seen the words MIME and RFC used, I don't know if these are related) Questions:

  • How do I know where a message ends and another starts?
  • How can I use a tool like awk to extract messages with a given subject field?
5

The format of the message is RFC 2282, the Internet Message Format. The file format sounds like you're describing the mbox format, which is RFC 4155.

According to RFC 4155, a new message in an mbox database begins with the From keyword, a single space then the address of the reported sender, a single space and the UTC date of when the message was received.

Thereafter header fields are separated by newline characters and begin with a letter. If the first character of the new line is whitespace (usually a tab I believe) then it is a continuation of the value of the previous header line. Header keys are separated from header values by : (colon space).

The header and body is separated by a blank line. Each message is also separated by a blank.

As for extracting the subject with awk, first extract the headers then pass it through this:

awk '/^Subject: .*/ {print substr($0,10)}'

This prints the line starting at position 10.

  • 1
    +1 for the reference to mbox format. There is (was at some point, I have not used mbox for 10+ years) a real problem with emails that contain email headers as message body that your approach would not handle. – Anthon Dec 13 '13 at 8:31
  • 3
    Bear in mind that Subject:Foo, Subject :Foo, Subject : Foo (etc, including ones with consecutive whitespace) are legal -- this will fail on some standards compliant headers. – Chris Down Dec 13 '13 at 9:01
  • You say that each message is separated by a blank line. What if the body of the message contains a blank line? – Håkon Hægland Dec 13 '13 at 9:28
  • 1
    @HåkonHægland: The next message must begin with From foo@bar... If the next line does not start a new message then it is considered to be part of the previous message. – bahamat Dec 14 '13 at 2:29
  • @ChrisDown: Yes, good point. Moral of the story, awk is probably not the best mbox processor. Expressing all of the possible permutations can be done much easier in other languages, not to mention existing mbox processing libraries. – bahamat Dec 14 '13 at 2:30
2

The message in your inbox are separated by a double newline followed by From (notice there is no : after from.

That is the simple answer. Of course you can send an email which contains one or more emails as text, including headers and that is where things get messy.

If the sending program (or your programs that write the inbox) does not add length information, you have to rely on heuristics, a simple and effective one being to look at the received date-timestamp of the emails, as your inbox is normally in consecutive timestamps. (but you could sent some email text with 'forged' future timestamps).

If you don't have emails about email headers you should be fine with the double newline + From

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.