Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

My problem is that I've been a heavy user of GMail's pop3-import feature and used it to pull mail from other GMail Accounts. Back then I did, however, only import new mail, not all the mail previously stored on the server. Now that I've started to use mutt as my mail client of choice I've decided to import all my former mail accounts and thus archive all my old mail locally.

My initial hope was that I could easily weed out the duplicate mails by using a tool like fdupes, but what I did not anticipate was that the Mail-Header is altered slightly by GMail when it it retrieves a mail via Pop3 as it can be seen here:

@@ -1,7 +1,16 @@
 Return-Path: <xxxx@googlemail.com>
 Delivered-To: unknown
 Received: from pop.gmail.com ( by localhost with POP3-SSL;
-  10 May 2011 13:35:06 -0000
+  10 May 2011 14:29:41 -0000
+Delivered-To: yyyy@gmail.com
+Received: by with SMTP id j7cs172325bkg;
+        Sun, 2 May 2010 15:33:19 -0700 (PDT)
+Received: by with SMTP id p15mr6011875bkt.172.1272839446530;
+        Sun, 02 May 2010 15:30:46 -0700 (PDT)
+Received-SPF: softfail (google.com: best guess record for domain of transitioning xxxx@googlemail.com does not designate as permitted sender) client-ip=;
+Received: by with POP3 id 17mf826641bwz.107;
+        Sun, 02 May 2010 15:30:46 -0700 (PDT)
+X-Gmail-Fetch-Info: xxxx@gmail.com 1 smtp.gmail.com 995 xxxx
 Received: from aequitas ( [])
         by mx.google.com with ESMTPS id e20sm18902485fga.1.2008.
         (version=TLSv1/SSLv3 cipher=RC4-MD5);

The original looked like this: http://pastebin.com/U6YzNySP Is there an easy way to get rid of those "duplicate files" in an easy way?

share|improve this question
up vote 3 down vote accepted

Use the ESMTPS id from mx.google.com to identify duplicates. These should be unmodified. In the example above: by mx.google.com with ESMTPS id e20sm18902485fga.1.2008.

A very simple implementation would put all mails in one dir, extract the id and symlink the file to the id without using -f. Like:

for FILE in *; do
     if test -f ${smptid}; then
         echo "DUPE: ${FILE}"
         ln -s ${FILE} ${smtpid}
share|improve this answer
For the most part this worked (finding many duplicates, but sadly not everything apparently [some with multiple daisy-chained SMTPids as it passed over multiple redirection]). But I also figured that I could just go to GMail and delete all the retrieved Mails (as I auto-labeled them anyway). So I did that and it re-downloading them via pop helped me clean out everything. – XQYZ May 11 '11 at 5:34

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.