Can I Override SPF Policy Using a Whitelist? How?

Question

I'm running postfix on a Linode.

Linux redacted 5.3.11-x86_64-linode131 #1 SMP PREEMPT Wed Nov 13 18:51:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

amavisd-new-postfix/xenial,xenial,now 1:2.10.1-2ubuntu1 all [installed]
postfix/xenial-updates,now 3.1.0-3ubuntu0.3 amd64 [installed]
postfix-mysql/xenial-updates,now 3.1.0-3ubuntu0.3 amd64 [installed]
postfix-pcre/xenial-updates,now 3.1.0-3ubuntu0.3 amd64 [installed]
postfix-policyd-spf-python/xenial,xenial,now 1.3.2-1 all [installed,automatic]

Instead of messing with a quarantine, I have my postfix installation configured to reject RBL and SPFFAIL e-mail. Unfortunately, I have a company that I need to receive e-mail from who has an incorrect SPF record that has been that way for some time. As such, instead of e-mail, I just get nice logs like this:

Apr 1 10:41:42 redacted postfix/smtpd[18833]: NOQUEUE: reject: RCPT from us-smtp-delivery-134.mimecast.com[216.205.24.134]: 550 5.7.1 : Recipient address rejected: Message rejected due to: SPF fail - not authorized. Please see http://www.openspf.net/Why?s=mfrom;id=redacted;ip=216.205.24.134;r=redacted; from=< redacted> to=< redacted> proto=ESMTP helo=< us-smtp-delivery-134.mimecast.com>

Said company doesn't have a contact that is likely to fix this on their website or in their WHOIS information (I assume a registrant @dnstinations.com [some third party vendor] is going to ignore an e-mail from a third party saying DNS is configured wrong).

The first time I got this log instead of an e-mail, I tried whitelisting *.mimecast.com, but that made no difference, so today, I took a look at my main.cf. I saw that the SPF configuration was on a different line than my whitelists, so I thought maybe I could make a separate spf-specific whitelist. Here is what my main.cf looked like before:

smtpd_client_restrictions = permit_sasl_authenticated, permit_mynetworks, check_client_access regexp:/etc/postfix/rbl_client_regex, check_client_access hash:/etc/postfix/rbl_client_override, reject_rbl_client zen.spamhaus.org, reject_rbl_client bl.spamcop.net, reject_rbl_client dnsbl.sorbs.net

smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination, check_policy_service unix:private/policy-spf

smtpd_sender_restrictions = permit_mynetworks, permit_sasl_authenticated, check_sender_access hash:/etc/postfix/access, reject_unknown_sender_domain

Here is what it looks like now:

smtpd_client_restrictions = permit_sasl_authenticated, permit_mynetworks, check_client_access regexp:/etc/postfix/rbl_client_regex, check_client_access hash:/etc/postfix/rbl_client_override, reject_rbl_client zen.spamhaus.org, reject_rbl_client bl.spamcop.net, reject_rbl_client dnsbl.sorbs.net

smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination, regexp:/etc/postfix/spf_client_regex, check_policy_service unix:private/policy-spf

smtpd_sender_restrictions = permit_mynetworks, permit_sasl_authenticated, check_sender_access hash:/etc/postfix/access, reject_unknown_sender_domain

Basically, all I did to the main.cf was add "regexp:/etc/postfix/spf_client_regex," before "check_policy_service unix:private/policy-spf" in the "smtpd_recipient_restrictions" section.

I also created /etc/postfix/spf_client_regex with this entry (this seems safe enough since mimecast is an antispam vendor):

/.*\.mimecast\.com$/    OK

I tested that with postmap -q, got the expected "OK" result, ran "postmap /etc/postfix/spf_client_regex" to update whatever that updates, and reloaded postfix. Unfortunately, mail from this sender continues to be blocked for SPF failures.

So, I expected the steps above to be correct considering that they're basically the same as the steps taken for the whitelist rules in smtpd_client_restrictions section that cause reject_rbl_client rules to be ignored, but it isn't working. So as stated in the title of this post: Can I Override SPF Policy Using a Whitelist? How?

Answer 1

OK, so I did some reading in the manual after the comment from Cliff Armstrong, and I think I've found a solution. First, I found that while smtpd_client_restrictions and smtpd_sender_restrictions both allow for tables to parse, neither allows for policies to process. Next, I found that custom policies can be created, but this seemed like something that shouldn't require me to learn a programming language in order to resolve, so I did a search that lead me to this post with some reasonable gripes about one particular spf policy delegate and explanations about how to whitelist in both spf policy delegates that are currently available.

As luck would have it, I already had the "better" policy delegate installed, so I didn't have to change that:

~$ apt list | grep policyd-spf

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

postfix-policyd-spf-perl/xenial,xenial 2.010-2 all
postfix-policyd-spf-python/xenial,xenial,now 1.3.2-1 all [installed,automatic]

Next, I reviewed the commented file (well, it was gzipped in my case, so I made a copy to gunzip and then deleted that copy when I was done) and found these relevant comments:

#  Whitelist: CIDR Notation list of IP addresses not to check SPF for.
#  Example (default is no whitelist):
#  Whitelist = 192.168.0.0/31,192.168.1.12

#  Domain_Whitelist: List of domains whose sending IPs (defined by passing
#  their SPF check should be whitelisted from SPF.
#  Example (default is no domain whitelist):
#  Domain_Whitelist = pobox.com,trustedforwarder.org

Based on that, it sounds like domain_whitelist won't work for me since the comment says "defined by passing their SPF check should be whitelisted from SPF." This seems odd (if the sender passed the domain's spf check, I wouldn't need to whitelist), but I still have the whitelist option, so I found mimecast spf documentation and then collected the IPs using nslookup:

~$ nslookup
> set type=txt
> us._extnetblocks.mimecast.com
;; Truncated, retrying in TCP mode.
Server:         50.116.58.5
Address:        50.116.58.5#53

Non-authoritative answer:
us._extnetblocks.mimecast.com   text = "v=spf1 ip4:207.211.30.40 ip4:207.211.30.41 ip4:207.211.30.42 ip4:207.211.30.43 ip4:207.211.30.44 ip4:207.211.30.45 ip4:207.211.30.46 ip4:207.211.30.47 ip4:207.211.30.48 ip4:207.211.30.49 ip4:205.139.111.40 ip4:205.139.111.41 ip4:205.139.111.42 " "ip4:205.139.111.43 ip4:205.139.111.44 ip4:205.139.111.45 ip4:205.139.111.46 ip4:205.139.111.47 ip4:205.139.111.48 ip4:205.139.111.49 ~all"

Authoritative answers can be found from:
mimecast.com    nameserver = dns02.mimecast.net.
mimecast.com    nameserver = dns03.mimecast.net.
mimecast.com    nameserver = dns04.mimecast.net.
mimecast.com    nameserver = dns01.mimecast.net.

Then I took those results and dropped them in the config file:

Whitelist = 207.211.30.40, 207.211.30.41, 207.211.30.42, 207.211.30.43, 207.211.30.44, 207.211.30.45, 207.211.30.46, 207.211.30.47, 207.211.30.48, 207.211.30.49, 205.139.111.40, 205.139.111.41, 205.139.111.42, 205.139.111.43, 205.139.111.44, 205.139.111.45, 205.139.111.46, 205.139.111.47, 205.139.111.48, 205.139.111.49

This isn't ideal since it will automatically allow these IPs to send from other domains and the IPs could change hands, but it is good enough for my private implementation. As such, I'm not going to be trying to learn to create my own delegate or hiring someone to create one for me. If I get another similar NOQUEUE result in spite of these changes, I'll add a note to this answer.