Sendmail Has Stopped Working

Question

I have multiple CentOS servers which all have sendmail setup with the masquerade_domain option. Our email server is Gmail; thus, each of these servers send mail to our domain on Gmail. I have whitelisted each of the server's IP addresses in our gmail domain. It has worked for years...literally years.

Suddenly one of my server's nightly emails stopped working.

With my setup, when an email is sent using the command line from a server, such as:

echo "Hello World" | mutt -s "Test Hello World Email" myname@mydomain.com

...or...

echo "Hello World" | mail -s "Test Hello World Email" myname@mydomain.com

...the maillog shows it being relayed by localhost then to gmail. A successful send looks like this:

from=<root@server.domain.com>, size=483096, class=0, nrcpts=1, msgid=<201806051845@server.domain.com>, proto=ESMTP, daemon=MTA, relay=localhost.localdomain [127.0.0.1]
to=myname@mydomain.com, ctladdr=root (0/0), delay=00:02:00, xdelay=00:00:00, mailer=relay, pri=512775, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (Message accepted for delivery)
STARTTLS=client, relay=aspmx.l.google.com., version=TLSv1/SSLv3, verify=FAIL, cipher=ECDHE-RSA-AES128-GCM-SHA256, bits=128/128
to=<myname@mydomain.com>, ctladdr=<root@server.domain.com> (0/0), delay=00:00:03, xdelay=00:00:03, mailer=esmtp, pri=603096, relay=aspmx.l.google.com. [74.125.124.27], dsn=2.0.0, stat=Sent (OK)

However currently, on the server that stopped working, I see this:

from=<root@server.domain.com>, size=483096, class=0, nrcpts=1, msgid=<201806051835@server.domain.com>, proto=ESMTP, daemon=MTA, relay=localhost.localdomain [127.0.0.1]
to=myname@mydomain.com, ctladdr=root (0/0), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=30006, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (Message accepted for delivery)
STARTTLS=client, relay=aspmx.l.google.com., version=TLSv1/SSLv3, verify=FAIL, cipher=AES128-SHA, bits=128/128
to=<myname@mydomain.com>, ctladdr=<root@server.domain.com> (0/0), delay=00:00:01, xdelay=00:00:01, mailer=relay, pri=120317, relay=aspmx.l.google.com. [108.177.112.27], dsn=5.0.0, stat=Service unavailable
DSN: Service unavailable
Losing ./: savemail panic
SYSERR(root): savemail: cannot save rejected email anywhere

The two /var/log/maillog lines that stand out are:

DSN: Service unavailable

...and...

SYSERR(root): savemail: cannot save rejected email anywhere

I originally assumed the latter log entry (cannot save rejected email) was due to the relay not working; however, since this issue originated, I have tried switching my sendmail configuration to allow local delivery of a few accounts and I updated my /etc/aliases to point many aliases to one of the accounts with local delivery per the virtusertable. ...but I have been failing miserably at making any progress.

It is worth noting that I have double-checked my /etc/hosts file as well as /etc/resolv.conf. Nslookups all appear to be functioning fine and my hostname matches the /etc/hosts file. Further, the public DNS entries for this server match the server's hostname and IP address.

Any help is greatly appreciated.

Answer 1

I wanted to share the solution for my problem. This is not to say this will be the solution for everyone's similar issue, but maybe the troubleshooting path I took may help others.

First, I watched the maillog. Which, for sendmail, on RHEL and CentOS defaults to : /var/log/maillog. You can see my maillog entries above and the two issues I witnessed, which were (simplified):

relay=aspmx.l.google.com., dsn=5.0.0, stat=Service unavailable

...shortened as: DSN: Service unavailable

... and ...

SYSERR(root): savemail: cannot save rejected email anywhere

The latter issues was because I was using sendmail to masquerade all email to our company's primary email server (which was at google). Ultimately what was happening was:

1. The initial email was delivered locally
2. the masquerade attempted to replay the email to google
3. the replay didn't work and a bounce message was created
4. the bounce message was failing to send because it too was trying to be replayed to google

Thus, I could not view the reason for the bounce.

I overcame this and was able to successfully view the bounce details with this command:

(echo Subject: test; echo; echo test)|/usr/sbin/sendmail -Am -i -v myemail@address.com

(Obviously change myemail@address.com to something else; though it's not going to send either way).

The above command allowed me to see the error message in detail which was:

550-5.7.1 [my.servers.ip.address] The IP you're using to send mail is not authorized to
550-5.7.1 send email directly to our servers. Please use the SMTP relay at your
550-5.7.1 service provider instead. Learn more at
550 5.7.1  https://support.google.com/mail/?p=NotAuthorizedError

In reading about this, I quickly learned my server's IP was simply in a Policy Block list that spanned 1000's of IP. I sent a request to have my IP removed from the Policy Block and everything worked again.