Mailx SSL/TLS handshake failed: Unknown error -5938

Question

I have 2 Linux CentOS 7 machines, a Samba server and a Postfix server. The Postfix server is configured to use SASL authentication and TLS encryption. On the Samba server, I can connect to the Postfix server using openssl.

[root@samba1 ~]# openssl s_client 
-connect mail.example.com:587 
-starttls smtp 
-CAfile /etc/pki/tls/mail.example.com.pem

When I use mailx on the Samba server, I get message "SSL/TLS handshake failed: Unknown error -5938".

[root@samba1 ~]@ echo "Hello World" | mailx -v -s "Test" -A default [email protected]
Resolving host mail.example.com . . . done.
Connecting to 192.168.0.10:587 . . . connected.
SSL/TLS handshake failed: Unknown error -5938.

In /var/log/maillog on the Postfix server, I do see a few TLS events.

Nov 25 19:23:05 mail postfix/smtpd[5659]: initializing the server-side TLS engine
Nov 25 19:23:05 mail postfix/smtpd[5659]: connect from samba1.example.com[192.168.0.12]
Nov 25 19:28:05 mail postfix/smtpd[5659]: timeout after UNKNOWN from samba1.example.com[192.168.0.12]

This is my /etc/mail.rc file on the Samba server.

[root@samba1 ~]# cat /etc/mail.rc
account default {
set smtp=smtps://mail.example.com:587

#Authentication
set smtp-auth=login
set [email protected]
set smtp-auth-password="my_password"

#Encryption
set smtp-use-starttls
set nss-config-dir=/etc/pki/nssdb
}

set hold
set append
set ask
set crt
set dot
set keep
set emptybox
set indentprefix="> "
set quote
set sendcharsets=iso-8859-1,utf-8
set showname
set showto
set newmail=nopoll
set autocollapse
set markanswered
ignore received in-reply-to message-id references
ignore mime-version content-transfer-encoding
fwdretain subject date from to
set bsdcompat

This article suggests to comment out smtp-use-starttls, which seems to not apply to my situation, since my Postfix server is configured to use TLS. If I comment out smtp-use-starttls, I stil get the 5938 message.

This post suggests that you can ensure both servers are using the same version of TLS. I added ssl-method=tls1 to /etc/mail.rc on the Samba server, and I still get the 5938 message.

Using certutil, I do see my certificate in /etc/pki/nssdb on the Samba server, and it is valid.

[root@samba1 ~]# certutil -V -n "mail.example.com.crt" -d /etc/pki/nssdb -u A
certutil: certificate is valid

Here is the contents of nssdb

-rw-r--r--. 1 root root 65536 Nov 25 17:56 cert8.db
-rw-r--r--. 1 root root  9216 Jun 26 08:51 cert9.db
-rw-r--r--. 1 root root 16384 Nov 25 17:56 key3.db
-rw-r--r--. 1 root root 11264 Jun 26 08:51 key4.db
-rw-r--r--. 1 root root   451 Apr 25  2016 pkcs11.txt
-rw-r--r--. 1 root root 16384 Apr 25  2016 secmod.db

I am not certain what I should do next for this anomaly.

Answer 1

dave_thompson_085 answered in the comments:

smtps: means to initially connect with SSL/TLS, which 587 does not support. You want to connect then start SSL/TLS, which is generically called starttls.

Use:

smtp=server:587

and

smtp-use-starttls