I have 2 Linux CentOS 7 machines, a Samba server and a Postfix server. The Postfix server is configured to use SASL authentication and TLS encryption. On the Samba server, I can connect to the Postfix server using openssl.

[root@samba1 ~]# openssl s_client 
-connect mail.example.com:587 
-starttls smtp 
-CAfile /etc/pki/tls/mail.example.com.pem

When I use mailx on the Samba server, I get message "SSL/TLS handshake failed: Unknown error -5938".

[root@samba1 ~]@ echo "Hello World" | mailx -v -s "Test" -A default first.last@hotmail.com
Resolving host mail.example.com . . . done.
Connecting to . . . connected.
SSL/TLS handshake failed: Unknown error -5938.

In /var/log/maillog on the Postfix server, I do see a few TLS events.

Nov 25 19:23:05 mail postfix/smtpd[5659]: initializing the server-side TLS engine
Nov 25 19:23:05 mail postfix/smtpd[5659]: connect from samba1.example.com[]
Nov 25 19:28:05 mail postfix/smtpd[5659]: timeout after UNKNOWN from samba1.example.com[]

This is my /etc/mail.rc file on the Samba server.

[root@samba1 ~]# cat /etc/mail.rc
account default {
set smtp=smtps://mail.example.com:587

set smtp-auth=login
set smtp-auth-user=my.email@example.com
set smtp-auth-password="my_password"

set smtp-use-starttls
set nss-config-dir=/etc/pki/nssdb

set hold
set append
set ask
set crt
set dot
set keep
set emptybox
set indentprefix="> "
set quote
set sendcharsets=iso-8859-1,utf-8
set showname
set showto
set newmail=nopoll
set autocollapse
set markanswered
ignore received in-reply-to message-id references
ignore mime-version content-transfer-encoding
fwdretain subject date from to
set bsdcompat

This article suggests to comment out smtp-use-starttls, which seems to not apply to my situation, since my Postfix server is configured to use TLS. If I comment out smtp-use-starttls, I stil get the 5938 message.

This post suggests that you can ensure both servers are using the same version of TLS. I added ssl-method=tls1 to /etc/mail.rc on the Samba server, and I still get the 5938 message.

Using certutil, I do see my certificate in /etc/pki/nssdb on the Samba server, and it is valid.

[root@samba1 ~]# certutil -V -n "mail.example.com.crt" -d /etc/pki/nssdb -u A
certutil: certificate is valid

Here is the contents of nssdb

-rw-r--r--. 1 root root 65536 Nov 25 17:56 cert8.db
-rw-r--r--. 1 root root  9216 Jun 26 08:51 cert9.db
-rw-r--r--. 1 root root 16384 Nov 25 17:56 key3.db
-rw-r--r--. 1 root root 11264 Jun 26 08:51 key4.db
-rw-r--r--. 1 root root   451 Apr 25  2016 pkcs11.txt
-rw-r--r--. 1 root root 16384 Apr 25  2016 secmod.db

I am not certain what I should do next for this anomaly.

  • There are lots of possibilities here but you should start by running mailx in verbose mode (-v parm). It appears to me that smtps should normally use port 465 and no starttls, so that may be your only/main problem. Of course, the certificate validity (whether self-signed or not) could cause problems. – Julie Pelletier Nov 26 '16 at 3:28
  • Thank you very much for the tips Julie! I updated my question, adding the -v (verbose) option to the mailx command. I also updated the certutil command to show that the certificate being used is valid. I also commented out starttls, and the same problem occurs. Thank you very much for sharing your thoughts. – JeremyCanfield Nov 26 '16 at 5:04
  • Can you try replicating the issue from a different email client such as Thunderbird (or many others) to see what type of error they would give? A simple certificate warning would be a likely answer. – Julie Pelletier Nov 26 '16 at 5:35
  • Thanks Julie. This is where things get very interesting. On the Samba server, I am able to connect to the Postfix server using OpenSSL. Also, using a Windows machine in the network, I can connect to the Postfix server using OpenSSL. On other Windows and Linux machines in the network, I am able to connect to the Postfix server using Thunderbird. In short, it is only when using Mailx that I am not able to connect to the Postfix server. – JeremyCanfield Nov 26 '16 at 5:38
  • 1
    Thank you very much for your explanation. I made the changes you recommended, and the problem is solved. The SSL/TLS handshake is successful, and I am able to connect to Postfix using mailx. If you would like to post your comment as the answer, I would love to accept your answer. – JeremyCanfield Nov 26 '16 at 16:14

dave_thompson_085 answered in the comments:

smtps: means to initially connect with SSL/TLS, which 587 does not support. You want to connect then start SSL/TLS, which is generically called starttls.





Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.