valid SSL certificate verification reports as "Self-signed" and fails on ubuntu 14.04 for godaddy CA signed sites, despite root CAs being installed

Question

System: Ubuntu 14.04, apt-upgraded to the latest. openssl, ca-certificates, wget installed

Symptoms:

wget https://api.xxx.io will result in this error:

ERROR: cannot verify api.xxx.io's certificate, issued by '/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2':
  Unable to locally verify the issuer's authority.
To connect to api.xxx.io insecurely, use `--no-check-certificate'.

Where as wget https://google.com works fine.

openssl s_client -connect api.xxx.io:443 results in this:

CONNECTED(00000003)
depth=3 C = US, O = "The Go Daddy Group, Inc.", OU = Go Daddy Class 2 Certification Authority
verify error:num=19:self signed certificate in certificate chain
verify return:0

I attempted to reinstall godaddy's root certificate:

1. downloaded their ca cert crt bundle to /usr/share/ca-certificates/extra
2. ran dpkg-reconfigure ca-certificates and followd the prompts.

No luck.

also tried c_rehash with no luck.

This site gets successfully verified on an ubuntu 16 system. Anything else I should try?

edit: running strace -e open openssl s_client -connect api.xxx.io:443 shows that it isn't even opening ca-certificates.crt:

open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libssl.so.1.0.0", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/ssl/openssl.cnf", O_RDONLY) = 3
open("/proc/meminfo", O_RDONLY|O_CLOEXEC) = 3
open("/dev/urandom", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 3
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 3
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 3
open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/etc/host.conf", O_RDONLY|O_CLOEXEC) = 3
open("/etc/hosts", O_RDONLY|O_CLOEXEC)  = 3
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libresolv.so.2", O_RDONLY|O_CLOEXEC) = 3

edit 2: If I pass the CApath to openssl, it works. openssl s_client -connect api.xxx.io:443 -showcerts -CApath /etc/ssl/certs

I've tried editing openssl.cnf and using the environment variable overrides and it still won't work otherwise.

edit3: updated the question with a single host rather than 2 different hosts with 2 different certs.

Answer 1

As editted much of your question is probably a crossdupe of https://superuser.com/questions/903247/ssl-root-ca-certificate-is-not-recognized-although-present-in-the-trust-store and https://serverfault.com/questions/607233/how-to-make-openssl-s-client-using-default-c .

I haven't confirmed Ubuntu source, but Ubuntu 14.04 nominally uses OpenSSL 1.0.1f which upstream has a bug that causes s_client (and some others) to not use the default truststore when you don't specify any -CA{path,file} options, and 16.04 uses 1.0.2g which has this fixed. (Apparently so did 16.10, but that was not LTS and is no longer supported.)

The config file is irrelevant; s_client doesn't use any config file settings other than the 'library-global' ones for ENGINE's and addedOID's, neither of which is relevant to this issue.

But note www.tcell.io (where your wget was redirected) and api.tcell.io (where you told openssl s_client to connect) are different machines. According to www.ssllabs.com/ssltest:

• api.tcell.io 52.8.231.1 correctly serves a GoDaddy-SecureG2 cert (serial 00c8c641d43c76286c) with the necessary chain certs (to be exact #2 and #3 are necessary; #4 is the root and is not necessary but is acceptable)

• www.tcell.io 13.57.73.170 serves the same cert but NOT the chain certs, in violation of the RFCs (which would reduce its grade except that it ALSO uses DH-1024 which already capped it at B). Without the chain certs wget was unable to verify the certificate. If the 'bundle' you installed added the chain certs to your truststore, which is not the way web-PKI is intended to operate, then wget using OpenSSL is able to build the chain and verify it.