This has kept me busy for a good number of hours. I have read a good deal other articles and Stackexchange-questions, and tried other things, but no positive result so far.
Running Ubuntu20/Nginx/Openssl v1.1.1.
Using wget, openssl s_client or curl on normal web resources, I get the message: "Verify return code: 20 (unable to get local issuer certificate)", or equivalent.
$ openssl s_client -connect google.com:443
CONNECTED(00000003)
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = *.google.com
verify return:1
---
Certificate chain
0 s:CN = *.google.com
i:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
i:C = US, O = Google Trust Services LLC, CN = GTS Root R1
2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1
i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIN...
A bit of background. The SSL-handshake used to work for these common web resources. But I had an application that required a self-signed certificate to be added to the trusted CA-certificate store. Worked on that for a good twenty hours, tried many things. In the end decided to 'start anew' and delete my whole trusted certificate store, by deleting everything in /etc/ssl/certs/ and /usr/(local/)share/ca-certificates/) and restoring backups of common CA-certs in these folders, and a restore backup of /etc/ca-certificates.conf. Then ran update-ca-certificates. Also: I downgraded OpenSSL from v1.1.1 to 1.0.2, and then upgraded it again from 1.0.2 to 1.1.1.
Output below to demonstrate that it looks alright.
$ update-ca-certificates -f
Clearing symlinks in /etc/ssl/certs...
done.
Updating certificates in /etc/ssl/certs...
129 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
As far as I see, my trusted cert-store seems fine: it contains the requested root-certificates in the chain. Notice in the above example there are two root-certificates: (1) C = US, O = Google Trust Services LLC, CN = GTS Root R1, and (2) C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA.
I am sure these two root-certificates are in my trusted CA-store. Here's a snippet of the output from a trick suggested by Marlon in NginX client cert authentication fails with “unable to get issuer certificate”
$ awk -v cmd='openssl x509 -noout -subject' ' /BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt
...
subject=OU = GlobalSign ECC Root CA - R4, O = GlobalSign, CN = GlobalSign
subject=OU = GlobalSign ECC Root CA - R5, O = GlobalSign, CN = GlobalSign
subject=C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
subject=OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
subject=OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign
subject=OU = GlobalSign Root CA - R6, O = GlobalSign, CN = GlobalSign
...
subject=C = US, O = Google Trust Services LLC, CN = GTS Root R1
subject=C = US, O = Google Trust Services LLC, CN = GTS Root R2
subject=C = US, O = Google Trust Services LLC, CN = GTS Root R3
subject=C = US, O = Google Trust Services LLC, CN = GTS Root R4
So the root-certificates that the host in my example (google.com) uses are there in my trusted CA-store. Why am I still getting "Verification error: unable to get local issuer certificate"?
Additionally, I'll add the output when I explicitly define the path to the trusted CA-cert store. The SSL-handshake succeeds! What am I overlooking?
$ openssl s_client -CApath /etc/ssl/certs -connect google.com:443
CONNECTED(00000003)
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = *.google.com
verify return:1
---
Certificate chain
0 s:CN = *.google.com
i:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
i:C = US, O = Google Trust Services LLC, CN = GTS Root R1
2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1
i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIN...
...
-----END CERTIFICATE-----
subject=CN = *.google.com
issuer=C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 6523 bytes and written 392 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
To conclude: I am likely overlooking something, some setting or parameter that may have gotten reset, or set wrongly during my hours of tinkering with the system. However, I just cannot see it, and the sources I've read and tried so far mostly mention making sure my trusted CA-cert store is complete, which I think it is. What am I overlooking? Where should I look, or what should I do to get a grip on this problem?
c_rehash /etc/ssl/certs
? Some programs will use theca-certificates.crt
file. Some programs look for a file that matches the cert hash.ca-certificates.crt
file to be skipped. It should contain the same certificates that are in the directory, so it is seen as a duplicate. github.com/openssl/openssl/blob/master/tools/c_rehash.in#L173/usr/lib/ssl/certs
which should symlink to/etc/ssl/certs
(whichupdate-ca-certificates
should have rebuilt correctly including the hashnames) (although maybe not if it was nonempty and you didn't do-f
?), and file/usr/lib/ssh/cert.pem
which should not exist -- unless you have envvar(s)SSL_CERT_{DIR,FILE}
set; do you? It does not use/etc/ssl/certs/ca-certificates.crt
(althoughupdate-ca-certificates
does build that) so what is in there doesn't matter foropenssl
andwget
;curl
apparently does use that file.apt-get reinstall
.