2

This has kept me busy for a good number of hours. I have read a good deal other articles and Stackexchange-questions, and tried other things, but no positive result so far.

Running Ubuntu20/Nginx/Openssl v1.1.1.

Using wget, openssl s_client or curl on normal web resources, I get the message: "Verify return code: 20 (unable to get local issuer certificate)", or equivalent.

$ openssl s_client -connect google.com:443
CONNECTED(00000003)
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = *.google.com
verify return:1
---
Certificate chain
 0 s:CN = *.google.com
   i:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
 1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
   i:C = US, O = Google Trust Services LLC, CN = GTS Root R1
 2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1
   i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIN...

A bit of background. The SSL-handshake used to work for these common web resources. But I had an application that required a self-signed certificate to be added to the trusted CA-certificate store. Worked on that for a good twenty hours, tried many things. In the end decided to 'start anew' and delete my whole trusted certificate store, by deleting everything in /etc/ssl/certs/ and /usr/(local/)share/ca-certificates/) and restoring backups of common CA-certs in these folders, and a restore backup of /etc/ca-certificates.conf. Then ran update-ca-certificates. Also: I downgraded OpenSSL from v1.1.1 to 1.0.2, and then upgraded it again from 1.0.2 to 1.1.1.

Output below to demonstrate that it looks alright.

$ update-ca-certificates -f
Clearing symlinks in /etc/ssl/certs...
done.
Updating certificates in /etc/ssl/certs...
129 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

As far as I see, my trusted cert-store seems fine: it contains the requested root-certificates in the chain. Notice in the above example there are two root-certificates: (1) C = US, O = Google Trust Services LLC, CN = GTS Root R1, and (2) C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA.

I am sure these two root-certificates are in my trusted CA-store. Here's a snippet of the output from a trick suggested by Marlon in NginX client cert authentication fails with “unable to get issuer certificate”

$ awk -v cmd='openssl x509 -noout -subject' ' /BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt
...
subject=OU = GlobalSign ECC Root CA - R4, O = GlobalSign, CN = GlobalSign
subject=OU = GlobalSign ECC Root CA - R5, O = GlobalSign, CN = GlobalSign
subject=C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
subject=OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
subject=OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign
subject=OU = GlobalSign Root CA - R6, O = GlobalSign, CN = GlobalSign
...
subject=C = US, O = Google Trust Services LLC, CN = GTS Root R1
subject=C = US, O = Google Trust Services LLC, CN = GTS Root R2
subject=C = US, O = Google Trust Services LLC, CN = GTS Root R3
subject=C = US, O = Google Trust Services LLC, CN = GTS Root R4

So the root-certificates that the host in my example (google.com) uses are there in my trusted CA-store. Why am I still getting "Verification error: unable to get local issuer certificate"?

Additionally, I'll add the output when I explicitly define the path to the trusted CA-cert store. The SSL-handshake succeeds! What am I overlooking?

$ openssl s_client -CApath /etc/ssl/certs -connect google.com:443
CONNECTED(00000003)
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = *.google.com
verify return:1
---
Certificate chain
 0 s:CN = *.google.com
   i:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
 1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
   i:C = US, O = Google Trust Services LLC, CN = GTS Root R1
 2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1
   i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIN...
...
-----END CERTIFICATE-----
subject=CN = *.google.com

issuer=C = US, O = Google Trust Services LLC, CN = GTS CA 1C3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 6523 bytes and written 392 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

To conclude: I am likely overlooking something, some setting or parameter that may have gotten reset, or set wrongly during my hours of tinkering with the system. However, I just cannot see it, and the sources I've read and tried so far mostly mention making sure my trusted CA-cert store is complete, which I think it is. What am I overlooking? Where should I look, or what should I do to get a grip on this problem?

5
  • Have you run c_rehash /etc/ssl/certs? Some programs will use the ca-certificates.crt file. Some programs look for a file that matches the cert hash. Jul 28 at 19:38
  • @AndrewLowther: Thanks for the suggestion. Ran it with root user. No positive result observed. This may be interesting output though: "Doing /etc/ssl/certs; WARNING: Skipping duplicate certificate ca-certificates.crt". Any idea why it is skipping this one essential file? Is this something to look into? Jul 28 at 19:59
  • it is fine for the ca-certificates.crt file to be skipped. It should contain the same certificates that are in the directory, so it is seen as a duplicate. github.com/openssl/openssl/blob/master/tools/c_rehash.in#L173 Jul 28 at 20:08
  • OpenSSL (library) on Ubuntu by default looks in directory /usr/lib/ssl/certs which should symlink to /etc/ssl/certs (which update-ca-certificates should have rebuilt correctly including the hashnames) (although maybe not if it was nonempty and you didn't do -f?), and file /usr/lib/ssh/cert.pem which should not exist -- unless you have envvar(s) SSL_CERT_{DIR,FILE} set; do you? It does not use /etc/ssl/certs/ca-certificates.crt (although update-ca-certificates does build that) so what is in there doesn't matter for openssl and wget; curl apparently does use that file. Jul 29 at 16:23
  • Also, in general, after messing up package-installed things on a well-packaged distro like Ubuntu, rather than trying to rebuild by hand I would first try apt-get reinstall. Jul 29 at 16:24
2

Solved my own problem and will share the solution here. The certificates in the trusted CA-store were indeed fine. My problem was caused by uninstalling and rebuilding OpenSSL.

The 'fresh' OpenSSL install directory contains among other things a /certs and /private folder and an openssl.cnf file.

$ ll
total 68
drwxr-xr-x  9 root root  4096 Jul 28 12:10 ./
drwxr-xr-x 11 root root  4096 Jul 26 21:58 ../
drwxr-xr-x  2 root root  4096 Jul 26 21:59 bin/
drwxr-xr-x  2 root root  4096 Jul 26 21:59 certs/
-rw-r--r--  1 root root   412 Jul 28 12:10 ct_log_list.cnf
-rw-r--r--  1 root root   412 Jul 28 12:10 ct_log_list.cnf.dist
drwxr-xr-x  3 root root  4096 Jul 26 21:59 include/
drwxr-xr-x  4 root root  4096 Jul 26 21:59 lib/
drwxr-xr-x  6 root root  4096 Jul 26 21:58 man/
drwxr-xr-x  2 root root  4096 Jul 28 12:10 misc/
-rw-r--r--  1 root root 10835 Jul 26 21:59 openssl.cnf
-rw-r--r--  1 root root 10909 Jul 28 12:10 openssl.cnf.dist
drwxr-xr-x  2 root root  4096 Jul 26 21:59 private/

But this server (provisioned by a third party, not a 'fresh, default Ubuntu20) stores its trusted CA-store in /etc/ssl/certs. That is why, in simple terms, OpenSSL did not 'find' my CA-store with all the common certificates. The solution was to delete this original folder, and add symlink to my actual trusted CA-store:

$ rmdir certs    
$ ln -s /etc/ssl/certs ./certs

Also added:

$ rm openssl.cnf
$ rmdir private
$ ln -s /etc/ssl/openssl.cnf ./openssl.cnf
$ ln -s /etc/ssl/private ./private

And now it looks like:

$ ll
total 48
drwxr-xr-x  7 root root  4096 Jul 30 17:02 ./
drwxr-xr-x 11 root root  4096 Jul 26 21:58 ../
drwxr-xr-x  2 root root  4096 Jul 26 21:59 bin/
lrwxrwxrwx  1 root root    14 Jul 30 16:57 certs -> /etc/ssl/certs/
-rw-r--r--  1 root root   412 Jul 28 12:10 ct_log_list.cnf
-rw-r--r--  1 root root   412 Jul 28 12:10 ct_log_list.cnf.dist
drwxr-xr-x  3 root root  4096 Jul 26 21:59 include/
drwxr-xr-x  4 root root  4096 Jul 26 21:59 lib/
drwxr-xr-x  6 root root  4096 Jul 26 21:58 man/
drwxr-xr-x  2 root root  4096 Jul 28 12:10 misc/
lrwxrwxrwx  1 root root    20 Jul 30 17:02 openssl.cnf -> /etc/ssl/openssl.cnf
-rw-r--r--  1 root root 10909 Jul 28 12:10 openssl.cnf.dist
lrwxrwxrwx  1 root root    16 Jul 30 17:02 private -> /etc/ssl/private/

Found the solution by logging in to another server that was provisioned the same way and looking up the OPENSSLDIR (OpenSSL install directory).

openssl version -d

They differred between these two servers. Went to look inside this folder. Found symlinks in the folder on the server where it was working. Found an empty 'certs' folder on the server where it wasn't. Added symlinks. Solved.

1
  • Interesting, today I've had a similar problem with dotnet new -i on a provisioned SLES 12.5 and I've found my openssl solution from here. Now I'm just reading offline to search similar situations and what you've written here makes sense to me to better understand (or to get closer to) the possible root cause.
    – Giulio
    Aug 18 at 18:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.