Unable to get local issuer certificate (but my trusted CA-certificate store seems OK)

Question

This has kept me busy for a good number of hours. I have read a good deal other articles and Stackexchange-questions, and tried other things, but no positive result so far.

Running Ubuntu20/Nginx/Openssl v1.1.1.

Using wget, openssl s_client or curl on normal web resources, I get the message: "Verify return code: 20 (unable to get local issuer certificate)", or equivalent.

$ openssl s_client -connect google.com:443
CONNECTED(00000003)
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = *.google.com
verify return:1
---
Certificate chain
 0 s:CN = *.google.com
   i:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
 1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
   i:C = US, O = Google Trust Services LLC, CN = GTS Root R1
 2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1
   i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIN...

A bit of background. The SSL-handshake used to work for these common web resources. But I had an application that required a self-signed certificate to be added to the trusted CA-certificate store. Worked on that for a good twenty hours, tried many things. In the end decided to 'start anew' and delete my whole trusted certificate store, by deleting everything in /etc/ssl/certs/ and /usr/(local/)share/ca-certificates/) and restoring backups of common CA-certs in these folders, and a restore backup of /etc/ca-certificates.conf. Then ran update-ca-certificates. Also: I downgraded OpenSSL from v1.1.1 to 1.0.2, and then upgraded it again from 1.0.2 to 1.1.1.

Output below to demonstrate that it looks alright.

$ update-ca-certificates -f
Clearing symlinks in /etc/ssl/certs...
done.
Updating certificates in /etc/ssl/certs...
129 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

As far as I see, my trusted cert-store seems fine: it contains the requested root-certificates in the chain. Notice in the above example there are two root-certificates: (1) C = US, O = Google Trust Services LLC, CN = GTS Root R1, and (2) C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA.

I am sure these two root-certificates are in my trusted CA-store. Here's a snippet of the output from a trick suggested by Marlon in NginX client cert authentication fails with “unable to get issuer certificate”

$ awk -v cmd='openssl x509 -noout -subject' ' /BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt
...
subject=OU = GlobalSign ECC Root CA - R4, O = GlobalSign, CN = GlobalSign
subject=OU = GlobalSign ECC Root CA - R5, O = GlobalSign, CN = GlobalSign
subject=C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
subject=OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
subject=OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign
subject=OU = GlobalSign Root CA - R6, O = GlobalSign, CN = GlobalSign
...
subject=C = US, O = Google Trust Services LLC, CN = GTS Root R1
subject=C = US, O = Google Trust Services LLC, CN = GTS Root R2
subject=C = US, O = Google Trust Services LLC, CN = GTS Root R3
subject=C = US, O = Google Trust Services LLC, CN = GTS Root R4

So the root-certificates that the host in my example (google.com) uses are there in my trusted CA-store. Why am I still getting "Verification error: unable to get local issuer certificate"?

Additionally, I'll add the output when I explicitly define the path to the trusted CA-cert store. The SSL-handshake succeeds! What am I overlooking?

$ openssl s_client -CApath /etc/ssl/certs -connect google.com:443
CONNECTED(00000003)
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = *.google.com
verify return:1
---
Certificate chain
 0 s:CN = *.google.com
   i:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
 1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
   i:C = US, O = Google Trust Services LLC, CN = GTS Root R1
 2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1
   i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIN...
...
-----END CERTIFICATE-----
subject=CN = *.google.com

issuer=C = US, O = Google Trust Services LLC, CN = GTS CA 1C3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 6523 bytes and written 392 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

To conclude: I am likely overlooking something, some setting or parameter that may have gotten reset, or set wrongly during my hours of tinkering with the system. However, I just cannot see it, and the sources I've read and tried so far mostly mention making sure my trusted CA-cert store is complete, which I think it is. What am I overlooking? Where should I look, or what should I do to get a grip on this problem?

Answer 1

Solved my own problem and will share the solution here. The certificates in the trusted CA-store were indeed fine. My problem was caused by uninstalling and rebuilding OpenSSL.

The 'fresh' OpenSSL install directory contains among other things a /certs and /private folder and an openssl.cnf file.

$ ll
total 68
drwxr-xr-x  9 root root  4096 Jul 28 12:10 ./
drwxr-xr-x 11 root root  4096 Jul 26 21:58 ../
drwxr-xr-x  2 root root  4096 Jul 26 21:59 bin/
drwxr-xr-x  2 root root  4096 Jul 26 21:59 certs/
-rw-r--r--  1 root root   412 Jul 28 12:10 ct_log_list.cnf
-rw-r--r--  1 root root   412 Jul 28 12:10 ct_log_list.cnf.dist
drwxr-xr-x  3 root root  4096 Jul 26 21:59 include/
drwxr-xr-x  4 root root  4096 Jul 26 21:59 lib/
drwxr-xr-x  6 root root  4096 Jul 26 21:58 man/
drwxr-xr-x  2 root root  4096 Jul 28 12:10 misc/
-rw-r--r--  1 root root 10835 Jul 26 21:59 openssl.cnf
-rw-r--r--  1 root root 10909 Jul 28 12:10 openssl.cnf.dist
drwxr-xr-x  2 root root  4096 Jul 26 21:59 private/

But this server (provisioned by a third party, not a 'fresh, default Ubuntu20) stores its trusted CA-store in /etc/ssl/certs. That is why, in simple terms, OpenSSL did not 'find' my CA-store with all the common certificates. The solution was to delete this original folder, and add symlink to my actual trusted CA-store:

$ rmdir certs    
$ ln -s /etc/ssl/certs ./certs

Also added:

$ rm openssl.cnf
$ rmdir private
$ ln -s /etc/ssl/openssl.cnf ./openssl.cnf
$ ln -s /etc/ssl/private ./private

And now it looks like:

$ ll
total 48
drwxr-xr-x  7 root root  4096 Jul 30 17:02 ./
drwxr-xr-x 11 root root  4096 Jul 26 21:58 ../
drwxr-xr-x  2 root root  4096 Jul 26 21:59 bin/
lrwxrwxrwx  1 root root    14 Jul 30 16:57 certs -> /etc/ssl/certs/
-rw-r--r--  1 root root   412 Jul 28 12:10 ct_log_list.cnf
-rw-r--r--  1 root root   412 Jul 28 12:10 ct_log_list.cnf.dist
drwxr-xr-x  3 root root  4096 Jul 26 21:59 include/
drwxr-xr-x  4 root root  4096 Jul 26 21:59 lib/
drwxr-xr-x  6 root root  4096 Jul 26 21:58 man/
drwxr-xr-x  2 root root  4096 Jul 28 12:10 misc/
lrwxrwxrwx  1 root root    20 Jul 30 17:02 openssl.cnf -> /etc/ssl/openssl.cnf
-rw-r--r--  1 root root 10909 Jul 28 12:10 openssl.cnf.dist
lrwxrwxrwx  1 root root    16 Jul 30 17:02 private -> /etc/ssl/private/

Found the solution by logging in to another server that was provisioned the same way and looking up the OPENSSLDIR (OpenSSL install directory).

openssl version -d

They differred between these two servers. Went to look inside this folder. Found symlinks in the folder on the server where it was working. Found an empty 'certs' folder on the server where it wasn't. Added symlinks. Solved.