5

I want to download a file (https://discovery.ucl.ac.uk/1575442/1/Palmisanoetal.zip) from a public server via the R command

temp <- tempfile()   
utils::download.file(db_url, temp, method = 'curl')

This does not work on my Ubuntu 18.04.3 LTS (Bionic Beaver) system. I get the following error:

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
Error in utils::download.file(db_url, temp, method = "curl") : 
'curl' call had nonzero exit status

I get the same error on the command line with curl (curl https://discovery.ucl.ac.uk/1575442/1/Palmisanoetal.zip).

I did some experiments and googling and realized that I can access the file without any problems with my Browser (Chromium). My system/curl seems to lack a CA certificate that my browser has. I tried to determine which certificate this server is using with openssl s_client -showcerts -servername discovery.ucl.ac.uk -connect discovery.ucl.ac.uk:443 and added the result (QuoVadis EV SSL ICA G3) to my /etc/ssl/certs/ca-certificates.crt file. This did not solve the problem.

I don't want to solve this with the curl --insecure flag. I also don't have any control over https://discovery.ucl.ac.uk. I just want to access the file with R.

3 Answers 3

12

Curl is failing because that site is incorrectly configured

Certificates are used to sign other certificates, forming chains. A CA has a root certificate, which is trusted by operating systems and browsers. This root certificate is most commonly used to sign one or several intermediate certificates, which in turn are used to sign leaf certificates (that can not sign other certificates), which are what websites use.

Browsers and operating systems tend to carry only the root certificates, but to verify a leaf certificate (and establish a secure connection), a client needs the entire chain of certificates. In practice, that means that a website must not just supply its leaf certificate, it must also supply the used intermediate certificate. And discovery.ucl.ac.uk fails to do that.

I'll show you.

Finding the problem

openssl is a X509 / SSL swiss army knife that proves very useful here:

% openssl s_client -connect discovery.ucl.ac.uk:443 -servername discovery.ucl.ac.uk -showcerts
CONNECTED(00000003)
depth=0 jurisdictionC = GB, businessCategory = Government Entity, serialNumber = November-15-77, C = GB, ST = London, L = London, O = University College London, CN = discovery.ucl.ac.uk
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 jurisdictionC = GB, businessCategory = Government Entity, serialNumber = November-15-77, C = GB, ST = London, L = London, O = University College London, CN = discovery.ucl.ac.uk
verify error:num=21:unable to verify the first certificate
verify return:1
140212799304832:error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small:../ssl/statem/statem_clnt.c:2150:
---
Certificate chain
 0 s:jurisdictionC = GB, businessCategory = Government Entity, serialNumber = November-15-77, C = GB, ST = London, L = London, O = University College London, CN = discovery.ucl.ac.uk
   i:C = BM, O = QuoVadis Limited, CN = QuoVadis EV SSL ICA G3
-----BEGIN CERTIFICATE-----
MIIH4DCCBcigAwIBAgIUWsWTIuklFQIki5zk7SzvkyYF4MswDQYJKoZIhvcNAQEL
BQAwSTELMAkGA1UEBhMCQk0xGTAXBgNVBAoMEFF1b1ZhZGlzIExpbWl0ZWQxHzAd
BgNVBAMMFlF1b1ZhZGlzIEVWIFNTTCBJQ0EgRzMwHhcNMTkwOTExMTAyNDExWhcN
MjEwOTExMTAzNDAwWjCBuzETMBEGCysGAQQBgjc8AgEDEwJHQjEaMBgGA1UEDwwR
R292ZXJubWVudCBFbnRpdHkxFzAVBgNVBAUTDk5vdmVtYmVyLTE1LTc3MQswCQYD
VQQGEwJHQjEPMA0GA1UECAwGTG9uZG9uMQ8wDQYDVQQHDAZMb25kb24xIjAgBgNV
BAoMGVVuaXZlcnNpdHkgQ29sbGVnZSBMb25kb24xHDAaBgNVBAMME2Rpc2NvdmVy
eS51Y2wuYWMudWswggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCvh4j4
ub+jjytAuayjz1jXpFooMEgg09Opvruzy1Vkz8KT7VYFurfQpp7xO0kDJV9bz4U6
vVUmqd9R2NaJDs0TtpKjyDFwNq1XR2+39L6JlJuIxdGRUMNLh1geNfBB7QJHac0I
xwstH/mXU9H4eU1JyS8TuVnpCbDZnSqCaQ08hl4137FGrloSL+EHqErzrmz8NzNd
725EKSG1/XP8d8O1FJDaAyvES2JfJWuhrcwa6WPPQdCu2cI4GzMRzPes3aD+IjJl
8tGVep5ketM+Kgsrn9tjiZhFcSOcxO0apRAAAYOA6NBoZvPCLr16CGQSJM/0e2N2
PM/PUh14db39Me79AgMBAAGjggNLMIIDRzAMBgNVHRMBAf8EAjAAMB8GA1UdIwQY
MBaAFOWEVNCQSZ84uvLJ4SoIxU6foEg/MHgGCCsGAQUFBwEBBGwwajA5BggrBgEF
BQcwAoYtaHR0cDovL3RydXN0LnF1b3ZhZGlzZ2xvYmFsLmNvbS9xdmV2c3NsZzMu
Y3J0MC0GCCsGAQUFBzABhiFodHRwOi8vZXYub2NzcC5xdW92YWRpc2dsb2JhbC5j
b20wMQYDVR0RBCowKIITZGlzY292ZXJ5LnVjbC5hYy51a4IRZXByaW50cy51Y2wu
YWMudWswWgYDVR0gBFMwUTBGBgwrBgEEAb5YAAJkAQIwNjA0BggrBgEFBQcCARYo
aHR0cDovL3d3dy5xdW92YWRpc2dsb2JhbC5jb20vcmVwb3NpdG9yeTAHBgVngQwB
ATAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwPAYDVR0fBDUwMzAxoC+g
LYYraHR0cDovL2NybC5xdW92YWRpc2dsb2JhbC5jb20vcXZldnNzbGczLmNybDAd
BgNVHQ4EFgQU0+IV/WaITVrZeCsIddZvFZSkuUswDgYDVR0PAQH/BAQDAgWgMIIB
fwYKKwYBBAHWeQIEAgSCAW8EggFrAWkAdgC72d+8H4pxtZOUI5eqkntHOFeVCqtS
6BqQlmQ2jh7RhQAAAW0f40mRAAAEAwBHMEUCIQDYLCvmTrDxh16qE30yqTirA3A+
Xv6TZlpUssZxI+ApqgIgSGicwtcECtcjsSnKmEwUVv6hQnvksG7dH5AqPZ7jbQ0A
dwBWFAaaL9fC7NP14b1Esj7HRna5vJkRXMDvlJhV1onQ3QAAAW0f40m4AAAEAwBI
MEYCIQCPhcwTIoiYCt6Esw49b7bcvRyREX29fRuaX36wJxQ6TAIhAJyPt8r3g++L
xWdb/sWRfF7Jn4zlyA7iUWFTF84dwK5xAHYAb1N2rDHwMRnYmQCkURX/dxUcEdkC
wQApBo2yCJo32RMAAAFtH+NKoAAABAMARzBFAiB/85erYq3OelUTEYpdJdIK//2N
AUG6EtuDCR/UspBmnQIhANbyKv+L+b02o5YIRqRKJ49LJEyJFyRxHrRM8lH9qRk8
MA0GCSqGSIb3DQEBCwUAA4ICAQAjJurMYSd9KFvcOcMZNO1DLsKytJ3N6SIkHXph
J2fpXD4sfBHxxG37r7a3hWi7vqNb4PTL8VIixKw+u/Si0tknJIyHsVf64eI4tfMD
kPDJGxMgr9qEsNukwVXgsnerqXYQRAcgYsnMLEdrgo+7SW7caTnm/adfqrc6r9Ar
4fHRidr9p7RuEM/eRCCmBqswHI7hpsE6miKLh1aXqF6I6JiSCApz3X7mJ4OiLVFN
GKw8rZHGEJUsLQBWIW0qZPjrzNG3M/LF5chVhS9D7HcUtXEFP7smNPdNGgbVTtfY
3+sXpFFdhED5ooRJCkX2/JfylXN3LT8v0iNI04HNQ1/fS27k9Q5QBahEBsvSzh88
OdHP/2jyyQwiGqNH9Q+UGGrYBW50OJB13ztobAeEWITPwI40nf3wU3qoCvM/nvJu
8kO0lD3kD4AyLqWnOYvwgjCzgVe2zuLI9F/BZiZnmXaiJq2SSzgTmIzv/HB0zSHF
BWQpgZpacZok7AhZ3vzpbOdJfgcSOCe/W6+drLyA5wTzV3m4+taU5eKvnI9NN5Xb
iUHXmqLElHVZYakpDAJkT20Uud5uIGHGwiHFYvyHgHlNBxa77Bn2gYxKtH95y3o/
C0SaHauNL7ghuyZVxNRWsKcVWlZ+1/TrOlEp00nTFyoWqxbFgwVP9WarCRDX/rZ/
Yzr/sQ==
-----END CERTIFICATE-----
---
Server certificate
subject=jurisdictionC = GB, businessCategory = Government Entity, serialNumber = November-15-77, C = GB, ST = London, L = London, O = University College London, CN = discovery.ucl.ac.uk

issuer=C = BM, O = QuoVadis Limited, CN = QuoVadis EV SSL ICA G3

---
No client certificate CA names sent
---
SSL handshake has read 2653 bytes and written 318 bytes
Verification error: unable to verify the first certificate
---
New, (NONE), Cipher is (NONE)
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 0BEE74506F0378851356FE55F7EA41ACE0E5C5C065C19C8EE24F5A1607BAD1FC
    Session-ID-ctx: 
    Master-Key: 
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1578589105
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
---

Relevant for us here is the part after Certificate chain. It shows only one certificate.

Feeding that -----BEGIN CERTIFICATE----- block through openssl x509 -text -noout presents the certificate in a more readable form:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            5a:c5:93:22:e9:25:15:02:24:8b:9c:e4:ed:2c:ef:93:26:05:e0:cb
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = BM, O = QuoVadis Limited, CN = QuoVadis EV SSL ICA G3
        Validity
            Not Before: Sep 11 10:24:11 2019 GMT
            Not After : Sep 11 10:34:00 2021 GMT
        Subject: jurisdictionC = GB, businessCategory = Government Entity, serialNumber = November-15-77, C = GB, ST = London, L = London, O = University College London, CN = discovery.ucl.ac.uk
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:af:87:88:f8:b9:bf:a3:8f:2b:40:b9:ac:a3:cf:
                    58:d7:a4:5a:28:30:48:20:d3:d3:a9:be:bb:b3:cb:
                    55:64:cf:c2:93:ed:56:05:ba:b7:d0:a6:9e:f1:3b:
                    49:03:25:5f:5b:cf:85:3a:bd:55:26:a9:df:51:d8:
                    d6:89:0e:cd:13:b6:92:a3:c8:31:70:36:ad:57:47:
                    6f:b7:f4:be:89:94:9b:88:c5:d1:91:50:c3:4b:87:
                    58:1e:35:f0:41:ed:02:47:69:cd:08:c7:0b:2d:1f:
                    f9:97:53:d1:f8:79:4d:49:c9:2f:13:b9:59:e9:09:
                    b0:d9:9d:2a:82:69:0d:3c:86:5e:35:df:b1:46:ae:
                    5a:12:2f:e1:07:a8:4a:f3:ae:6c:fc:37:33:5d:ef:
                    6e:44:29:21:b5:fd:73:fc:77:c3:b5:14:90:da:03:
                    2b:c4:4b:62:5f:25:6b:a1:ad:cc:1a:e9:63:cf:41:
                    d0:ae:d9:c2:38:1b:33:11:cc:f7:ac:dd:a0:fe:22:
                    32:65:f2:d1:95:7a:9e:64:7a:d3:3e:2a:0b:2b:9f:
                    db:63:89:98:45:71:23:9c:c4:ed:1a:a5:10:00:01:
                    83:80:e8:d0:68:66:f3:c2:2e:bd:7a:08:64:12:24:
                    cf:f4:7b:63:76:3c:cf:cf:52:1d:78:75:bd:fd:31:
                    ee:fd
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier: 
                keyid:E5:84:54:D0:90:49:9F:38:BA:F2:C9:E1:2A:08:C5:4E:9F:A0:48:3F

            Authority Information Access: 
                CA Issuers - URI:http://trust.quovadisglobal.com/qvevsslg3.crt
                OCSP - URI:http://ev.ocsp.quovadisglobal.com

            X509v3 Subject Alternative Name: 
                DNS:discovery.ucl.ac.uk, DNS:eprints.ucl.ac.uk
            X509v3 Certificate Policies: 
                Policy: 1.3.6.1.4.1.8024.0.2.100.1.2
                  CPS: http://www.quovadisglobal.com/repository
                Policy: 2.23.140.1.1

            X509v3 Extended Key Usage: 
                TLS Web Client Authentication, TLS Web Server Authentication
            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:http://crl.quovadisglobal.com/qvevsslg3.crl

            X509v3 Subject Key Identifier: 
                D3:E2:15:FD:66:88:4D:5A:D9:78:2B:08:75:D6:6F:15:94:A4:B9:4B
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            CT Precertificate SCTs: 
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : BB:D9:DF:BC:1F:8A:71:B5:93:94:23:97:AA:92:7B:47:
                                38:57:95:0A:AB:52:E8:1A:90:96:64:36:8E:1E:D1:85
                    Timestamp : Sep 11 10:34:12.241 2019 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:21:00:D8:2C:2B:E6:4E:B0:F1:87:5E:AA:13:
                                7D:32:A9:38:AB:03:70:3E:5E:FE:93:66:5A:54:B2:C6:
                                71:23:E0:29:AA:02:20:48:68:9C:C2:D7:04:0A:D7:23:
                                B1:29:CA:98:4C:14:56:FE:A1:42:7B:E4:B0:6E:DD:1F:
                                90:2A:3D:9E:E3:6D:0D
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 56:14:06:9A:2F:D7:C2:EC:D3:F5:E1:BD:44:B2:3E:C7:
                                46:76:B9:BC:99:11:5C:C0:EF:94:98:55:D6:89:D0:DD
                    Timestamp : Sep 11 10:34:12.280 2019 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:8F:85:CC:13:22:88:98:0A:DE:84:B3:
                                0E:3D:6F:B6:DC:BD:1C:91:11:7D:BD:7D:1B:9A:5F:7E:
                                B0:27:14:3A:4C:02:21:00:9C:8F:B7:CA:F7:83:EF:8B:
                                C5:67:5B:FE:C5:91:7C:5E:C9:9F:8C:E5:C8:0E:E2:51:
                                61:53:17:CE:1D:C0:AE:71
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 6F:53:76:AC:31:F0:31:19:D8:99:00:A4:51:15:FF:77:
                                15:1C:11:D9:02:C1:00:29:06:8D:B2:08:9A:37:D9:13
                    Timestamp : Sep 11 10:34:12.512 2019 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:20:7F:F3:97:AB:62:AD:CE:7A:55:13:11:8A:
                                5D:25:D2:0A:FF:FD:8D:01:41:BA:12:DB:83:09:1F:D4:
                                B2:90:66:9D:02:21:00:D6:F2:2A:FF:8B:F9:BD:36:A3:
                                96:08:46:A4:4A:27:8F:4B:24:4C:89:17:24:71:1E:B4:
                                4C:F2:51:FD:A9:19:3C
    Signature Algorithm: sha256WithRSAEncryption
         23:26:ea:cc:61:27:7d:28:5b:dc:39:c3:19:34:ed:43:2e:c2:
         b2:b4:9d:cd:e9:22:24:1d:7a:61:27:67:e9:5c:3e:2c:7c:11:
         f1:c4:6d:fb:af:b6:b7:85:68:bb:be:a3:5b:e0:f4:cb:f1:52:
         22:c4:ac:3e:bb:f4:a2:d2:d9:27:24:8c:87:b1:57:fa:e1:e2:
         38:b5:f3:03:90:f0:c9:1b:13:20:af:da:84:b0:db:a4:c1:55:
         e0:b2:77:ab:a9:76:10:44:07:20:62:c9:cc:2c:47:6b:82:8f:
         bb:49:6e:dc:69:39:e6:fd:a7:5f:aa:b7:3a:af:d0:2b:e1:f1:
         d1:89:da:fd:a7:b4:6e:10:cf:de:44:20:a6:06:ab:30:1c:8e:
         e1:a6:c1:3a:9a:22:8b:87:56:97:a8:5e:88:e8:98:92:08:0a:
         73:dd:7e:e6:27:83:a2:2d:51:4d:18:ac:3c:ad:91:c6:10:95:
         2c:2d:00:56:21:6d:2a:64:f8:eb:cc:d1:b7:33:f2:c5:e5:c8:
         55:85:2f:43:ec:77:14:b5:71:05:3f:bb:26:34:f7:4d:1a:06:
         d5:4e:d7:d8:df:eb:17:a4:51:5d:84:40:f9:a2:84:49:0a:45:
         f6:fc:97:f2:95:73:77:2d:3f:2f:d2:23:48:d3:81:cd:43:5f:
         df:4b:6e:e4:f5:0e:50:05:a8:44:06:cb:d2:ce:1f:3c:39:d1:
         cf:ff:68:f2:c9:0c:22:1a:a3:47:f5:0f:94:18:6a:d8:05:6e:
         74:38:90:75:df:3b:68:6c:07:84:58:84:cf:c0:8e:34:9d:fd:
         f0:53:7a:a8:0a:f3:3f:9e:f2:6e:f2:43:b4:94:3d:e4:0f:80:
         32:2e:a5:a7:39:8b:f0:82:30:b3:81:57:b6:ce:e2:c8:f4:5f:
         c1:66:26:67:99:76:a2:26:ad:92:4b:38:13:98:8c:ef:fc:70:
         74:cd:21:c5:05:64:29:81:9a:5a:71:9a:24:ec:08:59:de:fc:
         e9:6c:e7:49:7e:07:12:38:27:bf:5b:af:9d:ac:bc:80:e7:04:
         f3:57:79:b8:fa:d6:94:e5:e2:af:9c:8f:4d:37:95:db:89:41:
         d7:9a:a2:c4:94:75:59:61:a9:29:0c:02:64:4f:6d:14:b9:de:
         6e:20:61:c6:c2:21:c5:62:fc:87:80:79:4d:07:16:bb:ec:19:
         f6:81:8c:4a:b4:7f:79:cb:7a:3f:0b:44:9a:1d:ab:8d:2f:b8:
         21:bb:26:55:c4:d4:56:b0:a7:15:5a:56:7e:d7:f4:eb:3a:51:
         29:d3:49:d3:17:2a:16:ab:16:c5:83:05:4f:f5:66:ab:09:10:
         d7:fe:b6:7f:63:3a:ff:b1

Particularly relevant are these lines:

Issuer: C = BM, O = QuoVadis Limited, CN = QuoVadis EV SSL ICA G3
Subject: jurisdictionC = GB  [...]  CN = discovery.ucl.ac.uk

This shows that the provided certificate is a leaf certificate, for discovery.ucl.ac.uk, and that it is signed by some certificate (or rather entity) named QuoVadis EV SSL ICA G3. It will become clear later that this is not a root certificate (for now, the lack of CA in the name is a hint; and ICA commonly means intermediate certificate authority).

The certificate @little_dog suggested you download is the missing intermediate certificate (NOT the root certificate!). You can see that from the following lines in his answer:

Issuer: C = BM, O = QuoVadis Limited, CN = QuoVadis Root CA 2 G3
Subject: C = BM, O = QuoVadis Limited, CN = QuoVadis EV SSL ICA G3

That certificate is the QuoVadis EV SSL ICA G3 referenced by the leaf certificate above! But this certificate is not a root certificate. Root certificates are signed by themselves, but this certificate is signed by QuoVadis Root CA 2 G3. Which, by the way, has CA in its name.

So, where do we get the root certificate? Ideally, it should be in your browser or OS. For Debian at least (and probably Ubuntu as well), we can check with this monstrosity:

% awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt | grep 'QuoVadis Root CA 2 G3'
subject=C = BM, O = QuoVadis Limited, CN = QuoVadis Root CA 2 G3

The first part of the command produces the certificate subjects ("names") of all system-trusted CA certificates, which we then search for the relevant QuoVadis root certificate. On my system it finds this, so the root certificate is present.

To recap

  • Root cert QuoVadis Root CA 2 G3 (on your system)
    • signs intermediate cert QuoVadis EV SSL ICA G3 (missing)
      • signs leaf cert discovery.ucl.ac.uk (provided by web server)

Where should the intermediate cert come from? The answer to that is simple: the web server should provide it as well. Then the client can check the whole chain up until the root certificate (which comes from its trust store).

Getting it fixed

@little_dog's answer had you download the intermediate, and install that in your trust store, effectively turning that intermediate into a root cert for your system. That will work for this particular problem, for now, but there are drawbacks:

  • It will only solve this very particular problem on your particular machine. Download from another misconfigured web server? Same problem. Download from this site on another machine? Same problem.
  • Intermediates are usually shorter-lived than root certs. At some point in the future, your manually-installed intermediate will expire, and then it will stop working.
  • Intermediates are there for a reason. In the case of a CA compromise, intermediates can be compromised as well. A CA will then revoke those intermediates, and create new ones and re-issue leaf certs. But because you manually trusted your intermediate, it won't be revoked and your system might end up trusting servers it shouldn't.

The real solution is getting the website fixed. Try reporting it to the discovery.ucl.ac.uk webmasters. Any decent web server admin should know exactly what's up when you report to them that the webserver isn't serving the intermediate CA certificate. If they need more information, this answer has plenty :)

There are also dozens of online services that will check any web server you specify and report a list of potential security issues and configuration problems. I tried a handful, and they all complained about the missing intermediate certificate. A few popular ones include:

But it worked in Chrome?

The story becomes more complicated here. There's a mechanism called Authority Information Access (AIA) that allows HTTP clients to query the CA for the intermediate certificate. You can see URLs provided for it in the textual certificate output earlier in this answer.

But not every client implements AIA fetching. Internet Explorer and Safari do. Chrome relies on the OS to do this (so yes on some platforms, no on others). Android does not. Firefox does not, because of privacy concerns. Curl and wget do not, as far as I can tell.

Complicating things further, browsers can cache intermediate certificates they encountered, so if you visit a website that correctly sends the QuoVadis EV SSL ICA G3 intermediate with your browser, that certificate may be cached, and then website that otherwise wouldn't work suddenly would. Finally, browsers/OSes could come with (some) intermediate certificates pre-loaded, which would also hide this issue. At least Firefox is exploring this option.

None of these things can be relied on though; plenty of clients don't do AIA fetching or pre-loading. So until these mechanisms become mandatory and universally supported, web servers will still need to include all the certificates to complete the chain.

12
  • 1
    This is an excellent explanation of the underlying problem. I therefore decided to make this the accepted answer, although @little_dog s solution solved my immediate problem. I'll write to the discovery.ucl.ac.uk team.
    – nevrome
    Commented Dec 30, 2019 at 23:03
  • 4
    The browsers might also be downloading the intermediate cert themselves from the URL in the Authority Information Access extension. Commented Dec 31, 2019 at 0:28
  • 1
    s_client always displays only leaf cert under Server certificate -- even when server sends chain as it should. To see if server is sending chain correctly specify -showcerts and look under Certificate chain instead. Also many servers today -- though not this one -- only send correct certs if you send SNI, and OpenSSL below 1.1.1 does not send SNI by default so you must specify -servername $host. Commented Dec 31, 2019 at 4:38
  • 1
    @GordonDavisson Yes, it seems likely that this is what is happening here. Chrome relies on the OS to perform certificate validation, and most OSes do AIA fetching. research.google/pubs/pub46359.pdf (page 7) Firefox does not, for performance and privacy reasons. wiki.mozilla.org/Security/CryptoEngineering/…
    – Miles
    Commented Dec 31, 2019 at 9:05
  • 2
    @nevrome This answer is a great in-depth explanation. For writing to the team that owns the server, referring them something like ssltest is a convenient way to point out issues, which shows the incomplete chain problem.
    – Miles
    Commented Dec 31, 2019 at 9:10
6

UPDATE

As @marcelm pointed out, the answer below is not correct. [you could say it's dirty walkaround of a problem]

The [discovery.ucl.ac.uk] server was not sending intermediate certificate authority, which resulted in a incomplete certification chaine. By my mistake, answer below is to download the missing intermediate certificate authority, not root ca [which by the way is already in the system].

Also as @marcelm pointed out - this is server config error, and adding intermediate certificate authority to server trust store is wrong way.


From here you can download the required CA.

Convert to pem:

openssl x509 -inform der -in qvevsslg3.cer -out qvevsslg3.pem.cer

and you have:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            52:4f:c1:f1:6e:34:d1:70:2b:84:a1:3f:b0:42:bb:cc:7c:3c:90:32
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = BM, O = QuoVadis Limited, CN = QuoVadis Root CA 2 G3
        Validity
            Not Before: Nov 30 16:21:01 2016 GMT
            Not After : Nov 30 16:21:01 2026 GMT
        Subject: C = BM, O = QuoVadis Limited, CN = QuoVadis EV SSL ICA G3
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:a2:7c:9c:ce:6c:11:4a:7c:39:06:15:88:ad:f7:
                    03:d7:d1:1b:13:b6:d8:bb:97:45:b4:3f:28:ff:93:
                    6d:40:25:be:5d:da:0b:5f:e5:fd:b8:d4:ee:be:73:
                    f0:f4:7b:f6:45:d8:58:ac:98:1e:2e:01:76:c1:9e:
                    76:5c:c1:a4:9d:17:41:f2:d4:18:ae:49:c0:97:f8:
                    f4:37:97:fc:0c:3f:36:c4:fd:9e:06:40:a9:20:1d:
                    14:10:3e:33:35:5c:30:59:c7:56:bc:15:20:34:47:
                    4d:a1:bc:fd:bd:02:9b:ec:1e:4b:95:b5:e6:f6:46:
                    8e:bb:16:20:72:ff:16:b0:d3:22:bd:23:f7:9a:42:
                    52:84:43:a1:e1:16:77:65:d0:4c:fe:fb:49:ca:eb:
                    d4:c8:43:e3:ca:24:b3:7c:df:78:b5:91:f1:fc:7d:
                    7d:e1:2e:03:54:03:e0:13:b9:f2:dc:84:b3:37:e0:
                    1a:de:48:f2:2d:e6:cf:fe:c3:f3:23:50:18:d0:35:
                    b1:f1:88:38:49:31:b5:8f:43:c9:7a:e4:db:e8:08:
                    28:da:49:b1:e0:aa:a7:e2:ae:24:48:f2:fc:0a:02:
                    13:60:78:8a:68:9c:7a:0e:df:10:f7:48:9e:27:bc:
                    b0:1b:83:1b:fd:80:03:38:89:66:a1:76:7a:91:78:
                    73:18:0c:72:59:71:60:a2:52:db:e1:44:e8:22:1e:
                    94:eb:ab:f7:23:2a:be:81:7c:82:78:c5:c6:4f:89:
                    d4:82:cd:fe:3d:db:b3:39:e8:bd:eb:af:23:78:a4:
                    1a:a1:4e:5d:ec:b8:c9:50:bf:99:1f:6f:98:d5:b3:
                    e6:30:0c:a9:1c:52:d9:af:2c:e2:3b:30:b9:91:1a:
                    38:4d:a9:a0:01:fd:cb:1c:7a:f6:0b:bc:88:52:ea:
                    3e:6a:f9:6f:dd:c8:9c:dc:d5:28:75:c7:cc:8b:b0:
                    31:39:01:4b:6f:7d:82:b4:3a:03:79:56:f9:bf:7c:
                    7c:f1:1d:2d:20:42:53:8b:39:3a:33:50:7f:d2:91:
                    ad:66:21:5f:5e:da:cd:55:f1:e6:11:8b:d8:da:b3:
                    8b:6e:05:8a:33:cd:f5:ca:4a:99:49:81:d4:a6:2a:
                    a0:9d:a3:49:6f:84:2c:f1:67:31:a9:4c:35:c6:48:
                    ba:e1:6c:22:3d:c9:54:4a:7e:57:80:63:08:c3:14:
                    83:1a:35:08:24:72:91:af:38:10:f6:59:de:1c:e1:
                    d5:6f:ca:57:1c:d1:64:74:10:c7:4d:bd:4a:36:60:
                    ce:c8:bb:20:de:ad:0b:24:fe:8f:de:7c:d3:fd:a5:
                    83:02:3d:e0:96:92:6f:19:0e:5d:92:30:1b:8f:1f:
                    8d:16:4f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Certificate Policies:
                Policy: 1.3.6.1.4.1.8024.0.2.100.1.2
                  CPS: http://www.quovadisglobal.com/repository

            Authority Information Access:
                OCSP - URI:http://ocsp.quovadisglobal.com
                CA Issuers - URI:http://trust.quovadisglobal.com/qvrca2g3.crt

            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication, OCSP Signing
            X509v3 Authority Key Identifier:
                keyid:ED:E7:6F:76:5A:BF:60:EC:49:5B:C6:A5:77:BB:72:16:71:9B:C4:3D

            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://crl.quovadisglobal.com/qvrca2g3.crl

            X509v3 Subject Key Identifier:
                E5:84:54:D0:90:49:9F:38:BA:F2:C9:E1:2A:08:C5:4E:9F:A0:48:3F
    Signature Algorithm: sha256WithRSAEncryption
         63:f1:07:59:ba:4d:c0:28:06:55:0d:41:ed:c2:27:e7:7d:27:
         e9:19:e1:b7:ca:f2:37:a1:38:4a:2b:36:18:6d:3d:f3:4d:6c:
         78:25:4c:a0:65:01:d6:e0:42:5a:6e:ee:a9:0a:b2:76:34:db:
         8c:b8:6e:b9:be:e1:4e:34:89:d4:9d:f4:48:e2:b5:09:96:e4:
         f9:cd:55:3f:d0:dc:8d:8e:3a:87:27:32:b5:42:90:d9:66:e0:
         91:a6:3e:97:fb:59:34:32:ae:d1:d8:dc:da:10:39:6a:99:63:
         40:29:a2:23:37:24:6d:c1:eb:eb:24:67:14:ae:d9:3e:34:8f:
         08:05:0a:9b:6d:03:bc:e4:50:ee:1b:08:6a:89:ac:22:5a:97:
         9b:90:4f:b2:c3:1c:c6:32:38:f0:4c:e0:bf:fb:3c:ca:70:12:
         23:c4:b9:3f:6b:ce:9c:1a:34:f2:c2:41:33:f6:b4:29:bb:b0:
         df:9d:52:b9:b4:f3:8f:11:be:a3:54:87:7b:a9:40:ce:f2:10:
         32:e4:b0:c5:47:1a:f1:89:22:07:5f:70:e5:86:6d:f9:2b:36:
         25:73:5c:e8:5d:a3:67:0a:6b:d2:1b:68:21:77:be:37:df:f1:
         d0:2a:21:61:14:5a:f8:88:af:44:68:1d:0d:07:37:c3:63:fe:
         a5:f7:cd:40:ff:ea:74:fb:94:63:23:24:61:68:ae:1c:7f:d8:
         bf:05:f2:b8:3c:6f:c8:64:1f:bc:a9:87:af:5b:aa:fe:a8:aa:
         6c:ad:5b:0d:25:28:12:ae:12:bb:cc:97:f1:8a:05:f5:3b:b1:
         62:b6:88:a6:9f:62:12:b3:b9:ad:aa:c3:3b:a1:93:35:51:e1:
         d4:e5:c0:27:f7:8f:84:e5:b3:aa:8d:df:94:b4:e5:01:d4:dc:
         b3:73:2a:f7:b9:0a:5b:c5:d6:0a:7b:bf:72:32:49:82:57:f6:
         cd:57:cb:02:5b:fd:e6:9e:7a:07:d2:1f:d2:95:db:37:be:2a:
         0e:46:04:0b:c4:dd:2c:ec:2b:ca:17:2f:f3:2c:a2:9a:1f:74:
         fc:0b:d6:f4:ba:41:ee:cc:24:5a:75:14:60:d4:de:a7:f5:cc:
         5f:f4:4b:a4:72:e7:24:e5:6d:9d:1a:67:dd:ca:15:7d:24:7f:
         d2:bc:f4:5c:a5:57:79:91:a7:2b:3b:46:74:83:10:85:63:13:
         c6:f6:75:52:99:91:00:7d:80:6f:64:27:56:8d:5f:90:f5:72:
         a8:d4:89:71:eb:39:63:f5:4a:a4:8b:cb:06:4e:49:8f:9e:5f:
         bc:af:0c:13:ff:40:49:af:8b:b4:ba:c8:9a:cf:22:60:79:7b:
         e5:cb:a9:b9:86:59:96:0f
-----BEGIN CERTIFICATE-----
MIIGuDCCBKCgAwIBAgIUUk/B8W400XArhKE/sEK7zHw8kDIwDQYJKoZIhvcNAQEL
BQAwSDELMAkGA1UEBhMCQk0xGTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxHjAc
BgNVBAMTFVF1b1ZhZGlzIFJvb3QgQ0EgMiBHMzAeFw0xNjExMzAxNjIxMDFaFw0y
NjExMzAxNjIxMDFaMEkxCzAJBgNVBAYTAkJNMRkwFwYDVQQKDBBRdW9WYWRpcyBM
aW1pdGVkMR8wHQYDVQQDDBZRdW9WYWRpcyBFViBTU0wgSUNBIEczMIICIjANBgkq
hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAonyczmwRSnw5BhWIrfcD19EbE7bYu5dF
tD8o/5NtQCW+XdoLX+X9uNTuvnPw9Hv2RdhYrJgeLgF2wZ52XMGknRdB8tQYrknA
l/j0N5f8DD82xP2eBkCpIB0UED4zNVwwWcdWvBUgNEdNobz9vQKb7B5LlbXm9kaO
uxYgcv8WsNMivSP3mkJShEOh4RZ3ZdBM/vtJyuvUyEPjyiSzfN94tZHx/H194S4D
VAPgE7ny3ISzN+Aa3kjyLebP/sPzI1AY0DWx8Yg4STG1j0PJeuTb6Ago2kmx4Kqn
4q4kSPL8CgITYHiKaJx6Dt8Q90ieJ7ywG4Mb/YADOIlmoXZ6kXhzGAxyWXFgolLb
4UToIh6U66v3Iyq+gXyCeMXGT4nUgs3+PduzOei9668jeKQaoU5d7LjJUL+ZH2+Y
1bPmMAypHFLZryziOzC5kRo4TamgAf3LHHr2C7yIUuo+avlv3cic3NUodcfMi7Ax
OQFLb32CtDoDeVb5v3x88R0tIEJTizk6M1B/0pGtZiFfXtrNVfHmEYvY2rOLbgWK
M831ykqZSYHUpiqgnaNJb4Qs8WcxqUw1xki64WwiPclUSn5XgGMIwxSDGjUIJHKR
rzgQ9lneHOHVb8pXHNFkdBDHTb1KNmDOyLsg3q0LJP6P3nzT/aWDAj3glpJvGQ5d
kjAbjx+NFk8CAwEAAaOCAZcwggGTMBIGA1UdEwEB/wQIMAYBAf8CAQAwUQYDVR0g
BEowSDBGBgwrBgEEAb5YAAJkAQIwNjA0BggrBgEFBQcCARYoaHR0cDovL3d3dy5x
dW92YWRpc2dsb2JhbC5jb20vcmVwb3NpdG9yeTB0BggrBgEFBQcBAQRoMGYwKgYI
KwYBBQUHMAGGHmh0dHA6Ly9vY3NwLnF1b3ZhZGlzZ2xvYmFsLmNvbTA4BggrBgEF
BQcwAoYsaHR0cDovL3RydXN0LnF1b3ZhZGlzZ2xvYmFsLmNvbS9xdnJjYTJnMy5j
cnQwDgYDVR0PAQH/BAQDAgEGMCcGA1UdJQQgMB4GCCsGAQUFBwMBBggrBgEFBQcD
AgYIKwYBBQUHAwkwHwYDVR0jBBgwFoAU7edvdlq/YOxJW8ald7tyFnGbxD0wOwYD
VR0fBDQwMjAwoC6gLIYqaHR0cDovL2NybC5xdW92YWRpc2dsb2JhbC5jb20vcXZy
Y2EyZzMuY3JsMB0GA1UdDgQWBBTlhFTQkEmfOLryyeEqCMVOn6BIPzANBgkqhkiG
9w0BAQsFAAOCAgEAY/EHWbpNwCgGVQ1B7cIn530n6Rnht8ryN6E4Sis2GG09801s
eCVMoGUB1uBCWm7uqQqydjTbjLhuub7hTjSJ1J30SOK1CZbk+c1VP9DcjY46hycy
tUKQ2WbgkaY+l/tZNDKu0djc2hA5apljQCmiIzckbcHr6yRnFK7ZPjSPCAUKm20D
vORQ7hsIaomsIlqXm5BPssMcxjI48Ezgv/s8ynASI8S5P2vOnBo08sJBM/a0Kbuw
351SubTzjxG+o1SHe6lAzvIQMuSwxUca8YkiB19w5YZt+Ss2JXNc6F2jZwpr0hto
IXe+N9/x0CohYRRa+IivRGgdDQc3w2P+pffNQP/qdPuUYyMkYWiuHH/YvwXyuDxv
yGQfvKmHr1uq/qiqbK1bDSUoEq4Su8yX8YoF9TuxYraIpp9iErO5rarDO6GTNVHh
1OXAJ/ePhOWzqo3flLTlAdTcs3Mq97kKW8XWCnu/cjJJglf2zVfLAlv95p56B9If
0pXbN74qDkYEC8TdLOwryhcv8yyimh90/AvW9LpB7swkWnUUYNTep/XMX/RLpHLn
JOVtnRpn3coVfSR/0rz0XKVXeZGnKztGdIMQhWMTxvZ1UpmRAH2Ab2QnVo1fkPVy
qNSJces5Y/VKpIvLBk5Jj55fvK8ME/9ASa+LtLrIms8iYHl75cupuYZZlg8=
-----END CERTIFICATE-----

with above cert you can connect by:

curl https://discovery.ucl.ac.uk/1575442/1/Palmisanoetal.zip --cacert qvevsslg3.pem.cer

or by adding it to the trusted root certificates on the server, like you said you did; remember you need to:

sudo update-ca-certificates

4
  • Thank you very much! That worked. I used the following sequence of commands in the end: wget --no-check-certificate https://www.quovadisglobal.com/~/media/Files/Roots/qvevsslg3.ashx && sudo openssl x509 -inform der -in qvevsslg3.ashx -out /usr/local/share/ca-certificates/qvevsslg3.crt && sudo update-ca-certificates
    – nevrome
    Commented Dec 30, 2019 at 14:00
  • 2
    @little_dog The certificate you're having him install is NOT a CA cert, it's an intermediate certificate. I would not recommend that. See also my answer.
    – marcelm
    Commented Dec 30, 2019 at 22:46
  • 3
    @nevrome You downloaded the certificate with disabled SSL verification (same as --insecure for curl), and installed the certificate in your system trust store. Honestly, it would have been safer to just use --insecure to download your original URL ;)
    – marcelm
    Commented Dec 30, 2019 at 22:47
  • 1
    @marcelm You are absolutely right, my mistake, I gave an update to my answer. [hope it's better now]
    – little_dog
    Commented Dec 31, 2019 at 9:39
1

you can tell curl to not check the certificate at all

utils::download.file(db_url, temp, method = 'curl', extra='-k')

Since you have no way of knowing you have downloaded the correct file you should probably implement something extra like a sha256sum of the downloaded file (or if the file is signed, check it's signature)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .