0

I recently wanted to switch all my internal servers over to TLSv1.3 from TLSv1.2 as they are all kept very up to date and do not need to support older devices.

My setup is:

A reverse proxy, at 192.168.20.2 (Debian 11, NGINX v1.21.6). This terminates the public valid lets encrypt certificates and will continue to use TLSv1.2 and TLSv1.3 to support a variety of clients. The requests are forwarded via the server_name to the correct backend server IP via another TLS session.

Several backend servers, but for simplicity we will just talk about one (all the configs are identical).

Backend server, at 192.168.30.2 (Debian 11, NGINX v1.21.6). This terminates another TLS session (started by the reverse proxy, using a self-signed internal certificate) at which point the web server index files are served.

My goal has been to decommission TLSv1.2 on all the backend servers, since that TLS session has both client and server controlled by me, so maintaining compatibility with new TLS versions is do-able.

I went in to the ssl-params.conf that I point out in the main site configuration and simply removed TLSv1.2 from the ssl_protocols part. Upon restart of nginx (it did restart fine, no errors), I attempt to browse to the site and get a 502 Bad Gateway.

I thought perhaps the problem was removing TLS1.2, but upon running a curl from the Reverse Proxy itself, I was able to get the web server index file served no problem. Therefore, I assume, the issue lies in the Reverse Proxy NGINX server.

There is probably some obscure TLS NGINX gotcha' that I am missing here.
I can't seem to find anything online, so hopefully someone can point out my issue?

If you are wondering why I use TLS to a backend server, it is because the connection between the reverse proxy and the backend server ride over an untrusted portion of the network, therefore I encrypt.

Here are all the relevant configs for the connect to a particular backend site.

Reverse Proxy:

# configuration file /etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 768;
        # multi_accept on;
}
http {
        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        server_tokens off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # Rate Limiting
        ##

        limit_conn_zone $binary_remote_addr zone=limit_conn:1m;
        limit_conn limit_conn 100;
        limit_req_zone $binary_remote_addr zone=req_limit_per_ip:10m rate=5r/s;

        ##
        # Gzip Settings
        ##

        gzip on;

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        client_max_body_size 10000m;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/sites-enabled/*;

        ##
        # Hardening
        ##
 
        add_header Allow "GET, POST, HEAD" always;
}


# configuration file /etc/nginx/snippets/ssl-params.conf:
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:TLS_AES_128_CCM_8_SHA256:TLS_AES_128_CCM_SHA256:ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
ssl_ecdh_curve secp384r1;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
resolver 192.168.20.1 valid=300s;
resolver_timeout 5s;

ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

# configuration file /etc/nginx/sites-enabled/wiki.domain.com:
server {
        listen 443 ssl http2;
        ssl_certificate /etc/letsencrypt/live/wiki.domain.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/wiki.domain.com/privkey.pem;
        ssl_stapling on;
        ssl_stapling_verify on;
        ssl_trusted_certificate /etc/letsencrypt/live/wiki.domain.com/chain.pem;
        include snippets/ssl-params.conf;
        server_name wiki.domain.com;
        location / {
                proxy_pass https://192.168.30.2;
                proxy_set_header X-Real-IP $remote_addr;
        }
}

Backend server:

# configuration file /etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 768;
        # multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # Logging Settings
        ##

        access_log off;
        error_log  off;

        ##
        # Gzip Settings
        ##

        gzip on;

        client_body_buffer_size 1K;
        client_header_buffer_size 1k;
        client_max_body_size 10k;
        large_client_header_buffers 2 1k;

        server_tokens off;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/sites-enabled/*;
}

# configuration file /etc/nginx/sites-enabled/dokuwiki.conf:
server {
    listen 443 ssl http2;
    include snippets/self-signed.conf;
    include snippets/ssl-params.conf;

    server_name _; 

    root         /var/www;
    index        index.html;
}

# configuration file /etc/nginx/snippets/self-signed.conf:
ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt;
ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;

# configuration file /etc/nginx/snippets/ssl-params.conf:
ssl_protocols TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:TLS_AES_128_CCM_8_SHA256:TLS_AES_128_CCM_SHA256:ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
ssl_ecdh_curve secp384r1;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
ssl_stapling on;
ssl_stapling_verify on;
resolver 192.168.30.1 valid=300s;
resolver_timeout 5s;
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
add_header Content-Security-Policy "default-src 'self';";
add_header X-XSS-Protection "1; mode=block";

ssl_dhparam /etc/ssl/certs/dhparam.pem;

I just used a default index.html for the backend server for testing.

When I attempt to browse from a client machine to the reverse proxy:

user@clientmachine:~$ curl -vvvv https://wiki.domain.com
*   Trying PUBLICIP:443...
* Connected to wiki.domain.com (PUBLICIP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=wiki.domain.com
*  start date: May 17 09:56:32 2022 GMT
*  expire date: Aug 15 09:56:31 2022 GMT
*  subjectAltName: host "wiki.domain.com" matched cert's "wiki.domain.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55619a9a25c0)
> GET / HTTP/2
> Host: wiki.domain.com
> user-agent: curl/7.74.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 502 
< server: nginx
< date: Mon, 13 Jun 2022 21:31:20 GMT
< content-type: text/html
< content-length: 150
< allow: GET, POST, HEAD
< 
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host wiki.domain.com left intact

When I attempt to browse to the backend server from the Reverse Proxy to the backend:

user@revproxy:~$ curl -kvvvv https://192.168.30.2/
*   Trying 192.168.30.2:443...
* Connected to 192.168.30.2 (192.168.30.2) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=State; L=city; O=; CN=192.168.30.2
*  start date: Dec 10 16:29:30 2021 GMT
*  expire date: Feb 26 16:29:30 2031 GMT
*  issuer: C=US; ST=State; L=city; O=; CN=192.168.30.2
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x555ab8d305c0)
> GET / HTTP/2
> Host: 192.168.30.2
> user-agent: curl/7.74.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< server: nginx
< date: Mon, 13 Jun 2022 21:33:38 GMT
< content-type: text/html
< content-length: 329
< last-modified: Mon, 13 Jun 2022 21:05:52 GMT
< etag: "62a7a6b0-149"
< strict-transport-security: max-age=63072000; includeSubdomains
< x-frame-options: DENY
< x-content-type-options: nosniff
< content-security-policy: default-src 'self';
< x-xss-protection: 1; mode=block
< accept-ranges: bytes
< 
<!DOCTYPE html>

<html>
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
  <title>Spoon-Knife</title>
  <LINK href="styles.css" rel="stylesheet" type="text/css">
</head>

<body>

<!-- Feel free to change this text here -->
<p>
  Fork me? Fork you!
</p>
<p>
  I made a change
</p>

</body>
</html>
* Connection #0 to host 192.168.30.2 left intact

Of course, if I add the TLSv1.2 back into ssl-params.conf in the ssl_protocols section on the backend server, then the client request suddenly goes through (client still using TLSv1.3 for the initial connection to the Reverse Proxy)

1 Answer 1

4

I have found the solution here: https://forum.nginx.org/read.php?11,294147

Apparently NGINX needs to be told what protocol to use for proxy pass connections.

Using proxy_ssl_protocols TLSv1.3; in the reverse proxy server block fixed my issue.

You must log in to answer this question.