NanoPi M4 - RK3399: Apache2 + SSL: Large download hangs

Question

I've a brand new setup of Apache2 running under Debian 9 (armbian on NanoPi M4). Everything works fine until I try to download a large file over HTTPS. Small files download successfully however, larger ones seem to fail randomly at some point. I can't reproduce this issue with plain HTTP. Example on wget (same behaviour on a browser):

wget https://xyz---/test.bin --no-check-certificate
--2019-01-13 18:22:22--  https://xyz---/test.bin
Resolving xyz--- (xyz---)... 85.241.xxx.xxx
Connecting to xyz--- (xyz---)|85.241.xxx.xxx|:443... connected.
WARNING: cannot verify xyz---'s certificate, issued by 'CN=Let\'s Encrypt Authority X3,O=Let\'s Encrypt,C=US':
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: 1073741824 (1,0G) [application/octet-stream]
Saving to: 'test.bin.2'

test.bin.2                      39%[==================>                               ] 403,66M  11,6MB/s    eta 54s

It was downloading fine until it reached 403,66M, after this point nothing else happened. On the server side (error.log) I get this:

ssl_engine_io.c(2135): [client 85.243.xxx.xxx:59904] OpenSSL: write 16413/16413 bytes to BIO#5588cd8c50 [mem: 5588a87c23] (BIO dump follows)
core_filters.c(525): [client 85.243.xxx.xxx:59904] core_output_filter: flushing because of THRESHOLD_MAX_BUFFER
core_filters.c(547): (70007)The timeout specified has expired: [client 85.243.xxx.xxx:59904] core_output_filter: writing data to the network
ssl_engine_io.c(2144): [client 85.243.xxx.xxx:59904] OpenSSL: I/O error, 16413 bytes expected to write on BIO#5588cd8c50 [mem: 5588a87c23]
(70007)The timeout specified has expired: [client 85.243.xxx.xxx:59904] AH01993: SSL output filter write failed.

It seems to run the core_output_filter: flushing because of THRESHOLD_MAX_BUFFER and then nothing else happens. Sometimes I've also noticed this error:

ssl_engine_io.c(2135): [client 95.239.xxx.xxx:9937] OpenSSL: write 45/45 bytes to BIO#55bd9e23e0 [mem: 55bd9ec213] (BIO dump follows)
ssl_engine_io.c(2144): [client 95.239.xxx.xxx:9937] OpenSSL: I/O error, 5 bytes expected to read on BIO#55bd9e9d80 [mem: 55bd9ec213]

Here is the configuration of the VHosts serving this requests:

<VirtualHost *:443>
    ServerName xyz---
    ServerAdmin tcb13---
    DocumentRoot /test
    ErrorLog /test/error.log
    CustomLog /test/access.log combined

    SSLEngine on
    SSLCertificateFile /mnt/SU1/letsencrypt/config/live/xyz---/fullchain.pem
    SSLCertificateKeyFile /mnt/SU1/letsencrypt/config/live/xyz---/privkey.pem
    Header always set Strict-Transport-Security "max-age=15768000"

    LogLevel trace6
</VirtualHost>

I'm sure this isn't a network related problem because:

1. Only happens with SSL enabled, on non-ssl vhosts I can download without issues;
2. Other protocols (FTP and SCP) work just fine to download the same test file;
3. No issues while testing the network with iperf3.

Some system information:

lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 9.6 (stretch)
Release:        9.6
Codename:       stretch

uname -a
Linux testxyz 4.4.162-rk3399 #41 SMP Fri Oct 26 14:03:47 CEST 2018 aarch64 GNU/Linux

apache2ctl -V | grep -i "Server version"
Server version: Apache/2.4.25 (Debian)
root@testxyz:~# dpkg -l |grep apache2
ii  apache2                         2.4.25-3+deb9u6                                       arm64        Apache HTTP Server
ii  apache2-bin                     2.4.25-3+deb9u6                                       arm64        Apache HTTP Server (modules and other binary files)
ii  apache2-data                    2.4.25-3+deb9u6                                       all          Apache HTTP Server (common files)
ii  apache2-utils                   2.4.25-3+deb9u6                                       arm64        Apache HTTP Server (utility programs for web servers)
ii  libapache2-mod-php7.3           7.3.0-2+0~20181217092659.24+stretch~1.gbp54e52f       arm64        server-side, HTML-embedded scripting language (Apache 2 module)

How can I fix this? Thank you.

Answer 1

I manage to find out that this issue is related to the integrated Ethernet and known issues with the RK3399 CPU. With an USB 3 to Ethernet adapter this issue won't happen.

The boards like RK3399 need TCP/UDP offloading disabled to avoid the retransmissions and reset errors. This was already implemented by Ayufan on Rock64 and RockPro64 Rootfs and DietPi needs this too.

An easy fix for this is to disable offloading:

ethtool -K eth0 rx off tx off

You may check offloading status afterwards with ethtool --show-offload eth0. After disabling the offloading features I don't have the reported issue anymore.

In order to survive reboots (and network restarts) you may create a script at /etc/network/if-up.d/disable-offload similar to:

#!/bin/bash
/sbin/ethtool -K eth0 rx off tx off

(Be sure to NOT name the file with .sh and to chmod +x this file as well)