1

We have server under Ubuntu 12.04 with Apache HTTP 2.2 installed there.

Kernel 3.2.0.

Faced with weird behavior during dowloading some files.

Virtualhost config:

<Directory /var/www/name/*>
   ...
      AllowOverride AuthConfig

   # add these accordingly for the MIME types to be compressed
   AddOutputFilterByType DEFLATE text/plain
   AddOutputFilterByType DEFLATE text/html
   AddOutputFilterByType DEFLATE text/xml
   AddOutputFilterByType DEFLATE text/css
   AddOutputFilterByType DEFLATE application/xml
   AddOutputFilterByType DEFLATE application/xhtml+xml
   AddOutputFilterByType DEFLATE application/rss+xml
   AddOutputFilterByType DEFLATE application/javascript
   AddOutputFilterByType DEFLATE application/x-javascript

   <Files *.gz>
    SetOutputFilter DEFLATE
    Header set Content-Encoding: gzip
    Header append Content-Encoding: deflate
   </Files>

</Directory>

Problem is - sometimes for unknown reasons it's impossible to download some (!) files: when file download 99% - speed decreased to 0 and download stops.

Nothing unusual in logs - but I have found one oddity in tcpdump (after download speed == 0) results.

For example - during download attempt of badfile.gz:

10:36:37.611369 IP (tos 0x0, ttl 64, id 7954, offset 0, flags [DF], proto TCP (6), length 1420)
    37.**.**.176.80 > 10.**.**.25.55981: Flags [.], cksum 0x00a9 (correct), seq 228803:230171, ack 197, win 243, options [nop,nop,TS val 2097666946 ecr 811530774], length 1368
10:36:37.611396 IP (tos 0x0, ttl 64, id 64391, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->933a)!)
    10.**.**.25.55981 > 37.**.**.80: Flags [.], cksum 0xac28 (incorrect -> 0xf8fc), seq 197, ack 230171, win 4053, options [nop,nop,TS val 811530824 ecr 2097666946], length 0

There is Flags [.] - so, it's hang on on the data transmission - there is no Finalize flags (afaik).

Another tcpdump example during download another file goodfile.gz(from same Apache's directory on server side):

10:39:21.216118 IP (tos 0x0, ttl 64, id 18169, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->47c9)!)
    10.**.**.25.55981 > 37.**.**.80: Flags [F.], cksum 0xac28 (incorrect -> 0x83bb), seq 0, ack 1, win 4096, options [nop,nop,TS val 811691867 ecr 2097666946], length 0

There few files with different extension/size/grants etc - but problem come up only with few of them.

So - problem appear sometimes, without any changes on server side. Sometimes badfile.gz can be dowloaded without problems - sometimes (usually) it hang up.

Same during downloading with browsers - Chrome reports "Failed - Network error", Firefox - just says "Estimate unknown" during download.

Please, let me know if I can add more info.

A few examples.

badfile first:

$ wget http://static.content.domain.net/assets/json/en-GB/content3.json.gz
...
HTTP request sent, awaiting response... 200 OK
Length: 229874 (224K) [application/x-gzip]
Saving to: 'content3.json.gz.3'

content3.json.gz.3                                  99%[==============...=====> ] 224.42K  --.-KB/s   eta 0s    

And goodfile:

$ wget http://static.content.domain.net/assets/json/en-GB/24k.tar.gz
...
HTTP request sent, awaiting response... 200 OK
Length: 24576 (24K) [application/x-gzip]
Saving to: '24k.tar.gz.1'

24k.tar.gz.1                                       100%[=========...======>]  24.00K  --.-KB/s   in 0.05s  

2015-11-25 10:38:40 (440 KB/s) - '24k.tar.gz.1' saved [24576/24576]

P.S. We have complicated enough network configuration, including VPN tunnels between offices/datacenters - may be cause somewhere here too.

P.P.S We also have very old system there:

# /usr/lib/update-notifier/apt-check --human-readable
205 packages can be updated.
154 updates are security updates.

But it can not be updated now :-)

  • Please do tell your kernel version and if using Cisco ASAs – Rui F Ribeiro Nov 25 '15 at 10:36
  • Thanks, @RuiFRibeiro kernel 3.2.0 (added to Q). Can't say about Cisco ASAs. Will check with our IT team, if it's important. – setevoy Nov 25 '15 at 10:57
  • 1
    We had several teams complaining about the same behaviour. Managed to pinpoint it to incompatibilities between ASAs and kernel 3. Solved it updating to kernel 4. – Rui F Ribeiro Nov 25 '15 at 10:59
  • @RuiFRibeiro Um... Thanks. it can be an issue cause... But I'm a bit confused - how can it affect only some files on the Apache's box? – setevoy Nov 25 '15 at 11:17
  • Cannot answer that, it not reproducible 100% of the time, and it seems to affect only the bigger files. We have here a repository of a local Linux distro, and it was a bloody mess downloading the ISO images. – Rui F Ribeiro Nov 25 '15 at 11:19
1

There is story continuation: few days ago we also faced with apt-get update problems on some our Ubuntu 12/14 boxes - it can't be completed and also hang up during lists download.

Fix was made by NOC team in Cisco ASA's threat detection settings by changing it to "scanning to passive only".

  • 1
    Thanks for the feedback! somehow the board alerts did not send me a notification of this message, now I also know how to improve this for others in the ASA side. – Rui F Ribeiro Dec 9 '15 at 9:53
  • 1
    Apparently I have here remote devs complaining where the threat detection solution that you mention does not work. Sigh. Nonetheless, they are using kernel 3, and on a lighter note, I do not know what they do have at their side (Cisco or not). So it wont be enough to cast aside that solution. – Rui F Ribeiro Dec 17 '15 at 9:22
1

We are also aware of identical issues of downloads failing from linux servers exactly as you are describing. We tracked down it here to incompatibilities between the Linux kernel 3.x and Cisco ASAs.

Please note one of the machines involved has the latest 3.x Debian kernel 3.16.0, and our ASA firmware updates are recent.

The symptoms are not reproducible 100% of the time, and it seems to affect only the bigger files, and are exacerbated when the requests come from the outside networks (Internet in our case).

We have here a repository of a local Linux distro, and it was a bloody mess downloading the ISO images from the Internet.

We tried to mitigate this with net.ipv4.tcp_sack = 0 and even experimented net.ipv4.tcp_ecn = 0, and while it mitigated the issue, did not solve them in all the servers.

The issue was solved completely migrating to the kernel 4.2.0.

I also have found out a thread describing more workarounds than the ones we tried with sysctl.

sysctl -w net.ipv4.tcp_sack=0
sysctl -w net.ipv4.tcp_congestion_control=reno
sysctl -w net.ipv4.tcp_frto=1
sysctl -w net.ipv4.tcp_early_retrans=1

They also mention a ASA configuration fix that we are not aware. I will advise that if you have Cisco equipment and have Cisco Support, to involve them.

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1388786

  • Thanks. Unfortunately - got an error: "net.ipv4.tcp_early_retrans" is an unknown key error, and other settings doesn't helped. Anyway - I'm not sure yet about Cisco in our network - waiting for replay. – setevoy Nov 25 '15 at 12:04
  • One last tidbit. We initially suspected NAT issues, however we have at least one system affected with a public address. – Rui F Ribeiro Nov 25 '15 at 13:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.