3

Lately I've experienced some problems with my server freezing or having network problems (I can't really tell which at this point). I'm running OpenSSH on port 443 and it seems fail2ban's anti SSH DDoS filter is the culprit. Because I'm running on a port normally reserved for HTTPS¹ I sometimes get invalid requests:

# /var/log/auth.log
Aug 25 11:46:38 void sshd[13782]: pam_unix(sshd:session): session closed for user tim
Aug 25 11:46:38 void sshd[13782]: syslogin_perform_logout: logout() returned an error
Aug 25 11:46:41 void sshd[15911]: Bad protocol version identification 'CONNECT 194.109.153.5:11111 HTTP/1.0' from 194.109.21.230
Aug 25 11:46:45 void sshd[15915]: Bad protocol version identification 'CONNECT 194.109.153.5:11111 HTTP/1.0' from 194.109.21.230
Aug 25 11:46:53 void sshd[15912]: Did not receive identification string from 194.109.21.230
Aug 25 11:46:53 void sshd[15910]: Did not receive identification string from 194.109.21.230
Aug 25 11:46:57 void sshd[15916]: Did not receive identification string from 194.109.21.230
Aug 25 11:46:57 void sshd[15914]: Did not receive identification string from 194.109.21.230

Which in turn triggers fail2ban:

# /var/log/fail2ban.log
2014-08-25 11:46:58,609 fail2ban.actions: WARNING [ssh-ddos] Ban 194.109.21.230
2014-08-25 11:56:58,852 fail2ban.actions: WARNING [ssh-ddos] Unban 194.109.21.230

This is not something I previously worried about, just a little noise in my logs, but didn't affect my server. The last few days however I've gotten proxyscanned multiple times by 194.109.21.230 (proxyscan.xs4all.net. The result is disconnecting, possibly the server freezing, which I notice quickly as I'm also using this server as an IRC client. Looking at my Munin graphs it looks like it froze in the same period fail2ban banned a seemingly unimportant IP:

Munin - memory

I'm not sure how this could be related, but I also recently experienced a RCU stall² on the same server. Before this date I never experienced the server freezing even if fail2ban triggered. Since then I've fixed /etc/ntp.conf, but only after experiencing the freezes.

# /var/log/syslog
... (multiple 'host name not found' for ntpd)
Aug 20 20:24:22 void ntpd_intres[8124]: host name not found: 1.tid.uio.no
Aug 20 20:40:25 void ntpd_intres[8124]: host name not found: 0.ntp.uio.no
Aug 20 20:40:25 void ntpd_intres[8124]: host name not found: 1.tid.uio.no
Aug 20 20:57:44 void kernel: [18225085.384024] INFO: rcu_sched detected stall on CPU 0 (t=18892 jiffies)
Aug 20 20:57:44 void kernel: [18225085.384024] sending NMI to all CPUs:
Aug 20 20:57:44 void kernel: [18225085.384024] NMI backtrace for cpu 0
Aug 20 20:57:44 void kernel: [18225085.384024] Modules linked in: tun iptable_nat nf_nat iptable_mangle xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_mark bridge stp xt_multiport iptable_filter ip_tables x_tables vmwgfx snd_page_alloc parport_pc parport processor thermal_sys snd_timer snd soundcore evdev pcspkr serio_raw ttm drm button i2c_piix4 i2c_core ext4 crc16 jbd2 mbcache sg sr_mod cdrom ata_generic virtio_blk virtio_net floppy ata_piix libata virtio_pci virtio_ring virtio scsi_mod [last unloaded: scsi_wait_scan]
Aug 20 20:57:44 void kernel: [18225085.384024]
Aug 20 20:57:44 void kernel: [18225085.384024] Pid: 8124, comm: ntpd Not tainted 3.2.0-4-686-pae #1 Debian 3.2.41-2+deb7u2 Bochs Bochs
... (call stack)

As for now I just added an iptables rule blocking the proxyscanning server:

iptables -A INPUT -p tcp --dport 443 -s 128.39.74.65 -j DROP

I would like to force the issue by blocking myself out to see if the same thing happens every time, and if it's fixed after a reboot. Couldn't come up with a command that actually triggers ssh-ddos, failing to authenticate or wget doesn't seem to be enough. Any tips?

Could the RCU stall(*) be related to fail2ban/freezes, and if so, why? Rebooting would probably help, but I'm curious as to how this could happen and the reason for my server freezing.

Edit: Freeze not long after the RCU stall:

# /usr/var/log/auth.log.1
Aug 20 21:09:28 void sshd[9115]: Bad protocol version identification 'CONNECT 194.109.153.5:11111 HTTP/1.0' from 194.109.21.230
Aug 20 21:09:40 void sshd[9114]: Did not receive identification string from 194.109.21.230
Aug 20 21:09:40 void sshd[9116]: Did not receive identification string from 194.109.21.230

# /usr/var/log/fail2ban.log.1
2014-08-20 21:10:28,747 fail2ban.actions: WARNING [ssh-ddos] Ban 194.109.21.230
2014-08-20 21:20:29,515 fail2ban.actions: WARNING [ssh-ddos] Unban 194.109.21.230

¹ I'm running it on 443 because not all networks I frequent allow the use of 22. It also has the added bonus of less incoming SSH attacks.
² RCU stall (CPU), not kernel panic. See comments.

  • 1
    443 for SSH seems a dreadful choice to me. Presumably the networks which don't allow 22 out have a whitelist of ports, and that's why you can't use something more obsure (e.g. 567). If so and those networks present an external static IP, you could use a more obscure port and port-forward 443->567 for just those incoming IP's (blocking all others). WRT your "kernel panic", what's in that log isn't: kernel.org/doc/Documentation/RCU/stallwarn.txt – goldilocks Aug 25 '14 at 20:00
  • @goldilocks While related, which port I use for SSH is outside the scope of my question, but thanks for the tip. As for the "kernel panic"; if a CPU stall occur, will it be "resolved" after detected, e.g. process killed? Reading the documentation I wasn't able to figure out what really happens after the RCU's CPU Stall Detector does its job. I'm wondering if this could in any way be related to the server freezing on what seems to be fail2ban's "fault". – timss Aug 25 '14 at 20:26
  • I don't know anything about RCU, but most likely cause (there's a long list at the end of that page) is probably a bug in some kernel code. You should check if there is really a correspondence between this and the "freeze" by examining other examples: is there always a stall during a freeze, and are there other stalls logged when no freeze occurs. I'd guess casually that the RCU stall is actually a very brief (< 1 minute) event..but it may have consequences in userland that tie the system up. In your example the stall seems to be toward the end of the freeze period (the timing is unclear)? – goldilocks Aug 25 '14 at 20:55
  • In the last week (my log rotation limit) there has only been one stall, but multiple freezes. As far as I can tell there was a freeze not long after the RCU stall. However at the same time I got another invalid request (proxyscan), so they may or may not be related. Added relevant logs. Be sure to notice that some of the logs are from different dates. – timss Aug 25 '14 at 21:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.