0

Recently we have been seeing a lot of connections from IPs in to our public git server. When this happens, our devs are unable to commit via SSH as the server is very, very slow and I am unable to login via SSH remotely and have to login from the console to resolve the issue. HTTPs to gitlab still works when this happens. System resources are plentiful and I have edited sshd_config to block all port 22 access unless they have a key. I also have IPS setup to block multiple SSH attempts and have fail2ban on the server.

With all that setup, I am unsure how this is creating such an issue for the server. I can resolve the issues by restarting the sshd service and blocking the offending subnets, but why is it messing up SSH so much that it makes the server unusable via SSH? Are all of these attempts somehow using all of the ports and therefore blocking legit access attempts? Below is a copy of the secure log file with an attempt. Note, there is a big time gap from 14:30:18 to 14:30:50.

Sep  4 14:30:17 somegitserver sshd[5924]: Connection from 11.11.11.11 port 62290
Sep  4 14:30:17 somegitserver sshd[5924]: debug1: Client protocol version 2.0; client software version libssh-0.11
Sep  4 14:30:17 somegitserver sshd[5924]: debug1: no match: libssh-0.11
Sep  4 14:30:17 somegitserver sshd[5924]: debug1: Enabling compatibility mode for protocol 2.0
Sep  4 14:30:17 somegitserver sshd[5924]: debug1: Local version string SSH-2.0-OpenSSH_5.3
Sep  4 14:30:17 somegitserver sshd[5941]: debug1: permanently_set_uid: 74/74
Sep  4 14:30:17 somegitserver sshd[5941]: debug1: list_hostkey_types: ssh-rsa,ssh-dss
Sep  4 14:30:17 somegitserver sshd[5941]: debug1: SSH2_MSG_KEXINIT sent
Sep  4 14:30:17 somegitserver sshd[5941]: debug1: SSH2_MSG_KEXINIT received
Sep  4 14:30:17 somegitserver sshd[5941]: debug1: kex: client->server aes128-cbc hmac-sha1 none
Sep  4 14:30:17 somegitserver sshd[5941]: debug1: kex: server->client aes128-cbc hmac-sha1 none
Sep  4 14:30:17 somegitserver sshd[5941]: debug1: expecting SSH2_MSG_KEXDH_INIT
Sep  4 14:30:18 somegitserver sshd[5941]: debug1: SSH2_MSG_NEWKEYS sent
Sep  4 14:30:18 somegitserver sshd[5941]: debug1: expecting SSH2_MSG_NEWKEYS
Sep  4 14:30:18 somegitserver sshd[5941]: debug1: SSH2_MSG_NEWKEYS received
Sep  4 14:30:18 somegitserver sshd[5941]: debug1: KEX done
Sep  4 14:30:18 somegitserver sshd[5941]: debug1: userauth-request for user root service ssh-connection method keyboard-interactive
Sep  4 14:30:50 somegitserver sshd[5944]: debug1: Enabling compatibility mode for protocol 2.0
Sep  4 14:30:51 somegitserver sshd[5941]: debug1: do_cleanup
Sep  4 14:30:51 somegitserver sshd[5945]: debug1: rexec start in 5 out 5 newsock 5 pipe 10 sock 11
Sep  4 14:30:51 somegitserver sshd[5944]: debug1: Local version string SSH-2.0-OpenSSH_5.3
Sep  4 14:30:51 somegitserver sshd[5924]: debug1: do_cleanup
Sep  4 14:30:51 somegitserver sshd[5945]: debug1: inetd sockets after dupping: 3, 3
Sep  4 14:30:51 somegitserver sshd[5961]: debug1: permanently_set_uid: 74/74
Sep  4 14:30:51 somegitserver sshd[5924]: debug1: PAM: cleanup

What am I missing?

  • 1
    is fail2ban actually blocking failed connections? are you seeing lots of entries like NOTICE [sshd] Unban ip.ip.ip.ip in your /var/log/fail2ban.log (or wherever it is on your system)? also, if you are running gitlab in a container (e.g. the gitlab/gitlab-ce docker image), fail2ban on the host won't see the container's auth logs unless you configure it to watch them as well as the system's own auth logs. – cas Sep 5 at 8:43
  • btw, have you considered running a private git server for devs to upload to, and a public git server for others to clone from? – cas Sep 5 at 8:46
  • I am not seeing any entries in fail2ban.log file. Will fail2ban not work since sshd_config is blocking all connections unless it uses a key? – saleetzo Sep 5 at 17:45
  • 1
    no, that is an auth failure and is something that would be logged by sshd. are you seeing that in your /var/log/auth.log (or wherever auth failures are logged on your system)? is f2b configured to monitor that log file and block ssh failures? i.e. do you have enabled=true under the [sshd] section of your f2b jail.conf (or jail.local)? – cas Sep 5 at 23:24
  • I did not have fail2ban setup properly. It is now blocking SSH attempts from the correct log file – saleetzo Sep 9 at 23:28
2

The issue was with my fail2ban config not reading the correct log file. Once I updated my jail.local file and specified /var/log/secure instead of /var/log/sshd.log, I was starting to see IPs go in to the jail. This had fail2ban working but it still wasn't blocking certain log entries. I wanted to block specific entries that contained 11: Bye Bye but my custom filter in fail2ban was not working: ^%(__prefix_line)sReceived disconnect from <HOST>: 11: \S+: *$

To completely resolve my issue, I found that version 0.10.5.dev1 of fail2ban could catch that specific entry in the log file by default. I upgraded and redid the config and it is now working as I hoped.

  • thanks to @cas for guidance on this – saleetzo Sep 11 at 0:50
  • 1
    good answer. BTW, you probably could have just copied the rule from the updated version into your filter.d/ssh file. or the entire file. Something to keep in mind in future, because upgrading to the latest dev version isn't always a good idea. – cas Sep 11 at 7:01
  • I thought about doing this but the way the new version deals with the blocking is a little different. But you are right, I was hesitant to update to a dev version. – saleetzo Sep 11 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.