4

The Question

How can I exclude inconsequential¹ failed SSH logins?

Background

For ambient awareness, I like to have little background windows showing system error logs on every GNU/Linux box I admin. I used to do this with xconsole, but now use xterm running journalctl -f.

Unfortunately, on the machines where ssh is a needed service, the journalctl log is a constant stream of clutter from crackers trying to ssh in with lists of common names/passwords.² I see failures scrolling in every second for accounts that don't even exist. This makes it hard to see anything else in my console log.

Solutions?

I don't know systemd half as well as I ought, so it's possible there's a slick, simple answer, but I haven't found one yet.³ Any solution is welcome. I suspect it's going to entail messing around with pam, journalctl, and/or grep -v.

I'm looking for a solution which still shows me attempts against existing accounts. However, if that is too difficult, I'll accept an answer which hides all failed login attempts.

Journalctl examples

Do not show the following:

Dec 12 17:19:21 gaia sshd[10146]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=139.59.90.40  user=root
Dec 12 17:19:21 gaia sshd[10148]: Invalid user git from 14.29.201.30
Dec 12 17:19:21 gaia sshd[10148]: input_userauth_request: invalid user git [preauth]
Dec 12 17:19:21 gaia sshd[10148]: pam_unix(sshd:auth): check pass; user unknown
Dec 12 17:19:21 gaia sshd[10148]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=14.29.201.30
Dec 12 17:19:22 gaia sshd[10150]: Invalid user molisoft from 5.135.152.97
Dec 12 17:19:22 gaia sshd[10150]: input_userauth_request: invalid user molisoft [preauth]
Dec 12 17:19:22 gaia sshd[10150]: pam_unix(sshd:auth): check pass; user unknown
Dec 12 17:19:22 gaia sshd[10150]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=5.135.152.97
Dec 12 17:19:23 gaia sshd[10146]: Failed password for root from 139.59.90.40 port 37752 ssh2
Dec 12 17:19:23 gaia sshd[10146]: Received disconnect from 139.59.90.40: 11: Bye Bye [preauth]
Dec 12 17:19:23 gaia sshd[10148]: Failed password for invalid user git from 14.29.201.30 port 41178 ssh2
Dec 12 17:19:23 gaia sshd[10148]: Received disconnect from 14.29.201.30: 11: Bye Bye [preauth]
Dec 12 17:19:24 gaia sshd[10150]: Failed password for invalid user molisoft from 5.135.152.97 port 50730 ssh2
Dec 12 17:19:24 gaia sshd[10150]: Received disconnect from 5.135.152.97: 11: Bye Bye [preauth]

But do show valid logins:

Dec 10 08:56:16 gaia sshd[1414]: Accepted publickey for sophia from 24.22.130.192 port 41610 ssh2: RSA 6b:5f:aa:9c:d8:33:65:2c:c4:0c:88:12:ec:9b:ff:51
Dec 10 08:56:16 gaia sshd[1414]: pam_unix(sshd:session): session opened for user sophia by (uid=0)
Dec 10 21:06:37 gaia sshd[1414]: pam_unix(sshd:session): session closed for user sophia

Also show failed attempts on valid accounts (other than root):

Dec 12 00:46:28 gaia sshd[30924]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.9.134  user=sophia
Dec 12 00:46:30 gaia sshd[30924]: Failed password for sophia from 192.168.9.134 port 55888 ssh2
Dec 12 00:46:33 gaia sshd[30924]: Connection closed by authenticating user sophia 192.168.9.134 port 55888 [preauth]

And, of course, any service which is not sshd should be shown.

Dec 10 08:56:16 gaia systemd[1]: Started User Manager for UID 3237.

¹ Defined as "username does not exist" or "username is root". Root ssh is disabled on my boxen.
² I do use fail2ban. It helps, but the attempts are coming from too many IP addresses.
³ E.g., journalctl allows one to grep, but not grep -v to exclude certain criteria.

3
  • 2
    how about journalctl -f -u ssh | awk -f filter.awk where filter.awk reads your password database in a BEGIN statement and then processes the lines. I don't have a noisy sshd right now to see what login attempts look like, but I suspect you just would need a line like $4 in users {print;flush} once your BEGIN block had loaded all the users you care about into the users array.
    – icarus
    Dec 10, 2019 at 5:02
  • I had hoped for cleaner answer, but awk might work. Note, I can't use "-u ssh" as I want to see all the logs. I'll edit the question to include some example login attempts.
    – hackerb9
    Dec 12, 2019 at 7:53
  • 1
    Fun fact; I once had somebody fill up my logspace forcing a rotate to bitbucket trying a dictionary attack on root; on a box where password auth was completely disabled. Come on; you can tell that one from the first attempt.
    – Joshua
    Nov 28, 2022 at 21:09

2 Answers 2

2

Use

 journalctl -f | awk -f filter.awk

with a file filter.awk containing

# get the non sshd stuff out the way
$5 !~ /^sshd\[[0-9]*\]:$/ { print ; next }
# show authentication errors for valid users which are not root
/authentication failure/ && $15 != "" && $15 != "user=root"
/Failed password for/ && !/invalid user/ && !/for root/
# Show valid logins
/Connection closed by authenticating user/
/Accepted publickey/
/pam_unix\(sshd:session):/

Nothing very clever. If it is not an sshd line then print it and move on to the next line. Otherwise print the lines matching the expression. I used GNU awk version 5.0.1

5
  • Thank you! I'm still hoping a more elegant solution is out there, but it should work. I realized my previous script actually uses journalctl --output=cat to save space on the screen, but I think I can emulate that by having awk chop off the first five fields when printing.
    – hackerb9
    Dec 13, 2019 at 19:00
  • You can add a line { service=$5 ; $1=$2=$3=$4=$5="" } at the top and change the $5 to service on the 2nd line to do this saving of screen area.
    – icarus
    Dec 13, 2019 at 20:54
  • Setting fields 1 to 5 to the empty string like that seems to print a field separator at the beginning of each line. Does awk have an ability to shift 5 as in bash?
    – hackerb9
    Dec 14, 2019 at 19:35
  • I figured out how to do it using a for loop in awk, but it seems like there's a better way. No?
    – hackerb9
    Dec 15, 2019 at 4:42
  • the failed attempts are still written to the systemd journal. How can we prevent that in the first place?
    – Al Ro
    Mar 14, 2022 at 20:07
0

You can significantly reduce the log spam in /etc/ssh/sshd_config by:

  • Disabling password authentication and permitting only pubkeys:

    PasswordAuthentication no
    ChallengeResponseAuthentication no
    PermitRootLogin prohibit-password
    
  • Using a level of INFO or less (INFO is the sshd upstream default):

    LogLevel INFO
    

The only thing that this doesn't get rid of yet is when login spam bots don't your key exchange methods; then you get e.g.:

Unable to negotiate with 203.0.113.1 port 40442: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1 [preauth]

I haven't found a way to get rid of that bit yet.

1
  • Thanks, but that's not currently an option on most of the computers I admin. If that eventually does happen, it'll be for security reasons. It's a pretty disruptive change on all the users, so I don't think there'll be much sympathy for my inability to grep systemd logs. 😼🤔
    – hackerb9
    Oct 20, 2020 at 19:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .