So, I hacked this together while undergoing a DDOS attack to pull naughty ips out of my logs. Anyone have any improvements or other suggestions to make it better?

Here's the general idea:

  1. pull ip's only out of log file
  2. sort them
  3. uniq and count them
  4. sort them again

And the string o'pipes:
cut --delim " " -f7 /var/log/apache_access | sort | uniq -c | sort -rn > sorted-ips.txt

  • Not sure if this would be better asked in webmasters... but since it's all using unix utils, and the command line... I figured here would be more appropriate.
    – gabe.
    Aug 12 '10 at 19:00
  • Seems fine here. It's unix based, and isn't specific to web stuff (This could possibly apply to many different things in the Apache or firewall logs IPs) Aug 12 '10 at 21:15

I've always used this:

tail -1000 /var/log/apache_access | awk '{print $1}' | sort -nk1 | uniq -c | sort -nk1

With tail I'm able to set the limit of how far back I really want to go - good if you don't use log rotate (for whatever reason), second I'm making use of awk - since most logs are space delimited I've left my self with the ability to pull additional information out (possibly what URLs they were hitting, statuses, browsers, etc) by adding the appropriate $ variable. Lastly a flaw in uniq it only works in touching pairs - IE:


Will produce:

4 A
1 B
2 A

Not the desired output. So we sort the first column (in this case the ips, but we could sort other columns) then uniq them, finally sort the count ascending so I can see the highest offenders.

  • Yeah, I realized the issue with uniq as well, hence my first sort to put all IPS in order so that duplicates will be next to each other. The tail tip is nice, as well since parsing the whole log at around 4GB's can take some time. Good stuff, thanks.
    – gabe.
    Aug 13 '10 at 13:54
  • -k1 is redundant, (1) there is just one key (2) sort starts using the first word as key anyway.
    – Lekensteyn
    May 5 '13 at 8:23

It sounds like you're in the middle of reinventing the fail2ban wheel.

Give fail2ban a look. It probably does what you want already, and if not, it's easy to customize.

  • 1
    That's a pretty interesting project that I wasn't aware of, thanks. At the same time, I don't want to install anything on the log server that doesn't need to, and it's easy enough to do what I'm already doing. I'm just looking for suggestions on improvements. Thanks!
    – gabe.
    Aug 12 '10 at 19:39
  • This looks really cool, and GPL too.
    – Eli Frey
    Aug 12 '10 at 19:50

Marco Ceppi is right about awk being a better tool for this but awk is also a better tool than sort and uniq since that logic can be moved in to awk. It doesn't make much of a difference if you're just tailing 1000 lines but if you want to look at a huge multi gig log file it can be orders of magnitude faster to move that in to awk.

cat /var/log/apache_access | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -n will do what you need but is much faster for large files. It creates an array of IPs in awk, using the IP address as a key and the number of times the IPs occurs as the value.

The speed up comes because awk does one pass over the data and does most of the work, except for sorting the final output. Using the other method, if you have 1,000,000 lines in the transfer log, awk reads those 1,000,000 lines spitting out 1,000,000 IPs, then sort goes over the entire 1,000,000 IPs, sending the now sorted 1,000,000 IPs to uniq which reduces it down to a much smaller amount of data before giving that to sort. Instead of piping around/doing multiple passes on 1,000,000 IPs, awk does almost everything in one pass.

Using a 5,513,132 line apache log (1.1 gigs) on my laptop, here's a speed comparison:

  • 2m 45s cat ./apache_access | awk '{print $1}' | sort -nk1 | uniq -c | sort -nk1
  • 0m 40s cat ./apache_access | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -n
  • Very cool. I'm adding this to my bag of tricks.
    – gabe.
    Jun 27 '11 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.