Parsing log files for frequent IP's

Question

So, I hacked this together while undergoing a DDOS attack to pull naughty ips out of my logs. Anyone have any improvements or other suggestions to make it better?

Here's the general idea:

1. pull ip's only out of log file
2. sort them
3. uniq and count them
4. sort them again

And the string o'pipes:
cut --delim " " -f7 /var/log/apache_access | sort | uniq -c | sort -rn > sorted-ips.txt

Answer 1

I've always used this:

tail -1000 /var/log/apache_access | awk '{print $1}' | sort -nk1 | uniq -c | sort -nk1

With tail I'm able to set the limit of how far back I really want to go - good if you don't use log rotate (for whatever reason), second I'm making use of awk - since most logs are space delimited I've left my self with the ability to pull additional information out (possibly what URLs they were hitting, statuses, browsers, etc) by adding the appropriate $ variable. Lastly a flaw in uniq it only works in touching pairs - IE:

A
A
A
A
B
A
A

Will produce:

4 A
1 B
2 A

Not the desired output. So we sort the first column (in this case the ips, but we could sort other columns) then uniq them, finally sort the count ascending so I can see the highest offenders.

Answer 2

It sounds like you're in the middle of reinventing the fail2ban wheel.

Give fail2ban a look. It probably does what you want already, and if not, it's easy to customize.

Answer 3

Marco Ceppi is right about awk being a better tool for this but awk is also a better tool than sort and uniq since that logic can be moved in to awk. It doesn't make much of a difference if you're just tailing 1000 lines but if you want to look at a huge multi gig log file it can be orders of magnitude faster to move that in to awk.

cat /var/log/apache_access | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -n will do what you need but is much faster for large files. It creates an array of IPs in awk, using the IP address as a key and the number of times the IPs occurs as the value.

The speed up comes because awk does one pass over the data and does most of the work, except for sorting the final output. Using the other method, if you have 1,000,000 lines in the transfer log, awk reads those 1,000,000 lines spitting out 1,000,000 IPs, then sort goes over the entire 1,000,000 IPs, sending the now sorted 1,000,000 IPs to uniq which reduces it down to a much smaller amount of data before giving that to sort. Instead of piping around/doing multiple passes on 1,000,000 IPs, awk does almost everything in one pass.

Using a 5,513,132 line apache log (1.1 gigs) on my laptop, here's a speed comparison:

• 2m 45s cat ./apache_access | awk '{print $1}' | sort -nk1 | uniq -c | sort -nk1
• 0m 40s cat ./apache_access | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -n