Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I need to look for some requests in a huge pile of apache logs. My only requirement is this:

I need to only view the requests coming from any IP address that is NOT included in a list of 50 IP ranges I have.

How can I make that happen using any combination of regexes awk grep or anything? Can't think of an easy way. The idea would be to get each line, get the first part (the IP address), match it to a file with all the ranges, and if its not there, then display it. No idea on how to go about doing this, so any help is welcome!

Samples:

a Typical http log line is

123.456.789.012 - - [22/Oct/2012:06:37:48 +0100] "GET /test/test HTTP/1.1" 302 224 "-" "some user agent/4.3.5"

A typical line out of my IP ranges file is

192.168.0.1 - 192.168.0.255

Of cours ethe IP ranges file could be converted to 192.168.0.1/24 notation if necessary. The good thing is that all the ranges are Class C (just noticed that), so I guess only the first 3 parts of the IP address could be matched and that should be good enough.

share|improve this question
    
Can you please give a sample of your file and also the file containing the ranges? –  Guru Oct 22 '12 at 9:24
    
@Guru done, thx –  unixkiddie Oct 22 '12 at 9:35

2 Answers 2

up vote 2 down vote accepted

A simple and crude way could be to use grep.

Create a file ( ranges.txt ) with you're ranges something like this:

192\.168\.0\.[0-9]*
10\.0\.0\.[0-9]*

To create that file from the range-file you already had you could use sed like so:

sed -n -e 's/^\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\) .*$/^\1\\.\2\\.\3\\.[0-9]* /p' your-range-file > ranges.txt

Then exclude lines matching the pattern in that file using grep like so:

$ grep -v -f ranges.txt apache-log-file.log

or

$ cat apache-log-file.log | < do some pre cleaning > | grep -v -f ranges.txt

This could help you to get started but its probably not a good solutions if the query should be run often and on big log-files.

Good luck!

share|improve this answer

Log is your log file, and iprange is your file containing the ipranges. Perl part of the solution gets the first 3 components of your ip address, and the for loop prints it if it does not exist in the file ipranges:

for i in `perl -lne 'print $1 if (m/(\d{1,3}\.\d{1,3}\.\d{1,3})\.\d{1,3}/);' log`
> do
>  grep -q $i iprange || echo $i;
> done
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.