I need to count how many times different IP's were in the file.

I'm using this to open file because i don't have grep rights to do it only with grep

My Rights

User lotarc may run the following commands on herbert:
    (root) /bin/su - censored
    (root) /bin/su - censored
    (root) /bin/cat /var/log/nginx/access.log, /bin/cat /var/log/nginx/access.log.1, /bin/zcat /var/log/nginx/access.log.[0-9]*.gz, /bin/cat /var/log/nginx/error.log, /bin/cat /var/log/nginx/error.log.1, /bin/zcat

sudo cat /var/log/nginx/access.log.1 | grep -E '1ip|2ip|3ip'

My question is how to count this ip's and get the output. I need to count from multiple files like access.log.2.gz

Content of file

some ip - - [30/Sep/2019:07:26:03 +0300] "POST /clientapp/request/signUp HTTP/1.1" 200 0 "-" "python-requests/2.22.0" "-"

  • @msp9011 Of course. I need to count specific ip`s Check my edit to see content – Semyon Kolesnikov Sep 30 '19 at 12:49
  • And i guess i need to count each ip's, how many times it does this request from the question – Semyon Kolesnikov Sep 30 '19 at 12:53

Try this,

cat access_log | awk '{a[$1]++} END {for(i in a) print a[i],i}' | sort -n | tail -n1

For specific IPs:

cat access_log | awk '$1 == "" || $1 == "" {a[$1]++} END {for(i in a) print a[i],i}' 
  • It's working well. Now i'm going to try to sort it, because there's many ip's i want to find only with the highest value of requests – Semyon Kolesnikov Sep 30 '19 at 13:14
  • If you are trying to find the highest value, I think a combination of sort and uniq can help you, if you can use those. – number9 Sep 30 '19 at 13:27
  • @msp9011 I found a problem. Your first code is adding 0 after ip like 34228 but it's should be 34228 – Semyon Kolesnikov Sep 30 '19 at 13:36
  • @number9 I tried to use sort and uniq, but it doesn't help me. – Semyon Kolesnikov Sep 30 '19 at 13:37
  • @SemyonKolesnikov it shouldn't be... can u share few lines with those IP? – Siva Sep 30 '19 at 13:41

I might be off my rocker, but this looks straightforward if you just want the count.

 awk '{print $1}' access.log |uniq |wc -l

This will output the count from the one file. If you want all of them you could zgrep all of them and use the dreaded IP address regex:

zgrep '\(25[0-5]\|2[0-4][0-9]\|[01][0-9][0-9]\|[0-9][0-9]\)\.\(25[0-5]\|2[0-4][0-9]\|[01][0-9][0-9]\|[0-9][0-9]\)\.\(25[0-5]\|2[0-4][0-9]\|[01][0-9][0-9]\|[0-9][0-9]\)\.\(25[0-5]\|2[0-4][0-9]\|[01][0-9][0-9]\|[0-9][0-9]\)' access.log.*.gz | awk '{print $1}' | uniq |wc -l

Of course, I bet there is a much simpler way that some clever person can give you, but it is a bit early in the morning for me, I am just throwing this solution out there. Tested on my webserver running nginx, it works for my 50+ logfiles.

As per the OP, if you do not have zgrep, you can just run the logfiles through gzip first...

OP noted in a different message that this worked for him. This is also taken from msp9011 solution, so giving credit there.

cat access.log | awk '{a[$1]++} END {for(i in a) print a[i],i}' |sort -nr

OP would like the solution to print lines with greater than 100 requests. It can be done in one awk statement, I think, but I will do it the brute force way:

  cat access.log | awk '{a[$1]++} END {for(i in a) print a[i],i}' |awk '$1>100' | sort -nr
  • You beat me to the awk answer by about 45 seconds. The second answer here will not work if IPv6 addresses are in the log. – doneal24 Sep 30 '19 at 12:59
  • I can't use zgrep and grep without cat. My rights is in edit – Semyon Kolesnikov Sep 30 '19 at 13:01
  • Ohhh, good one. I always forget IPv6. I wish it was simpler – number9 Sep 30 '19 at 13:01
  • @number9 I want to exclude ip's where less than 100 requests. I tried a couple variants but it doesn't work, can you help pls? – Semyon Kolesnikov Sep 30 '19 at 13:55
  • I updated my solution – number9 Sep 30 '19 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.