0

New to this community. I doing this in bash script. My question pretty much summarizes what I am looking for. I have a log file that has a bunch of IP addresses amongst other data. I want to count the number of unique IP addresses on each specific date and store it into a variable. Any ideas on how I could do this using grep and awk?

Dates are formatted as 11/Feb/2020 (this is an example).

Sample text from log file:

57.34.156.99 - - [11/Feb/2020:04:32:18 +0330]
43.21.223.33 - - [11/Feb/2020:09:13:05 +0330]
87.44.212.82 - - [14/Mar/2020:06:22:01 +0330]
43.21.223.33 - - [11/Feb/2020:11:05:32 +0330]

Output for above:

11/Feb/2020:2
14/Mar/2020:1

As you can see, I want to count repeated IP addresses only once.

Any help is appreciated. Kindly let me know if I should provide more information.

0

1 Answer 1

1

Here is an answer for the sample format of the question, but in general the process is similar for other log formats (usually the date is ISO format and in the first field). To separate the task from the format, first see only IPs and dates:

> awk '{print substr($4,2,10), $1}' file
11/Feb/202 57.34.156.99
11/Feb/202 43.21.223.33
14/Mar/202 87.44.212.82
11/Feb/202 43.21.223.33

We can use an associative array where the hash will be the date and the ip, and it will increase for any appearence of "date-ip". And another array to count the actual result, where the hash will be the date alone.

awk '{d = substr($4,2,10)} !seen[d FS $1]++ {cnt[d]++}
    END {for (x in cnt) print x ":" cnt[x]}
    ' file | sort -t ":" -rnk2

Output:

11/Feb/202:2
14/Mar/202:1
  • Note that the order of the cnt array at the END is undefined, so it is useful to pipe to sort dates by ip count. Alternatively you could use the GNU awk array sorting functions.

  • Variables in awk are not defined implicitly and initially are zero or empty string, so for any new set of "date-ip", !seen[date-ip]++ will be true and after that it's increasing. So next time we encounter this "date-ip", it will be false, and we will not increase cnt[date].


Here is the same thing using sort and uniq, after having extracted only "date-ip" per line:

> awk '{print substr($4,2,10), $1}' file | sort -u | awk '{print $1}' | uniq -c
      2 11/Feb/202
      1 14/Mar/202

Here we remove duplicates with sort -u while sorting (because later uniq needs its input to be sorted), keep only the first field (the date) and finally uniq -c is printing the counts per unique date. This is more readable for people not familiar enough with awk. To demonstrate the process, you can print each step of this command to see how it is going.

4
  • thank you so much for your response! I was wondering whether there is a simpler way to do this without using arrays or hashing. I have not gotten there yet so trying to make it work using what I know so far. thanks again! Commented Oct 26, 2020 at 2:36
  • Yes we can pipe only the date-ip result to some sort|uniq. I will write some more.
    – thanasisp
    Commented Oct 26, 2020 at 2:38
  • thank you so much! I have only gotten to basics of awk and grep so far Commented Oct 26, 2020 at 2:40
  • 1
    @markovv.sim Please accept and upvote this answer if it solves your problem.
    – Quasímodo
    Commented Oct 26, 2020 at 11:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .