2

I have lines in a log similar to:

2015/11/02-07:55:39.735 INFO failed with ERR_AUTHORIZATION_REQUIRED.  (10.10.10.11:61618) is not a trusted source.
2015/11/02-07:55:40.515 INFO failed with ERR_AUTHORIZATION_REQUIRED.  (10.10.10.11:51836) is not a trusted source.
2015/11/02-07:55:39.735 INFO failed with ERR_AUTHORIZATION_REQUIRED.  (10.10.10.10:61615) is not a trusted source.
2015/11/02-07:55:40.515 INFO failed with ERR_AUTHORIZATION_REQUIRED.  (10.10.10.10:51876) is not a trusted source.
2015/11/02-07:55:39.735 INFO failed with ERR_AUTHORIZATION_REQUIRED.  (10.10.10.10:61614) is not a trusted source.
2015/11/02-07:55:39.735 INFO failed with ERR_AUTHORIZATION_REQUIRED.  (10.10.10.15:61614) is not a trusted source.
2015/11/02-07:55:39.735 INFO failed with ERR_AUTHORIZATION_REQUIRED.  (10.10.10.15:61618) is not a trusted source.
2015/11/02-07:55:39.735 INFO failed with ERR_AUTHORIZATION_REQUIRED.  (10.10.10.15:61613) is not a trusted source.

So, I've tried the following command to get the count of each uniq IP, sorted:

grep ERR_AUTHORIZATION_REQUIRED file.log | awk '{print $6}' | cut -s -d ':' -f1 | tr -d '(' | sort | uniq -c

The output I get though is similar to the following:

3 10.10.10.10
2 10.10.10.11
3 10.10.10.15

So it's like the IP is getting sorted before applying the uniq -c (which makes sense given the command), but if I swap the uniq and sort commands, every IP is printed with a count of 1.

5

From the uniq manpage:

DESCRIPTION
     Discard all but one of successive identical lines from INPUT (or standard input), writing to OUTPUT (or standard output).

Here the critical word is "successive". It doesn't search for duplicates at any point in the stream, only those that immediately follow. Sorting forces all duplicates to be next to each other, so they can be removed (and counted).

| improve this answer | |
  • So would the solution here be to sort twice? Once for catching uniqs properly, and again for display purposes? – MrDuk Nov 2 '15 at 17:46
  • 1
    If you just want the data sorted (as in sort | uniq | sort) then the second sort doesn't do anything (uniq doesn't unsort it). If you want it sorted by the count (as in sort | uniq -c | sort -n), then yes you'd need two sorts because the second sort order is potentially different. – BowlOfRed Nov 2 '15 at 17:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.