14

I have a file in the format as follows:

$ cat file.txt

27.33.65.2
27.33.65.2
58.161.137.7
121.50.198.5
184.173.187.1
184.173.187.1
184.173.187.1

What's the best way to parse the file file.txt into a format like:

27.33.65.2: 2
58.161.137.7: 1
121.50.198.5: 1
184.173.187.1: 3

In other words, I want to loop through the file and count the number of times each IP address appears. I've already run it through sort so all the IP addresses are in order and directly after each other.

1
  • I, personally, would import this sort of file into a handy nearby DB (by creating a temp table in any postgres instance I've got around), followed by a quick SQL action and export back to a text file.
    – oakad
    Mar 6, 2014 at 1:58

4 Answers 4

27

You're looking for uniq -c

If the output of that is not to your liking, it can be parsed and reformatted readily.

For example:

$ uniq -c logfile.txt | awk '{print $2": "$1}'
27.33.65.2: 2
58.161.137.7: 1
121.50.198.5: 1
184.173.187.1: 3
11
  • Combining uniq and awk does not seem to be a great approach to me... Mar 6, 2014 at 1:30
  • 3
    Because uniq only works on sorted input (it matches adjacent matching lines, not any lines from the file).
    – oakad
    Mar 6, 2014 at 1:54
  • 1
    You must sort the results prior to piping them to uniq. If you read the original Q the OP states that he's already sorted the results using sort!
    – slm
    Mar 6, 2014 at 1:56
  • 2
    @HaukeLaging - I appreciate what you're saying but in the same way that most users of computers will never venture beyond OSX & Windows, further still most users of Unix will not venture beyond using designated tools for specific tasks. Using AWK is not for the faint of heart, look at what you had to do to perform this basic task using AWK vs. what Glenn's solution required. I think I'm being fair in saying that his is a simpler solution to mentally grasp, though yours is likely more efficient. BTW, I did UV both since they're both correct!
    – slm
    Mar 6, 2014 at 2:10
  • 1
    @HaukeLaging - Yes, exactly. As you hang around the site our responsibilities change slightly, IMO. We're responsible for making comprehensive A'ers and looking at the A'ers we provide as teaching moments to the OP and every future visitor that comes across it, again IMO. But it's a personal choice so if you have only a few minutes to spare then providing an A in any form is always appreciated.
    – slm
    Mar 6, 2014 at 2:22
6

uniq seems to be the cleverer solution, indeed. The awk way:

awk '{ip_count[$0]++}; '\
'END {for (ip in ip_count) printf "%15s: %d\n",ip,ip_count[ip];}' file
6
  • +1. If the order of the output is important to the OP, this answer does not make any guarantees: iterating over the keys of an associative array has not inherent order. Mar 6, 2014 at 11:16
  • @glennjackman But adding sort to my answer is still faster as fewer items have to be sorted. ;-) Mar 6, 2014 at 11:29
  • oh yeah? OH YEAH?!? ;) the input is already sorted. This awk answer shuffles them, so it's still more work. Nyah! ;) Mar 6, 2014 at 15:26
  • Looking back at that comment, I'm embarrassed. Even with the smilies, it's pretty damn obnoxious. My apologies. Jul 19, 2021 at 15:22
  • 1
    heh heh, as a Canadian, it is my patriotic duty to say "sorry" as often as possible. Jul 20, 2021 at 1:12
0

firest sort file then get count by unic -c

sort filename | uniq -c

2
  • 1
    The file is already sorted (according to the user in the question), and uniq -c would work but provide the output on the wrong format. This is why the accepted answer does not use sort and instead reformats the output of uniq -c.
    – Kusalananda
    Aug 10, 2019 at 8:32
  • Thanks @Aeyd. I was looking for this command. It helps Nov 18, 2019 at 9:23
0

I would use python. Every linux ststem nowadays has python2 installed.

Add each ip address into a dict (associative array) as key=value pairs i.e. {"12.34.56.78":1, "87.76.43.21":3}.

You 'verify' the ip address as a key and increment the value by 1. If you use defaultdict("ip"), if the key doesn't exist, it is created with a default value of 0. If the key does exist already, defaultdict does nothing. The value is incremented on the next line.

#!/usr/bin/python2

infile = open("file.txt","r")
iplist = {}  # create an empty dict

for line in infile:
    line = line.strip()   # remove newline.
    if line: # if not a blank line.
        iplist.setdefault(line, 0) # check for ip and add with default value of 0
        iplist[line] += 1 # increment

outfile = open("out.txt","w") #open output file

for key in iplist.keys():
    line = "%-15s = %s" % (key, iplist[key])
    print line   # print uf desired.
    outfile.write(line + "\n")

outout file:

cat out.txt                                                          
27.33.65.2      = 2
58.161.137.7    = 1
121.50.198.5    = 1
184.173.187.1   = 3

I know you were looking for a command line solution, but as you can see it is an elegantly formatted display that only took a dozen lines or so. Python is an excellent tool for administration.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .