Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a file in the format as follows:

$ cat file.txt

What's the best way to parse the file file.txt into a format like: 2 1 1 3

In other words, I want to loop through the file and count the number of times each IP address appears. I've already run it through sort so all the IP addresses are in order and directly after each other.

share|improve this question
I, personally, would import this sort of file into a handy nearby DB (by creating a temp table in any postgres instance I've got around), followed by a quick SQL action and export back to a text file. – oakad Mar 6 '14 at 1:58
up vote 17 down vote accepted

You're looking for uniq -c

If the output of that is not to your liking, it can be parsed and reformatted readily.

For example:

$ uniq -c logfile.txt | awk '{print $2": "$1}' 2 1 1 3
share|improve this answer
Combining uniq and awk does not seem to be a great approach to me... – Hauke Laging Mar 6 '14 at 1:30
Doesn't seem to work here, doesn't catch and count duplicates properly. – Naftuli Tzvi Kay Mar 6 '14 at 1:34
Because uniq only works on sorted input (it matches adjacent matching lines, not any lines from the file). – oakad Mar 6 '14 at 1:54
@HaukeLaging - I appreciate what you're saying but in the same way that most users of computers will never venture beyond OSX & Windows, further still most users of Unix will not venture beyond using designated tools for specific tasks. Using AWK is not for the faint of heart, look at what you had to do to perform this basic task using AWK vs. what Glenn's solution required. I think I'm being fair in saying that his is a simpler solution to mentally grasp, though yours is likely more efficient. BTW, I did UV both since they're both correct! – slm Mar 6 '14 at 2:10
@HaukeLaging - Yes, exactly. As you hang around the site our responsibilities change slightly, IMO. We're responsible for making comprehensive A'ers and looking at the A'ers we provide as teaching moments to the OP and every future visitor that comes across it, again IMO. But it's a personal choice so if you have only a few minutes to spare then providing an A in any form is always appreciated. – slm Mar 6 '14 at 2:22

uniq seems to be the cleverer solution, indeed. The awk way:

awk '{ip_count[$0]++}; '\
'END {for (ip in ip_count) printf "%15s: %d\n",ip,ip_count[ip];}' file
share|improve this answer
Im confused by your comment about awk not giving you the array keys. Isn't that what the for loop is doing? This awk '{ip_count[$0]++}; END {for (ip in ip_count) printf "%15s: %d\n",ip,ip_count[ip];}' file seems to return the same result. (I removed ips). – FDinoff Mar 6 '14 at 2:06
@FDinoff Great. I misunderstood the way for var in array works. I was sure it worked your way but because the man page didn't tell me what I expected I didn't just give it a try... – Hauke Laging Mar 6 '14 at 2:15
You need to also change ips to ip_count in the for loop. – FDinoff Mar 6 '14 at 2:15
@FDinoff Once more we learn the risks of laziness the hard way... :-S – Hauke Laging Mar 6 '14 at 2:21
@slm To err is human but to really mess it up you need a computer... – Hauke Laging Mar 6 '14 at 2:25

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.