Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a file contains around million number of lines. In the lines I have a field called transactionid, which has repeatitive values. What I need to do is to count them distinctly.

No matter of how many times a value is repeated, it should be counted only once.

Can you help with a way.

share|improve this question
it would be more easier, if you could just give a glimpse of format of the file..not necessarily the data. – Nikhil Mulley Jan 11 '12 at 14:20
btw, do you want the value to be counted as 1 irrespective of how many times it exists, or you want the count of the number of occurrences/repetitions? if you just want it to be counted once, then how the distinct values are counted? Can you please check my edit on your question and confirm if I am right in interpreting. – Nikhil Mulley Jan 11 '12 at 14:27
@Nikhil This is clear from the question: ... No matter of how many times a value is repeated, it should be counted as 1. ... – user13742 Jan 11 '12 at 14:28
ok, then answer from @hesse would do your need. – Nikhil Mulley Jan 11 '12 at 14:30
sorry for latency. I was out of internet connection. seperator is 2|' and field is field 28. I used; cat <file_name> | awk -F"|" '{if ((substr($2,0,8)=='20120110')) print $28}' | sort -u | wc -l the if clause was for another check of date as it seems obvious :) – Olgun Kaya Jan 12 '12 at 6:29
up vote 11 down vote accepted

Ok, Assuming that your file is a text file, having the fields seperated by comma seperator ','. You would also know which field 'transactionid' is in terms of its position. Assuming that your 'transactionid' field is 7th field.

awk -F ',' '{print $7}' text_file | sort | uniq -c

This would count the distinct|unique occurrences in the 7th field and prints the result.

share|improve this answer

Maybe not the sleekest method, but this should work:

awk '{print $1}' your_file | sort | uniq | wc -l

where $1 is the number corresponding to the field to be parsed.

share|improve this answer

There is no need to sort the file .. (uniq requires the file to be sorted)
This awk script assumes the field is the first whitespace delimiited field.

awk 'a[$1] == "" { a[$1]="X" } END { print length(a) }' file 
share|improve this answer
+1 and thanks :-) – Nikhil Mulley Jan 11 '12 at 14:36
For a huge file (as in, getting close to the size of RAM), awk will consume a lot of memory. Most sort implementations are designed to cope well with huge files. – Gilles Jan 12 '12 at 1:59

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.