I have a file contains around million number of lines. In the lines I have a field called transactionid, which has repetitive values. What I need to do is to count them distinctly.

No matter of how many times a value is repeated, it should be counted only once.

  • it would be more easier, if you could just give a glimpse of format of the file..not necessarily the data. – Nikhil Mulley Jan 11 '12 at 14:20
  • btw, do you want the value to be counted as 1 irrespective of how many times it exists, or you want the count of the number of occurrences/repetitions? if you just want it to be counted once, then how the distinct values are counted? Can you please check my edit on your question and confirm if I am right in interpreting. – Nikhil Mulley Jan 11 '12 at 14:27
  • @Nikhil This is clear from the question: ... No matter of how many times a value is repeated, it should be counted as 1. ... – user13742 Jan 11 '12 at 14:28
  • ok, then answer from @hesse would do your need. – Nikhil Mulley Jan 11 '12 at 14:30
  • sorry for latency. I was out of internet connection. seperator is 2|' and field is field 28. I used; cat <file_name> | awk -F"|" '{if ((substr($2,0,8)=='20120110')) print $28}' | sort -u | wc -l the if clause was for another check of date as it seems obvious :) – Olgun Kaya Jan 12 '12 at 6:29

OK, Assuming that your file is a text file, having the fields separated by comma separator ','. You would also know which field 'transactionid' is in terms of its position. Assuming that your 'transactionid' field is 7th field.

awk -F ',' '{print $7}' text_file | sort | uniq -c

This would count the distinct/unique occurrences in the 7th field and prints the result.


There is no need to sort the file .. (uniq requires the file to be sorted)
This awk script assumes the field is the first whitespace delimiited field.

awk 'a[$1] == "" { a[$1]="X" } END { print length(a) }' file 
  • For a huge file (as in, getting close to the size of RAM), awk will consume a lot of memory. Most sort implementations are designed to cope well with huge files. – Gilles Jan 12 '12 at 1:59

Maybe not the sleekest method, but this should work:

awk '{print $1}' your_file | sort | uniq | wc -l

where $1 is the number corresponding to the field to be parsed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.