Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a data file which contains 15000 lines, but only 400 unique values. I am looking for a way to identify the number of unique values and then the number of occurrences of those values in the file. I came up with the following but it is very very slow. Any thoughts?

for value in `cat mylist.txt | uniq`
do
    counter=`grep $value mylist.txt |wc -l`
    echo $value $counter
done
share|improve this question
1  
Don't do cat file | uniq; like most Unix text utilities, uniq can take a filename as an argument for input –  itsbruce Oct 9 '12 at 16:07
    
@itsbruce: True, but you still need to sort before passing to uniq. sort file | uniq is better. –  terdon Oct 9 '12 at 16:43
    
I'm not disputing that at all; just picking up on a particular style problem of the originally posted code. cat's primary purpose is for concatenating multiple inputs. –  itsbruce Oct 9 '12 at 16:55

1 Answer 1

up vote 6 down vote accepted

Just use sort and uniq:

sort mylist.txt | uniq | wc -l

That will give you the number of unique values. To get the number of occurrences of each unique value, use uniq's -c option:

sort mylist.txt | uniq -c

From the uniq man page:

   -c, --count
               prefix lines by the number of occurrences

Also, for future reference, grep's -c option is often useful:

 -c, --count
              Suppress  normal  output;  instead  print  a  count  of
              matching  lines  for  each  input  file.   With the -v,
              --invert-match option (see below),  count  non-matching
              lines.  (-c is specified by POSIX.)
share|improve this answer
    
Thank you terdon... "sort mylist.txt | uniq -c" was exactly what I needed. –  Dana Max Oct 9 '12 at 17:17

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.