0

I want to get the unique words and its count from a file. When I run the bellow command,

sort words.txt | uniq -c



   2 america
   4 and
   1 england
   1 file
   1 for
   1 place

But I want the output in following format

america,2
and,4
england,1
file,1
for,1
place,1

My input file is around 30-40Gb. So what is the best way to print the output in this format?

4

You can add an awk line to the end of your command. For example,

sort words.txt | uniq -c | awk '{print $2","$1}'

Basically, it takes the second column and places it before the first column while separating it by a comma. I do not know how expensive it is to run it on a 30-40Gb file.

1

we can do with awk itself...

Try below,

awk '{j[$0]++} END {for (i in j) print i","j[i]}' words.txt
  • @drewbenn: you mean few duplicates (thus many distinct values, filling memory). And with GNU awk 4 up setting PROCINFO["sorted_in"]="@ind_str_asc" will give the sorted order. – dave_thompson_085 Sep 21 '19 at 5:58
1

With sed:

sort words.txt | uniq -c | sed -E 's/^ *([0-9]) (.+)/\2,\1/g' 

Tested with GNU, Busybox and BSD implementations of sed. Output would be:

america,2
and,4
england,1
file,1
for,1
place,1

I ran a test on 200MB file and noticed that sed itself is still quite fast:

$ time sed  -E 's/^ *([0-9]) (.+)/\2,\1/g'  HUGE | head
america,2
and,4
england,1
file,1
for,1
place,1
america,2
and,4
england,1
file,1

real    0m0.006s
user    0m0.003s
sys     0m0.006s

But saving data to the file on disk takes a lot of time with both -i option and by redirecting output using > shell operator:

$ time sed -i -E 's/^ *([0-9]) (.+)/\2,\1/g'  HUGE
real    0m45.793s
user    0m31.965s
sys     0m13.574s
$ time sed -E 's/^ *([0-9]) (.+)/\2,\1/g'  HUGE > HUGE_NO_I
real    0m29.016s
user    0m28.684s
sys     0m0.119s

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.