I've a program producing large amount of redundant information. I'd like to store only unique lines produced on stdout, but I can't do it in multiple steps, like: ./prog > out cat out | sort | uniq > out2

out file would be too big. How can I achieve a "stateful uniq" functionality, where the stateful_uniq would consume lines and emit only the ones it didn't see before?


Doesn't ./prog | sort | uniq > out2 do what you want? Or even ./prog | sort -u > out2 if your sort supports it...

To see unique lines as they come out without sorting the output, you can use an associative array with awk:

./prog | awk '!(count[$0]++)'

As pointed out by Stéphane Chazelas, sort limits its memory use using temporary files; the awk solution doesn't, so if memory use is a concern then it might not be appropriate.

  • sort has to sort everything first, no? It'll run out of memory.
    – user117120
    May 28 '15 at 9:46
  • Ah right, that's what you meant by "too big"; I thought it was because of the disk space used! May 28 '15 at 9:48
  • 2
    sort limits its memory usage by resorting to temporary files. The awk solution stores all unique lines in memory so could end up using more memory than sort would. May 28 '15 at 10:41
  • If amount of uniqueness is finite, then I would hope that the amount of memory used by sort is proportional to uniqueness (not total input size). Though it would never produce any output until the last bit of input has been read. May 28 '15 at 11:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.