I've a program producing large amount of redundant information. I'd like to store only unique lines produced on stdout, but I can't do it in multiple steps, like: ./prog > out cat out | sort | uniq > out2

out file would be too big. How can I achieve a "stateful uniq" functionality, where the stateful_uniq would consume lines and emit only the ones it didn't see before?


Doesn't ./prog | sort | uniq > out2 do what you want? Or even ./prog | sort -u > out2 if your sort supports it...

To see unique lines as they come out without sorting the output, you can use an associative array with awk:

./prog | awk '!(count[$0]++)'

As pointed out by Stéphane Chazelas, sort limits its memory use using temporary files; the awk solution doesn't, so if memory use is a concern then it might not be appropriate.

  • sort has to sort everything first, no? It'll run out of memory. – user117120 May 28 '15 at 9:46
  • Ah right, that's what you meant by "too big"; I thought it was because of the disk space used! – Stephen Kitt May 28 '15 at 9:48
  • 2
    sort limits its memory usage by resorting to temporary files. The awk solution stores all unique lines in memory so could end up using more memory than sort would. – Stéphane Chazelas May 28 '15 at 10:41
  • If amount of uniqueness is finite, then I would hope that the amount of memory used by sort is proportional to uniqueness (not total input size). Though it would never produce any output until the last bit of input has been read. – ctrl-alt-delor May 28 '15 at 11:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.