I've a program producing large amount of redundant information. I'd like to store only unique lines produced on
stdout, but I can't do it in multiple steps, like:
./prog > out
cat out | sort | uniq > out2
out file would be too big. How can I achieve a "stateful uniq" functionality, where the
stateful_uniq would consume lines and emit only the ones it didn't see before?