New answers tagged duplicate
With comm from GNU coreutils: $ comm -3 <(sort -u <(./c1)) <(sort -u <(./c2)) | tr -d '\t' /usr/bin/baz /usr/bin/buzz /usr/bin/qux From man comm: Compare sorted files FILE1 and FILE2 line by line. With no options, produce three-column output. Column one contains lines unique to FILE1, column two contains lines unique to ...
awk-pipe to only let the 1st occurance of an input line pass thru: ( ./c1 ; ./c2 ) | awk '!u[$0]++' This does not take time for sorting but needs a memory of seen lines. So for huge amounts of input sort and uniq may be better...
A fairly simple pipeline should do the trick: (./c1; ./c2) | sort -u The parentheses get stdout of both ./c1 and ./c2 into stdin of the sort command. The option -u prints only 1 of each group of matching lines. Thanks to John WH Smith for noticing a simplification, and Bakuriu for an insight.
I would recommend utilizing sed to parse the text and remove duplicate lines. So the first command keeps the duplicate line sed '$!N; /^\(.*\)\n\1$/!P; D' The second command will delete the duplicates sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'
Top 50 recent answers are included