Say, I have two commands that return some text. For example:

$ ./c1
$ ./c2

I want to remove the duplicate lines; i.e. the output will be (order isn't important):


How would I go about doing this?

  • 1
    is the order does not matter then concatenate them together and pipe the result through sort -u – rob Jun 26 '14 at 15:14
  • Just to confirm: based on your example, you want to remove the lines that are present in both files, right? Many people have given answers that retain a single copy of each line, which is what the text of your question says. Please edit your question to make it consistent. – Gilles Jun 28 '14 at 7:29

With comm from GNU coreutils:

$ comm -3 <(sort -u <(./c1)) <(sort -u <(./c2)) | tr -d '\t'

From man comm:

Compare sorted files FILE1 and FILE2 line by line.

       With  no  options,  produce  three-column  output.  Column one contains
       lines unique to FILE1, column two contains lines unique to  FILE2,  and
       column three contains lines common to both files.

       -1     suppress column 1 (lines unique to FILE1)

       -2     suppress column 2 (lines unique to FILE2)

       -3     suppress column 3 (lines that appear in both files)
  • What's the point of using comm here? Why not sort the whole thing? – Gilles Jun 27 '14 at 22:33
  • @Gilles: What do you mean "sort the whole thing"? comm did its jobs, compare two sorted files. – cuonglm Jun 28 '14 at 2:05
  • Yes, but why sort the two files separately and then remove lines that appear in both, when a single application sort -u on both files would immediately give the desired result? This isn't even correct — you're suppressing lines that appear in both files instead of keeping one copy. You also aren't removing duplicates already present in one of the files, although this wasn't a clearly stated requirement. – Gilles Jun 28 '14 at 7:11
  • @Gilles: Oh, I don't think the case where a file has duplicated lines. I will add -u in my answer. I use comm because it works both with files and process substitution. With process substitution all thing have done immediately. In the case use only sort, you must spawn a process to run ./c1, then ./c2 then pipe it to sort. – cuonglm Jun 28 '14 at 7:26
  • Apologies: I've just reread the question, and the example isn't consistent with the text — you (alone) gave an answer that matches the example, whereas others answered based on the text. – Gilles Jun 28 '14 at 7:30

A fairly simple pipeline should do the trick:

(./c1; ./c2) | sort -u

The parentheses get stdout of both ./c1 and ./c2 into stdin of the sort command. The option -u prints only 1 of each group of matching lines.

Thanks to John WH Smith for noticing a simplification, and Bakuriu for an insight.

  • 1
    No, it isn't related. The decorate-sort-undecorate pattern is used when you want to sort a sequence using a different type of comparison then the one defined by the type of the elements. You create a sequence of pairs with the new "key" as first element and sort that instead, dropping the key afterwards. In this case you aren't doing anything like that. (btw: that pattern is "obsolete", in the sense that you don't have to do that explicitly anymore). – Bakuriu Jun 26 '14 at 16:59
  • 1
    Since the order does not matter, is the awk call really necessary ? Calling both commands and passing their outputs to sort and uniq (without the -c option) should be enough. – John WH Smith Jun 26 '14 at 18:46

awk-pipe to only let the 1st occurance of an input line pass thru:

( ./c1 ; ./c2 ) | awk '!u[$0]++'

This does not take time for sorting but needs a memory of seen lines. So for huge amounts of input sort and uniq may be better...


I would recommend utilizing sed to parse the text and remove duplicate lines. So the first command keeps the duplicate line sed '$!N; /^\(.*\)\n\1$/!P; D'

The second command will delete the duplicates sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'

  • Why [ -~]? What's so special about characters in the ASCII range, or whatever this ends up meaning in non-C locales? And do note that sed will end up storing the whole input in memory, which is slow and could fail if the input is large. Since the order of the output doesn't matter, sort -u is correct, and it's simpler, faster (for any input for which there is a measurable difference), and less memory-intensive (ditto). – Gilles Jun 27 '14 at 22:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.