Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Say, I have two commands that return some text. For example:

$ ./c1
/usr/bin/foo
/usr/bin/bar
/usr/bin/baz
$ ./c2
/usr/bin/foo
/usr/bin/qux
/usr/bin/buzz
/usr/bin/bar

I want to remove the duplicate lines; i.e. the output will be (order isn't important):

/usr/bin/baz
/usr/bin/qux
/usr/bin/buzz

How would I go about doing this?

share|improve this question
1  
is the order does not matter then concatenate them together and pipe the result through sort -u –  rob Jun 26 at 15:14
    
Just to confirm: based on your example, you want to remove the lines that are present in both files, right? Many people have given answers that retain a single copy of each line, which is what the text of your question says. Please edit your question to make it consistent. –  Gilles Jun 28 at 7:29

4 Answers 4

up vote 5 down vote accepted

With comm from GNU coreutils:

$ comm -3 <(sort -u <(./c1)) <(sort -u <(./c2)) | tr -d '\t'
/usr/bin/baz
/usr/bin/buzz
/usr/bin/qux

From man comm:

Compare sorted files FILE1 and FILE2 line by line.

       With  no  options,  produce  three-column  output.  Column one contains
       lines unique to FILE1, column two contains lines unique to  FILE2,  and
       column three contains lines common to both files.

       -1     suppress column 1 (lines unique to FILE1)

       -2     suppress column 2 (lines unique to FILE2)

       -3     suppress column 3 (lines that appear in both files)
share|improve this answer
    
What's the point of using comm here? Why not sort the whole thing? –  Gilles Jun 27 at 22:33
    
@Gilles: What do you mean "sort the whole thing"? comm did its jobs, compare two sorted files. –  cuonglm Jun 28 at 2:05
    
Yes, but why sort the two files separately and then remove lines that appear in both, when a single application sort -u on both files would immediately give the desired result? This isn't even correct — you're suppressing lines that appear in both files instead of keeping one copy. You also aren't removing duplicates already present in one of the files, although this wasn't a clearly stated requirement. –  Gilles Jun 28 at 7:11
    
@Gilles: Oh, I don't think the case where a file has duplicated lines. I will add -u in my answer. I use comm because it works both with files and process substitution. With process substitution all thing have done immediately. In the case use only sort, you must spawn a process to run ./c1, then ./c2 then pipe it to sort. –  cuonglm Jun 28 at 7:26
    
Apologies: I've just reread the question, and the example isn't consistent with the text — you (alone) gave an answer that matches the example, whereas others answered based on the text. –  Gilles Jun 28 at 7:30

A fairly simple pipeline should do the trick:

(./c1; ./c2) | sort -u

The parentheses get stdout of both ./c1 and ./c2 into stdin of the sort command. The option -u prints only 1 of each group of matching lines.

Thanks to John WH Smith for noticing a simplification, and Bakuriu for an insight.

share|improve this answer
1  
No, it isn't related. The decorate-sort-undecorate pattern is used when you want to sort a sequence using a different type of comparison then the one defined by the type of the elements. You create a sequence of pairs with the new "key" as first element and sort that instead, dropping the key afterwards. In this case you aren't doing anything like that. (btw: that pattern is "obsolete", in the sense that you don't have to do that explicitly anymore). –  Bakuriu Jun 26 at 16:59
1  
Since the order does not matter, is the awk call really necessary ? Calling both commands and passing their outputs to sort and uniq (without the -c option) should be enough. –  John WH Smith Jun 26 at 18:46

awk-pipe to only let the 1st occurance of an input line pass thru:

( ./c1 ; ./c2 ) | awk '!u[$0]++'

This does not take time for sorting but needs a memory of seen lines. So for huge amounts of input sort and uniq may be better...

share|improve this answer

I would recommend utilizing sed to parse the text and remove duplicate lines. So the first command keeps the duplicate line sed '$!N; /^\(.*\)\n\1$/!P; D'

The second command will delete the duplicates sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'

share|improve this answer
    
Why [ -~]? What's so special about characters in the ASCII range, or whatever this ends up meaning in non-C locales? And do note that sed will end up storing the whole input in memory, which is slow and could fail if the input is large. Since the order of the output doesn't matter, sort -u is correct, and it's simpler, faster (for any input for which there is a measurable difference), and less memory-intensive (ditto). –  Gilles Jun 27 at 22:31

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.