Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

We have two lists.

A bigger "A":

A=`echo -e '1\n2\n3\n4\n5'`
echo "$A"
1
2
3
4
5

and a smaller "B":

B=`echo -e '1\n2\n3'`
echo "$B"
1
2
3

Q: But we need a third list that contains all the elements of "A", but doesn't have any of "B", how do I do it in bash?

echo "$C"
4
5

The numbers could be anything, from "foo" to 99, etc..

UPDATE:

It's working in the shell by hand, but it's strange because if I put it in a script, it doesn't works!

cat a.txt 
A=$(seq 5)
B=$(seq 3)
comm -23 <(sort <<< "$A") <(sort <<< "$B")
sh a.txt 
a.txt: line 3: syntax error near unexpected token `('
a.txt: line 3: `comm -23 <(sort <<< "$A") <(sort <<< "$B")'

doing it by hand it works..:

A=$(seq 5)
B=$(seq 3)
comm -23 <(sort <<< "$A") <(sort <<< "$B")
4
5

Why? update on update: Need to use bash instead of "sh" :D

share|improve this question
2  
Are the lists sorted? –  Mark Plotnick Apr 28 at 15:09
2  
Are the elements unique within each list? –  Stéphane Chazelas Apr 28 at 15:13
    
This is a very strange feeling, but I know how to do this easily in PowerShell, but can't think how off the top of my head in *sh, which I've used a lot more :/ –  Joel Coehoorn Apr 29 at 2:07
    
Your examples are sorted, but would not be for larger parameters to seq (i.e. > 9). Is keeping the original ordering important? –  Anthon Apr 29 at 4:48
    
it would be a nice thing to have the original order :\ –  evachristine Apr 29 at 6:25

4 Answers 4

up vote 12 down vote accepted

The comm command is what you need:

$ A=$(seq 5)
$ B=$(seq 3)
$ comm -23 <(sort <<< "$A") <(sort <<< "$B")
4
5

Here's a method that does not require the input to be sorted. This is a common idiom in awk that reads the first file into memory, and then does some filtering on the 2nd file based on the 1st. Let's try with randomized data

$ A=$(seq 5 | sort -R); echo "$A"
3
5
1
2
4
$ B=$(seq 3 | sort -R); echo "$B"
2
1
3

We expect the output to be 5 then 4:

$ awk 'NR==FNR {b[$1]=1; next} !($1 in b) {print}' <(echo "$B") <(echo "$A")
5
4
share|improve this answer
1  
I wanted to say a harder word for "wooow", but I don't want to get banned. :) THANKS! –  evachristine Apr 28 at 15:27
    
What's that <<< operator? –  Brilliand Apr 28 at 23:03
    
It is a bash "here-string" -- see Redirections in the manual. –  glenn jackman Apr 29 at 0:19

As glenn jackman provided, the comm utility is the simplest way of doing this. However that method destroys the sort order.

There is another way of accomplishing this that preserves the original sort order (though both lists must be pre-sorted in the same order):

diff --unchanged-line-format '' --old-line-format '' file_a file_b

This will return all lines that are unique to file_b in their original order.
 

I believe this would also be more efficient if the dataset were very large as well. As a sort operation can be expensive. But this is just a guess.

share|improve this answer
sort a b b | uniq -u

Older than the hills (UNIX 7), but still works.

share|improve this answer
1  
Nice hack! But for variables you'll have to split them into lines for sort. The following assumes the elements are separated by spaces. tr puts them into lines and $(..) puts them back into one line. echo A - B is $(sort <(tr ' ' '\012' <<<"$A $B $B") | uniq -u) –  Matthew Hannigan Apr 29 at 6:05

Or, you know, Perl:

#!/usr/bin/perl -s
if($#ARGS == 0) {print "Usage: $0 -exclude=fileWithLinesToExclude [inputFile]\n"; exit(0)}
open(EXCL, $exclude);
%excluded = map { $_ => 1 } <EXCL>;
while(<>) {
   print $_ unless $excluded{$_};
} 

Things

  • perl -s permits switches to become variable values
  • There is no chomping going on; if the exclude line is "foobar_" and the processed line is "foobar", it will not exclude it.
  • No sorting going on, except conceivably the hash insertion, so the file to process can be as large as you like, or a stream of stuff, whatever.
  • Pass the input filename after the exclude switch or just pipe stuff in.
share|improve this answer
    
Thank you very much! –  evachristine May 3 at 19:08

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.