Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

a.txt

cat a.txt
a
b
x
c

b.txt

cat b.txt
d
e
a
f

the q:

SOMEMAGICK *.txt
a

Q: how can I show only the lines that are in all *.txt files?

share|improve this question

6 Answers 6

up vote 4 down vote accepted

How about

cat *.txt | sort | uniq -c | egrep "^ +$(ls -1 *.txt | wc -l) "

And then to get the number of appearances stripped off, you could add ...

cat *.txt | sort | uniq -c | egrep "^ +$(ls -1 *.txt | wc -l) " | sed -re 's/^ +[0-9]+ //'

As per the comment by @Stephane the above won't work if a line appears multiple times within a single file. Here I sort and uniq each file first to avoid that:

for f in *.txt; do sort -u $f > $f.uniqd; done
cat *.uniqd | sort | uniq -c | egrep "^ +$(ls -1 *.uniqd | wc -l) " | sed -re 's/^ +[0-9]+ //'

Although now it's not a one-liner anymore. :)

share|improve this answer
    
sort *.txt | uniq -c | egrep "^ +$(/bin/ls -1 *.txt | wc -l) " –  gasko peter Sep 5 '13 at 18:48
    
this one worked, THANKS!! –  gasko peter Sep 5 '13 at 18:48
    
That doesn't work for lines that appear more than once in a file. –  Stéphane Chazelas Sep 5 '13 at 21:14
    
sort -u is a shortcut for sort | uniq. Also, cat * | sort can be shortened to sort *. –  200_success Sep 6 '13 at 15:04
awk 'FNR == 1 { FILENUM++ }
     SEEN[$0] == FILENUM - 1 { SEEN[$0] = FILENUM }
     END { for (s in SEEN) if (FILENUM == SEEN[s]) print s }' *.txt

Explanation

When reading the first line of each file, increment FILENUM, so that when reading the n th file, FILENUM is n.

When reading each line, count the number of files in which that line has been seen (but you only need to bother doing this if the line has been seen in every previous file).

When there is no more input to read, print all lines that have been seen in all files.

Caution: As with several of the solutions posted here, this one also has a weakness. According to the question, if any of the input files is empty, there is supposed to be no output at all. However, since awk is a line-oriented tool, it ignores empty files. That is, the FNR == 1 { FILENUM++ } fails to increment FILENUM for empty files.

With GNU awk, it's possible to fix this bug using the ARGIND built-in variable.

gawk 'SEEN[$0] == ARGIND - 1 { SEEN[$0] = ARGIND }
      END { for (s in SEEN) if (ARGIND == SEEN[s]) print s }' *.txt
share|improve this answer

Using GNU awk

awk '{
      x[$0][FILENAME]
     }
     END{
      num_files=ARGC-1;
      for (b in x)
       if (length(x[b]) == num_files) 
        print b
     }' a.txt b.txt c.txt
share|improve this answer
    
how to use this? :\ :D –  gasko peter Sep 5 '13 at 18:41
    
@gaskopeter, assuming you have 3 files abc.txt, b.txt, c.txt, invoke the command exactly as above –  1_CR Sep 5 '13 at 18:42
    
it gives syntax error at 'x[$0][FILENAME]' - using Ubuntu 12.04 –  gasko peter Sep 5 '13 at 18:44
    
@gaskopeter, it would appear you are running on older version of awk –  1_CR Sep 5 '13 at 18:54
    
@1_CR Ubuntu/Debian uses mawk by default. Perhaps you are using a gawk feature? –  jordanm Sep 5 '13 at 19:51

You could do:

export LC_ALL=C
sort -u a.txt |
  comm -12 - <(sort -u b.txt) |
  comm -12 - <(sort -u c.txt) |
  comm -12 - <(sort -u d.txt)

Which would be relatively efficient, but that's not easy to extend to an arbitrary number of files.

share|improve this answer

I like an easier solution using join:

join <(sort a.txt) <(sort b.txt)

This does work on your two input files, but may not behave as you expect on lines containing spaces, it will also output duplicate lines multiple times.

To remedy the second issue, just

join <(sort a.txt) <(sort b.txt) | uniq

The first is a bit more complicated, but I cheated a bit with the -t flag, to use a non-occuring charachter as field separator:

$ cat a.txt 
This test
foo bar
does work
$ cat b.txt 
This is a test
foo does not work
does work
$ join <(sort a.txt) <(sort b.txt) | uniq
does work work
foo bar does not work
This test is a test
$ join -t : <(sort a.txt) <(sort b.txt) | uniq
does work
share|improve this answer
    
This won't work for three files or more, then you would have to reside to the a loop as described by @slm –  Bernhard Sep 6 '13 at 6:55

For 2 files

This is no more complicated than making use of grep's ability to use a word list. For example:

$ grep -f b.txt a.txt 

Example

# a.txt
$ cat a.txt 
a
abc defg
de
bcd
xyz bcd
c

# b.txt
$ cat b.txt 
d
e bcd
a
f
bcd

# common lines to a.txt & b.txt
$ grep -Fxf b.txt a.txt
a
bcd

NOTE: Depending on the data you may need to add a | sort -u after the grep if any lines are duplicated in the files!

Details

-F, --fixed-strings
     Interpret PATTERN as a list of fixed strings, separated by newlines, 
     any of which is to be matched.  (-F is specified by POSIX.)

-x, --line-regexp
     Select only those matches that exactly match the whole line.  
     (-x is specified by POSIX.)

-f FILE, --file=FILE
     Obtain patterns from FILE, one per line.  The empty file contains 
     zero patterns, and therefore matches nothing.  (-f is specified by 
     POSIX.)

For 3 or more

You can use the fact that if you compare any one file to the others that what ever is common across all of them as compared to this one file, then all files must share this common line. Again using grep -f as above but this time we'll have to loop through the files using a for loop.

$ mf=""; for i in *.txt; do [ -z "$mf" ] && mf=$i && continue; grep -Fxf $mf $i;done | sort -u

If we add some additional files into the mix:

# c.txt
$ cat c.txt 
a
z
d bcd
e
q
bcd

# d.txt
$ cat d.txt 
a
z
e
z bcd
bcd

Running our code produces this:

$ mf=""; for i in *.txt;do [ -z "$mf" ] && mf=$i && continue; grep -Fxf $mf $i;done | sort -u
a
bcd
share|improve this answer
    
To get this to work with more than 2 input files, you are going to have to resort to process substitution –  1_CR Sep 5 '13 at 19:01
    
@1_CR - sorry I'm not following you in terms of the process sub. comment. I understand what you're saying by >2 .txt files. This approach could be modified so that you run it against 2+ files though. –  slm Sep 5 '13 at 19:05
    
I meant grep -f <(grep -f a.txt b.txt) c.txt, but I see that you seem to have solved it in a different way –  1_CR Sep 5 '13 at 19:59
    
@1_CR - thanks, makes sense. I wanted something that was more dynamic if there were *.txt files. –  slm Sep 5 '13 at 20:10
    
@200_success - thanks for pointing that out. Should be fixed now. –  slm Sep 6 '13 at 15:51

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.