16

I have a large file composed of text fields separated by semicolons in the form of a large table. It has been sorted. I have a smaller file composed of the same text fields. At some point, someone concatenated this file with others and then did a sort to form the large file described above. I would like to subtract the lines of the small file from the big one (i.e. for each line in the small file, if a matching string exists in the big file, delete that line in the big file).

The file looks roughly like this

GenericClass1; 1; 2; NA; 3; 4;
GenericClass1; 5; 6; NA; 7; 8;
GenericClass2; 1; 5; NA; 3; 8;
GenericClass2; 2; 6; NA; 4; 1;

etc

Is there a quick classy way to do this or do I have to use awk?

28

You can use grep. Give it the small file as input and tell it to find non-matching lines:

grep -vxFf file.txt bigfile.txt > newbigfile.txt

The options used are:

   -F, --fixed-strings
          Interpret PATTERN as a  list  of  fixed  strings,  separated  by
          newlines,  any  of  which is to be matched.  (-F is specified by
          POSIX.)
   -f FILE, --file=FILE
          Obtain  patterns  from  FILE,  one  per  line.   The  empty file
          contains zero patterns, and therefore matches nothing.   (-f  is
          specified by POSIX.)

   -v, --invert-match
          Invert the sense of matching, to select non-matching lines.  (-v
          is specified by POSIX.)
   -x, --line-regexp
          Select only those matches that exactly match the whole line.  
          (-x is specified by POSIX.)
  • Nice, worked perfectly. Thanks very much. – Escher Sep 29 '14 at 15:10
  • 1
    It's cool that it worked, but it seems to me that it would have been better with the -x option as well, in case a line in the smaller file happened to me a substring of another line in main file. Also, it's quite possible that @UlrichSchwarz's answer is faster. – rici Sep 30 '14 at 3:49
18

comm is your friend:

NAME comm - compare two sorted files line by line

SYNOPSIS comm [OPTION]... FILE1 FILE2

DESCRIPTION Compare sorted files FILE1 and FILE2 line by line.

   With  no  options, produce three-column output.  Column one contains lines unique to FILE1, column two contains
   lines unique to FILE2, and column three contains lines common to both files.

   -1     suppress column 1 (lines unique to FILE1)

   -2     suppress column 2 (lines unique to FILE2)

   -3     suppress column 3 (lines that appear in both files)

(comm will probably have a performance benefit over grep since it takes the sortedness into account.)

For example:

comm -1 -3 file.txt bigfile.txt > newbigfile.txt
  • 2
    Good point about using comm over grep for sorted lists. This would be a better answer if you gave a specific command line example such as comm -1 -3 file.txt bigfile.txt > newbigfile.txt – Steve Midgley Sep 30 '14 at 7:13
  • I confirm that I tryed grep command reported above with files around 100MB and I got a "killed" error. Trying with comm it finished successfully. – Gianluca Casati Mar 2 '18 at 18:07
  • Command redirection is useful for unsorted files or if you need more than two files: comm -1 -3 <(sort BAD.txt GOOD.txt) <(sort FILES.txt) – odinho - Velmont Jan 25 at 12:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.