I have a large file composed of text fields separated by semicolons in the form of a large table. It has been sorted. I have a smaller file composed of the same text fields. At some point, someone concatenated this file with others and then did a sort to form the large file described above. I would like to subtract the lines of the small file from the big one (i.e. for each line in the small file, if a matching string exists in the big file, delete that line in the big file).

The file looks roughly like this

GenericClass1; 1; 2; NA; 3; 4;
GenericClass1; 5; 6; NA; 7; 8;
GenericClass2; 1; 5; NA; 3; 8;
GenericClass2; 2; 6; NA; 4; 1;


Is there a quick classy way to do this or do I have to use awk?


2 Answers 2


You can use grep. Give it the small file as input and tell it to find non-matching lines:

grep -vxFf file.txt bigfile.txt > newbigfile.txt

The options used are:

   -F, --fixed-strings
          Interpret PATTERN as a  list  of  fixed  strings,  separated  by
          newlines,  any  of  which is to be matched.  (-F is specified by
   -f FILE, --file=FILE
          Obtain  patterns  from  FILE,  one  per  line.   The  empty file
          contains zero patterns, and therefore matches nothing.   (-f  is
          specified by POSIX.)

   -v, --invert-match
          Invert the sense of matching, to select non-matching lines.  (-v
          is specified by POSIX.)
   -x, --line-regexp
          Select only those matches that exactly match the whole line.  
          (-x is specified by POSIX.)
  • Nice, worked perfectly. Thanks very much.
    – Escher
    Commented Sep 29, 2014 at 15:10
  • 1
    It's cool that it worked, but it seems to me that it would have been better with the -x option as well, in case a line in the smaller file happened to me a substring of another line in main file. Also, it's quite possible that @UlrichSchwarz's answer is faster.
    – rici
    Commented Sep 30, 2014 at 3:49

comm is your friend:

NAME comm - compare two sorted files line by line


DESCRIPTION Compare sorted files FILE1 and FILE2 line by line.

   With  no  options, produce three-column output.  Column one contains lines unique to FILE1, column two contains
   lines unique to FILE2, and column three contains lines common to both files.

   -1     suppress column 1 (lines unique to FILE1)

   -2     suppress column 2 (lines unique to FILE2)

   -3     suppress column 3 (lines that appear in both files)

(comm will probably have a performance benefit over grep since it takes the sortedness into account.)

For example:

comm -1 -3 file.txt bigfile.txt > newbigfile.txt
  • 2
    Good point about using comm over grep for sorted lists. This would be a better answer if you gave a specific command line example such as comm -1 -3 file.txt bigfile.txt > newbigfile.txt Commented Sep 30, 2014 at 7:13
  • I confirm that I tryed grep command reported above with files around 100MB and I got a "killed" error. Trying with comm it finished successfully. Commented Mar 2, 2018 at 18:07
  • Command redirection is useful for unsorted files or if you need more than two files: comm -1 -3 <(sort BAD.txt GOOD.txt) <(sort FILES.txt) Commented Jan 25, 2019 at 12:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .