Still having issues trying to remove lines in a LARGE file containing strings listed in another LARGE file.

  grep -vwFf file1 file2 - FAILS due to memory exhaustion.

I have used:

  comm -23 file1 file2


It works for sorted files and really large files - but it only works for removing duplicate lines - not removing lines containing a string.

The two large files I have are sorted so that the strings I am searching for are at the begining of every line:

text file:

  AAAAA blah blah blah
  AAAAB blas blas blas
  CCCCC sdf sf sdf

string file




1 Answer 1


Why do you not burst $file1 into many smaller ones (in /tmp or use mktemp), then loop over each $file1, using it for grep ... what the ideal size of the pattern file ($file1) is, depends on your system.

Here, each $file1 will have 1000 lines.

i=1 while [ $i -lt $count ] do sed -n "$i,$(($i +1000))p" file1.txt >> /tmp/file${1}.txt i=$(( $i + 1001)) done

Now you have a bunch of files in /tmp named file.txt, so you do:

for file1 in $(ls /tmp/file*.txt) do grep -vwFf $file1 file2 done

Safer with mktemp:


for file1 in $(ls ${TEMP_DIR}/file*.txt) do grep -vwFf $file1 file2 done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.