I have a master file with more than 50000 lines and I need to remove around 20000 lines from actual file, for this I used grep -vE command which is showing an error as too many arguments.

I'm using sed command to remove those using for loop. But it is taking so much time to remove as 20000 sed commands are going to initialize.

for i in `cat 20000-words.txt`; do 
    sed -i -e "/$i/ d" 50000-lines.txt

Assuming that 20000-words.txt is already in the format of one word per line, do:

grep -vFf 20000-words.txt 50000-lines.txt >50000-filtered-lines.txt

The -f argument to grep tells it to read patterns from a file, one pattern per line, instead of taking them as command line arguments. The -F argument to grep tells it that the patterns should be used as literal strings rather than regular expressions.

  • 1
    This is ultimately super. It worked in rocket speed. Thank You so much @godlygeek – Sriharsha Kalluru Jul 24 '14 at 14:11
  • @SriharshaKalluru, awesome - please mark this answer as accepted by clicking the check mark to the left of it! – godlygeek Jul 24 '14 at 14:29

To do it without grep and assume that you don't have duplicated lines, you can:

$ sort 20000-words.txt 50000-lines.txt | uniq -u


$ comm -23 <(sort 50000-lines.txt) <(sort 20000-words.txt)
  • This assumes that both files are just lists of words - my reading of the OP's intent was that 50000-lines.txt contains lines with multiple words per line. But I could be wrong... – godlygeek Jul 24 '14 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.