Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a master file with more than 50000 lines and my requirement is to remove around 20000 lines from actual file, for this I used grep -vE command which is showing an error as too many arguments.

I'm using sed command to remove those using for loop. But it is taking so much time to remove as 20000 sed commands are going to initialize. I need help in doing it as fast as possible.

for i in `cat 20000-words.txt`; do 
    sed -i -e "/$i/ d" 50000-lines.txt
done
share|improve this question
    
Is 20000-words.txt contains each word a line? –  cuonglm Jul 24 at 14:05
    
@Gnouc : Yes it is having one word for one line and it is single word only. –  Sriharsha Kalluru Jul 24 at 14:06
1  
Please do not cross post: stackoverflow.com/questions/24935727/… –  fedorqui Jul 24 at 14:10

2 Answers 2

up vote 9 down vote accepted

Assuming that 20000-words.txt is already in the format of one word per line, do:

grep -vFf 20000-words.txt 50000-lines.txt >50000-filtered-lines.txt

The -f argument to grep tells it to read patterns from a file, one pattern per line, instead of taking them as command line arguments. The -F argument to grep tells it that the patterns should be used as literal strings rather than regular expressions.

share|improve this answer
1  
This is ultimately super. It worked in rocket speed. Thank You so much @godlygeek –  Sriharsha Kalluru Jul 24 at 14:11
    
@SriharshaKalluru, awesome - please mark this answer as accepted by clicking the check mark to the left of it! –  godlygeek Jul 24 at 14:29

To do it without grep and assume that you don't have duplicated lines, you can:

$ sort 20000-words.txt 50000-lines.txt | uniq -u

or:

$ comm -23 <(sort 50000-lines.txt) <(sort 20000-words.txt)
share|improve this answer
    
This assumes that both files are just lists of words - my reading of the OP's intent was that 50000-lines.txt contains lines with multiple words per line. But I could be wrong... –  godlygeek Jul 24 at 15:51

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.