I have a text file, say sitelist1.txt (File A), that has some URLs like below:


I have another text file, say sitelist2.txt (File B), that has numerous existing URLs.

I tried:

fgrep -v -f sitelist1.txt sitelist2.txt 

The problem is - This also displays the URLs that are present in sitelist2.txt and not in sitelist1.txt

  • I think removing the -v will get you what you want. – Munir Mar 28 '16 at 15:00
  • I want to see only those URLs from sitelist1.txt that are not present on sitelist2.txt. – Koshur Mar 28 '16 at 15:03
  • 4
    In that case, the order of files in your grep is incorrect. It should be fgrep -v -f sitelist2.txt sitelist1.txt Also, your question is unclear as to which file is fileA and which is fileB. I suggest you edit it to clarify that. – Munir Mar 28 '16 at 15:37

The -v you passed is making it do the opposite of what you intend - it is showing you, for each line in sitelist1.txt, all lines in sitelist2.txt that are not matches. So you're seeing a lot of duplicates, I would bet. You want to use the same command without the -v option:

fgrep -f sitelist1.txt sitelist2.txt

That will execute the following english directive: For each line in sitelist1.txt, show me all lines in sitelist2.txt that contain the file 1 line I'm interested in, as the entire line or as part of the file 2 line.

The -v option is an "invert match" option, which shows you all non-matching lines.

  • I am interested to see only those URLs from sitelist1.txt that are not present on sitelist2.txt. Maybe I should reframe the question. – Koshur Mar 28 '16 at 15:04
  • Updated the question. Want to see which strings from sitelist1.txt are not present in sitelist2.txt. – Koshur Mar 28 '16 at 15:15

John's answer (beside the misunderstanding pointed out in the comments under the question) is the answer to the question, however I'd like to also point out that for simple operations like this one combine (part of moreutils) is a good choice; it's very easy to use as it allows to express the operation to be done using boolean operators (AND, OR, NOT and XOR supported):

combine sitelist1.txt NOT sitelist2.txt
% cat sitelist1.txt 
% cat sitelist2.txt 
% combine sitelist1.txt NOT sitelist2.txt

Contrarily to most utilities the input files don't need to be sorted (though I'd guess they're still sorted internally), however as when using fgrep there is a pitfall in case of duplicate lines in sitelist1.txt that one doesn't want to be printed. If sitelist1.txt contains duplicate lines and you don't want to print them you'll have to pipe combine's output to sort -u:

combine sitelist1.txt NOT sitelist2.txt | sort -u

If you are looking for complete urls in each line and not substrings, you could use comm

comm -23 <(sort -u fileA) <(sort -u fileB)

comm expects sorted input, so we first sort and unique-fy the 2 files (sort -u file).
Next - use process substitution to pass the output of sort -u as a "file" to comm, since comm expects files: comm <() <().
Last - suppress columns 2 (lines unique to file B) and 3 (lines common to both files) to output only lines in column 1 (lines unique to file A).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.