1

I want to perform what some data analysis software call an anti-join: remove from one list those lines matching lines in another list. Here is some toy data and the expected output:

$ echo -e "a\nb\nc\nd" > list1
$ echo -e "c\nd\ne\nf" > list2
$ antijoincommand list1 list2
a
b
  • 1
  • 2
    Does this answer your question? Is there a tool to get the lines in one file that are not in another? – muru May 25 at 7:16
  • @Muru, yes, that post provides the solutions presented in Terdon's answer. However, when I was searching for "bash anti-join" (the terminology I associate with this kind of process), I didn't find anything useful. My OP (which others have edited) stated that my explicit purpose in asking this question was to associate the term "anti-join" with the solutions, so that searching this term yields these solutions. Thanks. – Josh May 25 at 15:08
6

I wouldn't use join for this because join requires input to be sorted, which is an unnecessary complication for such a simple job. You could instead use grep:

$ grep -vxFf list2 list1
a
b

Or awk:

$ awk 'NR==FNR{++a[$0]} !a[$0]' list2 list1
a
b

If the files are already sorted, an alternative to join -v 1 would be comm -23

$ comm -23 list1 list2 
a
b
| improve this answer | |
  • Avoiding sort with grep is great for the toy data I provided. Thanks! In the real world, my file1 often has multiple columns of data, one of which is being used for the join. A modified version of your awk code would address this use case. – Josh May 24 at 13:46
  • 1
    @Josh yes, just change the $0 with $N where N is the field number you are joining on. – terdon May 24 at 13:47
  • 1
    This works even if the column numbers in file1 and file2 are different: like awk 'NR==FNR{++a[$2]} !a[$5]' list2 list1; quite usual for the tag file to be a different format to the main data. – Paul_Pedant May 24 at 14:14
1

One way to do this with the join utility is:

$ join -v 1 list1 list2
a
b
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.