1

I have two files, file1 is a subset of file2, which means all the lines in file1 can be found in file2 but some lines in file2 are not in file1. Now I want to find the different lines(or extra lines) between two files. Because of the content(contains long RNA sequences) of the file, it's not very feasible to sort the file, so I'm wondering if I can find the extra or different lines between two files without sorting.

I tried this command, but seems not working:

    grep -vxFf barcodes1.tsv barcodes.tsv > diff.csv

Below is a section of file1 ( which have 6113 lines):

AAACCTGAGAAAGTGG-1
AAACCTGAGCCACGCT-1
AAACCTGAGCCTATGT-1
AAACCTGAGCTGCAAG-1
AAACCTGCATTCGACA-1
AAACCTGCATTGGGCC-1
AAACCTGCATTTCACT-1
AAACCTGGTAACGTTC-1
AAACCTGGTATTCGTG-1
AAACCTGGTCGGCACT-1
AAACCTGTCACTATTC-1
AAACCTGTCAGGCGAA-1
AAACCTGTCCTAGTGA-1
AAACCTGTCGGATGTT-1
AAACCTGTCTGATTCT-1
AAACGGGAGTGAACAT-1
AAACGGGCACTCTGTC-1
AAACGGGGTAAGAGAG-1
AAACGGGGTACCTACA-1
AAACGGGGTTCAGACT-1
AAACGGGGTTTGACTG-1
AAACGGGTCAAAGTAG-1
AAACGGGTCAACCAAC-1
AAACGGGTCATGCATG-1
AAACGGGTCCTATTCA-1
AAACGGGTCGTATCAG-1
AAACGGGTCTATGTGG-1
AAAGATGAGCACAGGT-1
AAAGATGAGCAGGCTA-1
AAAGATGCAAACTGTC-1

Below is a section of file2(which have 6121 lines):

AAACCTGCAAGCGAGT-1
AAACCTGCACACAGAG-1
AAACCTGGTAAACACA-1
AAACCTGGTCGGATCC-1
AAACCTGGTCTCTTTA-1
AAACCTGGTTTAAGCC-1
AAACCTGTCAACACCA-1
AAACCTGTCTATCCCG-1
AAACCTGTCTTTAGGG-1
AAACGGGAGCCACCTG-1
AAACGGGAGCGTTTAC-1
AAACGGGAGCTGTTCA-1
AAACGGGAGTAGGCCA-1
AAACGGGAGTGCCATT-1
AAACGGGCAACACCTA-1
AAACGGGCAATAGCGG-1
AAACGGGCACAGATTC-1
AAACGGGCACAGTCGC-1
AAACGGGCAGATGAGC-1
AAACGGGCAGGAATCG-1
AAACGGGCAGGTCCAC-1
AAACGGGCAGGTGGAT-1
AAACGGGCATAAAGGT-1
AAACGGGGTAGCTCCG-1
AAACGGGGTAGCTGCC-1
AAACGGGGTTCAACCA-1
AAACGGGGTTTGGGCC-1
AAACGGGTCAAACAAG-1
AAACGGGTCGAACTGT-1
AAACGGGTCTTAACCT-1

It's not feasible to sort these two files. Therefore, what the command will be to the job without sorting?

  • 1
    @K7AAY edited the answer already. – Wendy Huang May 31 at 20:38
  • 1
    Don't mean to be argumentative, but sorting 7,500 lines of 80 characters each on my five year old desktop, which has a 4th Generation i7 at 4GHz, took less than a second, so an answer dependent on sorting may be very reasonable. – K7AAY May 31 at 20:56
  • @K7AAY time is not the problem, the problem is how to sort it based on the RNA sequences? Actually the files content I posted at the answer already sorted using "sort", but it's still different. – Wendy Huang May 31 at 20:59
  • 2
    Why does it seem that grep -vxFf is not working? Given your examples and grep -vxFf file1.tsv file2.tsv > diff.tsv results in file2.tsv and diff.tsv having the same content, because all lines are unique. Maybe you should edit your question and provide better sample data and tell us why grep doesn't work. – Freddy May 31 at 21:23
  • 1
    How to sort based on the RNA sequence? sort originalfile > sortedfile – K7AAY May 31 at 23:01
0

Instead of grep, may I substitute comm ? Tutorials may be helpful, but here's a one liner to do what you want:

comm -3 bigfile smallfile > differences
  • 2
    The command comm only works correctly for already "sorted" files (something the title of the question asks "not to do") – Isaac May 31 at 21:37
  • The original poster also commented "Actually the files content I posted at the answer already sorted using "sort", I noted the entire 6,121 line file takes less than a second to sort, which resolved the X,Y problem unix.meta.stackexchange.com/questions/5062/… . – K7AAY May 31 at 22:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.