1

I have two files, file1 is a subset of file2, which means all the lines in file1 can be found in file2 but some lines in file2 are not in file1. Now I want to find the different lines(or extra lines) between two files. Because of the content(contains long RNA sequences) of the file, it's not very feasible to sort the file, so I'm wondering if I can find the extra or different lines between two files without sorting.

I tried this command, but seems not working:

    grep -vxFf barcodes1.tsv barcodes.tsv > diff.csv

Below is a section of file1 ( which have 6113 lines):

AAACCTGAGAAAGTGG-1
AAACCTGAGCCACGCT-1
AAACCTGAGCCTATGT-1
AAACCTGAGCTGCAAG-1
AAACCTGCATTCGACA-1
AAACCTGCATTGGGCC-1
AAACCTGCATTTCACT-1
AAACCTGGTAACGTTC-1
AAACCTGGTATTCGTG-1
AAACCTGGTCGGCACT-1
AAACCTGTCACTATTC-1
AAACCTGTCAGGCGAA-1
AAACCTGTCCTAGTGA-1
AAACCTGTCGGATGTT-1
AAACCTGTCTGATTCT-1
AAACGGGAGTGAACAT-1
AAACGGGCACTCTGTC-1
AAACGGGGTAAGAGAG-1
AAACGGGGTACCTACA-1
AAACGGGGTTCAGACT-1
AAACGGGGTTTGACTG-1
AAACGGGTCAAAGTAG-1
AAACGGGTCAACCAAC-1
AAACGGGTCATGCATG-1
AAACGGGTCCTATTCA-1
AAACGGGTCGTATCAG-1
AAACGGGTCTATGTGG-1
AAAGATGAGCACAGGT-1
AAAGATGAGCAGGCTA-1
AAAGATGCAAACTGTC-1

Below is a section of file2(which have 6121 lines):

AAACCTGCAAGCGAGT-1
AAACCTGCACACAGAG-1
AAACCTGGTAAACACA-1
AAACCTGGTCGGATCC-1
AAACCTGGTCTCTTTA-1
AAACCTGGTTTAAGCC-1
AAACCTGTCAACACCA-1
AAACCTGTCTATCCCG-1
AAACCTGTCTTTAGGG-1
AAACGGGAGCCACCTG-1
AAACGGGAGCGTTTAC-1
AAACGGGAGCTGTTCA-1
AAACGGGAGTAGGCCA-1
AAACGGGAGTGCCATT-1
AAACGGGCAACACCTA-1
AAACGGGCAATAGCGG-1
AAACGGGCACAGATTC-1
AAACGGGCACAGTCGC-1
AAACGGGCAGATGAGC-1
AAACGGGCAGGAATCG-1
AAACGGGCAGGTCCAC-1
AAACGGGCAGGTGGAT-1
AAACGGGCATAAAGGT-1
AAACGGGGTAGCTCCG-1
AAACGGGGTAGCTGCC-1
AAACGGGGTTCAACCA-1
AAACGGGGTTTGGGCC-1
AAACGGGTCAAACAAG-1
AAACGGGTCGAACTGT-1
AAACGGGTCTTAACCT-1

It's not feasible to sort these two files. Therefore, what the command will be to the job without sorting?

7
  • 1
    @K7AAY edited the answer already. May 31, 2019 at 20:38
  • 1
    Don't mean to be argumentative, but sorting 7,500 lines of 80 characters each on my five year old desktop, which has a 4th Generation i7 at 4GHz, took less than a second, so an answer dependent on sorting may be very reasonable.
    – K7AAY
    May 31, 2019 at 20:56
  • 2
    Why does it seem that grep -vxFf is not working? Given your examples and grep -vxFf file1.tsv file2.tsv > diff.tsv results in file2.tsv and diff.tsv having the same content, because all lines are unique. Maybe you should edit your question and provide better sample data and tell us why grep doesn't work.
    – Freddy
    May 31, 2019 at 21:23
  • 1
    How to sort based on the RNA sequence? sort originalfile > sortedfile
    – K7AAY
    May 31, 2019 at 23:01
  • 1
    What do you mean by 'already sorted using "sort", but it's still different'? Different in what way? I feel like this is an X-Y problem. Sorting seems the simplest way to go.
    – Sparhawk
    Jun 1, 2019 at 1:06

2 Answers 2

1

This one liner should do the trick, And it takes care that the output file is not sorted:

cat -n barcodes1.tsv barcodes.tsv | sort -uk2 | sort -nk1 | cut -f2- > diff.csv

Basically, what it does is:

  • Concatenates the file.
  • Removes the duplicates.
  • Outputs in the desired file which won't be sorted.
0

Instead of grep, may I substitute comm ? Tutorials may be helpful, but here's a one liner to do what you want:

comm -3 bigfile smallfile > differences
2
  • 2
    The command comm only works correctly for already "sorted" files (something the title of the question asks "not to do")
    – user232326
    May 31, 2019 at 21:37
  • The original poster also commented "Actually the files content I posted at the answer already sorted using "sort", I noted the entire 6,121 line file takes less than a second to sort, which resolved the X,Y problem unix.meta.stackexchange.com/questions/5062/… .
    – K7AAY
    May 31, 2019 at 22:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .