1

I have two files, both of which I need to extract things from. The first file contains a barcoded line and ends with an OTU number. I need to extract lines that have specific OTU numbers.

Once I have the file of extracted lines, then I need to extract lines from my next file that match the barcodes from the first file.

For example, let's say I want to extract all of the lines that contain OTU_1 from this file enter image description here

There are unique barcodes associated with each line that contains OTU 1, in this example there are 5 shown: 

907.2::M02542:207:000000000-AWDAH:1:1115:18838:201661:N:0:GTGAAA
905.2::M02542:207:000000000-AWDAH:1:1101:24324:103291:N:0:GTGAAA
1205.2::M02542:207:000000000-AWDAH:1:2115:22195:238121:N:0:GTGAAA
906.2::M02542:207:000000000-AWDAH:1:1115:24086:126561:N:0:GTGAAA
910.2::M02542:207:000000000-AWDAH:1:1112:26236:215801:N:0:GTGAAA

I will need to use these barcodes to extract sequences from my next file:

enter image description here

As you can see, the barcodes start after > and I will need all of the information between the > (i.e. my sequences).

I have tried the obvious thing, which would be to use a spreadsheet type of software and sort by OTU #, but my files are too big (~ several billion lines long).

|improve this question
  • 2
    HI, How you were able to add pictures with 1 rep!! But could you please post the actual text instead of in snapshot? – αғsнιη Aug 1 '17 at 13:54
  • @AFSHIN: That restriction on new users applies only to Stack Overflow, Super User, Server Fault, and Ask Ubuntu — at least for the past four years. – G-Man Aug 2 '17 at 16:53
  • 1
    @Mara Cloutier: (1) AFSHIN is right; we really prefer that you post text as text. (2) Most of use aren’t geneticists (or whatever you are), so we aren’t familiar with your file formats.  You say “I will need all of the information between the > (i.e., my sequences),” and you show a file where (exactly) every other line begins with >.  Is this true in general?  Do barcode lines in the second file always appear in pairs?  You should address information like this in your question.  … (Cont’d) – G-Man Aug 2 '17 at 16:57
  • 1
    (Cont’d) …  (3) Some barcodes appear multiple times.  Do you want every occurrence? Please do not respond in comments; edit your question to make it clearer and more complete (including actual sample text, if at all possible). – G-Man Aug 2 '17 at 16:57
1

With GNU grep, something like this should work:

grep -o '\S\+\s\+OTU_1$' Fasta.readmap.uc | \
grep -o '^\S\+' | \
grep -f - -A 1 092016MABits-pr.fasta | \
grep -v '^>'

The -o makes grep output only the matching text. The -f - tells grep to search for the patterns piped in from STDIN. The -A 1 tells grep to show the line after the match. The final grep only matches lines that don't being with a ">".

|improve this answer
  • I think the last grep is missing option -A 1 to include the lines with the sequences – Philippos Aug 1 '17 at 14:53
  • @Philippos, Perhaps so. My assumption is that whatever text editor the OP used for those snapshots had word wrap turned on. – agc Aug 1 '17 at 15:50
  • I doubt that because (1) he wrote we wants to extract "information between the >", (2) the screenshot shows a scrollbar indicating no need to wrap and (3) AFAIK this fasta format carries the sequences after the > lines. – Philippos Aug 2 '17 at 6:23
  • @Philippos: I believe that you are mostly right — the second file has two-line records, so the grep needs -A 1.  But the first file appears to be line-wrapped, and I don’t see a scroll bar in either image.  (The orange bar at the top of the second image appears (to me) to be a progress bar.) – G-Man Aug 2 '17 at 16:58
  • @agc: I appreciate the appreciation, but you're thanking the wrong person: Philippos is the one who posted the Wikipedia link. – G-Man Aug 2 '17 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.