2

I am trying to extract entries from within a large Genbank file, with many thousands of entries. For a search string, I'm using a unique gene name – that works fine. The tricky bit is that I'd like to print the entire entry for that particular gene – entries begin with the word LOCUS and end with //, and contain the gene name at some point between. I understand that I can use grep’s flags -A, -B, and -C to print n lines after/before a string match, but the actual entries are variable in length. How would I use grep to search for my string (gene name), and then print all the lines before the match up to and including a line beginning with "LOCUS", and all lines up to and including a line indicating the end of the entry, which is just "//"?

I'm open to all suggestions – is there a way to have the -A and -B flags match a strings ("LOCUS" and "//") or something to that effect? Should I be using awk instead?

Edit: This is a simplified input example – each record begins with "LOCUS" and ends with "//". This example contains three records:

LOCUS scaffold1|size100
/gene="gene1"
GATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACA
GATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACA
//
LOCUS scaffold99|size
/gene="gene2"
CGTTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACA
GATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACA
GATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACA
//
LOCUS scaffold199|size1000
/gene="gene3"
AGTTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACA
GATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACA
AGTTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACA
GATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACA
AGTTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACA
GATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACA
//

I would like to search for "gene2" and print the text from the first instance of "LOCUS" before the match through the first in "//" after the match. Ideally, I would like the following output:

LOCUS scaffold99|size
/gene="gene2"
CGTTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACA
GATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACA
GATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACA
//
  • The example input contains // instead of //... – Hauke Laging Apr 30 '18 at 7:15
2

This is fairly easy in awk:

awk -vtarget=fox '
    /LOCUS/ { in_gene = 1 }
    in_gene { if (gene == "") gene = $0; else gene = gene ORS $0; }
    $0 ~ target { found = 1 }
    /\/\//  { if (in_gene && found) print gene
              gene = ""; in_gene = 0; found = 0
            }
    '
  • Set the target variable to the string (gene name) you are searching for.  I used fox as an example.
  • When we see the word LOCUS, we know we’re looking at a gene.
  • As long as we’re looking at a gene, accumulate its contents.  The first line (the LOCUS line) just gets assigned to the gene variable.  Thereafter, we add (append) the current line ($0) to the gene variable with a newline (ORS = Output Record Separator) between the old value and the added value.
  • If the current gene contains the gene name you’re looking for, set the found flag.
  • We have to use the rather ugly /\/\// to search for a //.  When we see one, we check whether the current gene is the one we’re looking for, and, if so, print it.  Then reset to continue searching.  If you’re sure that the gene you’re looking for occurs only once in the file (or if you want only the first occurrence), you could just exit here.
  • "We have to use the rather ugly /\/\// to search for a //" – no. $0=="//" or $0 ~ "//" – Hauke Laging Apr 30 '18 at 7:21
  • "This is fairly easy in awk" – SeaBiscuit Nov 23 '18 at 9:49
2

When each Gene record is specifying that it's between LOCUS...//, then you could do:

gawk '/gene2/{printf $0 RS}' RS='\n//\n' infile

We defined the RS with a unique value each records ends with, say "\newline//\newline" (a line containing only //), then for each record matched with /gene2/ printf the record $0 and revert back RS.

Note: To manage the RS when if it was including leading/trailing whitespace (Spaces/Tabs), you can change to RS='\n( |\t)*//( |\t)*\n' but you need to use RT? (GNU awk extension) to keep the RS intact or either printf it directly "//".

gawk '/gene2/{printf $0 RT}' RS='\n( |\t)*//( |\t)*\n' infile

from man gawk:

RS The input record separator, by default a newline.

RT The record terminator. Gawk sets RT to the input text that matched the character or regular expression specified by RS.


?: After the end of the record has been determined, gawk sets the variable RT to the text in the input that matched RS. When RS is a single character, RT contains the same single character. However, when RS is a regular expression, RT contains the actual input text that matched the regular expression.

  • I love simplicity, and this is simplicity exemplified. It was noticed that the "sample" data included ending flags that might have trailing white space. Not unlikely as records may be collected from human-entered data which could easily include invisible white space on any line. Can the record separator, RS, be changed to accommodate any and all white space, including none, at the end of the line? – Gypsy Spellweaver May 1 '18 at 0:44
  • Thanks! I've run into a problem though! Instead of returning the expected result, I'm instead left with: /gene="symbB.v1.2.000186.t1" // /gene="symbB.v1.2.000186.t1" // But unfortunately, no actual gene information – anth May 1 '18 at 3:28
  • @anth – It sounds like you are using a Mac. This method works properly with GNU awk but not with Mac's default version of awk. If you install GNU awk on your Mac via homebrew, that'll work. – Gaultheria May 1 '18 at 4:58
  • @anth what is your awk version awk --version? and what distribution are you on? – αғsнιη May 1 '18 at 5:07
  • Sweetness, and it even works when the final entry is not newline terminated. At least it does with GNU Awk 4.1.3, API: 1.1. Too bad Mac has a different default setup. – Gypsy Spellweaver May 1 '18 at 6:43
1
sed -ne '
   /^LOCUS/,\|^//|!d
   H;/^LOCUS/h
   \|^/gene="gene2"|{
      s/.*//;x;H
   }
   \|^//|!d;g
   s/^\n//p
'       input_file

Working:

¶ The range operator "," can be used here

¶ First select the correct range which is locus begins and // ends the range.

¶ Store the lines in the hold space.

¶ When we encounter that golden line with gene2 we put a newline char in the beginning as a flag to remind us when the time comes to print or not to print.

¶ // line will trigger the printing activity based on whether we saw the newline in the front in the hold area.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.