3

I am parsing a file using grep and the output on screen contains newline, as here:

$ grep 'gene' sequence.gb
     gene            89..1483
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
     gene            complement(1987..2763)
                     /gene="nucleocapsid protein"
                     /gene="nucleocapsid protein"

I can assign this to a variable and print out still with the newlines:

$ gene=$(grep 'gene' sequence.gb)
echo "$gene"
     gene            89..1483
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
     gene            complement(1987..2763)
                     /gene="nucleocapsid protein"
                     /gene="nucleocapsid protein"

but this does not contain real newlines, since if I grep again for the lines containing '..' I get the whole lot:

$ echo "$gene" | grep '..'
     gene            89..1483
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
     gene            complement(1987..2763)
                     /gene="nucleocapsid protein"
                     /gene="nucleocapsid protein"

We can see that this is a single string by not using the quotes:

$ echo $gene
gene 89..1483 /gene="non-structural protein" /gene="non-structural protein" /gene="non-structural protein" /gene="non-structural protein" /gene="non-structural protein" /gene="non-structural protein" /gene="non-structural protein" gene complement(1987..2763) /gene="nucleocapsid protein" /gene="nucleocapsid protein"

So my question is, how can I maintain newline formatting or introducing it?

Thank you

5
  • 2
    For the purposes of accuracy, your last statement is incorrect. Removing the quotes merely means that the shell's natural expansion of gene will have all whitespace collapsed into a single space; it does not indicate what the composition of that white space would have been.
    – DopeGhoti
    May 22, 2018 at 17:58
  • While your problem comes from the meaning . in regexes, there is the semi-related issue that command substitution removes trailing newlines when returning the value. You can't see it here, since echo adds one for you. But if your command produced empty lines at the end, they'd be lost. (the grep here won't do that, of course.)
    – ilkkachu
    May 22, 2018 at 18:55
  • And as for what happens when you leave the double quotes out, see mywiki.wooledge.org/WordSplitting
    – ilkkachu
    May 22, 2018 at 19:11
  • I've submitted an edit to the title, which doesn't really capture what the problem is (grepping for literal '..' rather than for two characters). Linefeeds have nothing whatsoever to do with this.
    – Monty Harder
    May 22, 2018 at 22:31
  • You can easily see that they are newlines with echo "$var" | od -c (or xxd or hd etc) or echo "$var" | cat -n (or -E or -En if GNU coreutils) or echo "$var" | awk '{print NR,$0}' etc.
    – dave_thompson_085
    May 23, 2018 at 3:06

1 Answer 1

Reset to default
14

Because . is a regex wildcard, grep '..' matches every line that has at least two characters:

$ echo "$gene" | grep '..'
     gene            89..1483
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
                     /gene="non-structural protein"
     gene            complement(1987..2763)
                     /gene="nucleocapsid protein"
                     /gene="nucleocapsid protein"

In regular expressions, . is truly wild: it will match not just any letter or number but also any punctuation, blank, tab, or any other character at all.

To match just periods, use -F:

$ echo "$gene" | grep -F '..'
     gene            89..1483
     gene            complement(1987..2763)

-F is short for --fixed-strings and tells grep to treat the pattern as a fixed string, not a regular expression.

Alternatively, one can escape the periods so that they just match periods (hat tip: Nick):

$ echo "$gene" | grep '\.\.'
     gene            89..1483
     gene            complement(1987..2763)

Or else we can force grep to treat the periods as literal periods by putting them inside character classes (hat tip: dave_thompson):

$ echo "$gene" | grep '[.][.]'
     gene            89..1483
     gene            complement(1987..2763)

If you don't need regular expressions, though, use -F because it makes grep processing much faster.

4
  • 1
    .. or escape the special character: grep '\.\.' . This is less readable in this case, but escaping might be necessary in more complicated cases.
    – NickD
    May 22, 2018 at 18:30
  • @Nick Excellent suggestion. I just added escaped periods to the answer.
    – John1024
    May 22, 2018 at 19:17
  • 1
    If we're being completist, using a (trivial) character class grep '[.][.]' also suppresses the special meaning.
    – dave_thompson_085
    May 23, 2018 at 3:08
  • @dave_thompson_085 Completist is good. I added character classes to the answer also.
    – John1024
    May 23, 2018 at 3:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .