10

I'm sure some version of this question has been asked and answered before, but I've looked around and haven't found the exact answer. Perhaps someone here can help the lightbulb go on for me. I'm on a Mac with Mojave 10.14.6 and bash 3.2.57(1)-release.

I'm learning the basics of regular expressions by following along with an online tutorial, and practicing both on the online site https://regexr.com, and by using grep in bash on my local machine.

I'm practicing with a small text file (called small.txt) with three things in it:

9.00
9-00
9500

I understand that the . wildcard will match any one character at that spot. So, in the online regex engine (JavaScript) that I'm using /9.00/g will match all three strings 9.00 9-00 and 9500.

It's the same if I use grep on the command line:

~/bin $ grep 9.00 small.txt
9.00
9-00
9500

So far, so good. The tutorial says that to turn the . from a metacharacter into a literal, you have to escape it. Okay. so putting /9\.00/g into the online regex box will only match 9.00, as expected, not 9-00 nor 9500. Great.

However, if I enter that same syntax into grep on the command line, I get an unexpected result:

~/bin $ grep 9\.00 small.txt
9.00
9-00
9500

Same as before. To get grep to work, I either have to double quote the whole string:

~/bin $ grep "9\.00" small.txt
9.00

or just double quote the escaped character:

~/bin $ grep 9"\."00 small.txt
9.00

There may well be some other quoting choices that I could make that would also give me the correct result.

This is making it hard for me to wrap my head around the basics of regular expression, because, clearly, I first have to understand how grep in the shell differs from traditional regular expression syntax. It's hard enough learning all of the rules for regular expressions, but when you add in the differences between classic regular expression and the behavior of the bash shell, my head explodes.

Anyway, wondering if there was a clear explanation that will clear this up for me and set me on the path to properly learning regular expressions that I can use with grep on the command line.

(None of the courses on regular expression point out the differences between the command line version of grep with bash, and the "pure" regular expression syntax that you see on the online regex testers.) I know that there are differences between engines at the advanced level, but this seems to be something so basic, that I feel that I must be missing something.

Thanks.

  • 8
    You're running into issues involving the shell itself. Try echo \. vs echo "\." vs echo \\. and notice the differences. That might help understand why "9\.00" does what you expect but 9\.00 does not. – Andy Dalton Apr 29 at 23:01
  • 6
    The shell processes the parameters before grep gets to see them. You don't need to learn how regexes in grep are different to regexes in JavaScript only, you also need to learn how the shell works. – choroba Apr 29 at 23:01
  • Okay, got it. one \ so that grep treats . as a literal rather than a meta character, and another \ so that the shell actually passes \. along for grep to see in the first place. I'll have to dig into this a bit more. Thanks. – dbates Apr 29 at 23:14
  • that is not quite it ... the second \ is literal ... it does not escape the . – jsotola Apr 30 at 0:29
  • 1
    see mywiki.wooledge.org/Quotes to learn about various quoting in bash, wrt resources for learning regex specific to grep, I have book (currently free) if you are using GNU grep - github.com/learnbyexample/learn_gnugrep_ripgrep (though there are a few things I need to update/correct) – Sundeep Apr 30 at 6:19
13

Why? because your shell interprets some special characters, such as \ in your example.

You are running into troubles because you do not protect the string that you try to pass as argument to grep via the Shell.

Several solutions:

  • singlequoting the string,
  • doublequoting the string (with doublequoting the shell will interpret several things, such as $variables , before sending the resulting string to the command),
  • or not use quoting (which I strongly advise against) but add backslashes in the right places to prevent the shell to interpret the next characters before sending it to the command.

I recommend to protect the string via single quotes, as it keeps almost everything literraly:

grep '9\.0' #send those 4 characters to grep in a single argument

The Shell pass the singlequoted string literally.

Note: The only thing you can't include inside a single quoted shell string is a single quote (as this ends the singlequoting). To include a singlequote inside a singlequoted shell string, you need to first end the singlequoting, immediately add an escaped singlequote \' (or one between doublequotes: "'" ) and then immediately reenter the singlequoting to continue the single quoted string : for exemple to have the shell execute the command grep a'b , you could write the parameter as 'a'\''b' so that the shell sends a'b to grep: so write: grep 'a'\''b' , or grep 'a'"'"'b'

If you insist on not using quoting, you need your shell to have a \\ to have it send a \ to grep.

grep 9\\.0  # ie: a 9, a pair \\, a ., and a 0 , and the shell interprets the pair \\ into a literal \

If you use doublequotes: you need to take into account that the shell will interprets several things first ($vars, \, etc). for exemple when it sees an unescaped or unquoted \, it waits the next character to decide how to interpret it. \w is seen as a single letter w, \\ is seen as a single letter \, etc.

grep "9\\.0"  # looks here the same as not quoting at all... 
    #but doublequoting allows you to have spaces, etc, inside the string
| improve this answer | |
  • 1
    So it sounds like one way to try to learn this stuff is to follow along with the online resources like regex.com and learn the regular expression syntax there, but put everything inside single quotes when I'm trying it out with grep on the command line (unless I need to actually grep a single quote, then I'll have to jump through the hoops described above.) That way, I can learn one set of syntax commands, and it should work reasonably well with both the online sites and grep at the command line. At least I'll get some consistency as I try to learn the basics. – dbates Apr 30 at 17:40
  • @dbates: you got it :) single quotes are rarely part of the regexp, so just singlequoting the regexp should pass it "as is" to the grep (or rather, egrep) command. – Olivier Dulac Apr 30 at 22:01
  • 2
    @dbates: another thing to be careful about: there are many "flavors" of regular expressions, and also different implementations of each flavor... sometimes you'll to a s/first\(second\)thirdregexp/\1/ to keep just the thing matched by "second" (as a regexp), sometimes you will have to use non-backslashed parenthesis to do the same. and maaany other variations. I like the informations on : regular-expressions.info, and a cool tool (regular-expressions.info/regexbuddy.html) "emulates the regular expression flavors of 262 applications and programming languages" – Olivier Dulac Apr 30 at 22:23
  • @dbates: as for the shell itself, you may want to have a thorough read of several informations on : mywiki.wooledge.org (read the whole bashFAQ, then the marvellous bashpitfalls part (which applies to many other shells as well), and then the bashGuide). all good stuff. – Olivier Dulac Apr 30 at 22:26
  • Concerning different "flavors" of regex: by default, grep uses "Basic Regular Expression" (BRE) syntax, which is... very basic. grep -E will use "Extended Regular Expression" (ERE) syntax, which is actually still pretty basic. Some versions of grep accept a -P option, which invokes "Perl-Compatible Regular Expression" (PCRE) syntax, which is much closer to most other modern flavors. – Gordon Davisson Apr 30 at 23:27
9

Turning the comments into an answer:

The problem is that \ is the escape character both for regexes and the shell. \. is to the shell the same as '.'. echo and set -x help understand, what the shell does:

> echo \.
.

> echo '\.'
\.

> echo \\.
\.


> set -x
> echo 9_00 | grep 9\.00
+ echo 9_00
+ grep 9.00
9_00

So if the command shall see the \ then it has to be protected by quotes or a second \.

| improve this answer | |
2

To add to the other answer and comments, another thing that you can do to get grep to return what you want is to use the following:

grep -F 9.00 small.txt

Output:

9.00

The -F makes grep see the pattern as a fixed string and not a regular expression so that it will only return lines with that exact string. Because of this, you don't even need to escape the . or use quotes because it will only match 9.00 exactly instead of seeing the . as any character.

| improve this answer | |
1

Why do I have to quote an escaped character in a regular expression for grep, but not on online regex engines?

You don't have to quote it for grep, but for the shell.

Using grep -f to read the pattern from a file shows that the 9\.00 pattern you showed works fine when it's not passed through the shell.

$ cat re.txt 
9\.00
$ grep -f re.txt small.txt 
9.00

The fact that the issue is not grep itself is probably why you don't see it in articles about regexes. You might see the relevant points on an article about how the shell works, though...

I know that there are differences between engines at the advanced level

Doesn't have to be too advanced even. Something like + already works differently in BRE vs. ERE. Also at least some online tools default to Perl regexes or similar, which have many features not in the standard regexes.

See:

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.