The following regex example is from pg 22 of "Mastering Regular Expressions".
'\<([a-z]+) +\1\>'
I'm a novice in regular expressions, but my impression, though the book does not make this clear, is that the \<
and the \>
are not standard regex symbols. From searching, it seems these are GNU extensions to the grep
regular expression syntax. Possibly it's not specific to grep
- I'm not sure.
In any case, it's not clear what these symbols mean. The rest of this question quotes different definitions, or attempts at definitions. Some of which are clearly wrong, incomprehensible, or at least incomplete. If anyone can give me a precis definition I'd appreciate it. I'd say the last one quoted, in https://www.grymoire.com/Unix/Regular.html#uh-9, is the most likely to be correct. But official documentation it is not.
The book says
You can use the (perhaps odd looking) metasequences
\<
and\>
[...]. You can think of them as word-based versions of^
and$
that match the position at the start and end of a word, respectively.
and then later
The “start of a word” is simply the position where a sequence of alphanumeric characters begins; “end of word” is where such a sequence ends. Figure 1-2 on the next page shows a sample line with these positions marked.
Figure 1-2 is quite helpful.
It's also "documented" in The GNU findutils manual which says:
‘\<’ matches the beginning of a word
‘\>’ matches the end of a word
Also in The GNU grep manual it says:
'\<'
Match the empty string at the beginning of word.
'\>'
Match the empty string at the end of word.
I don't know what either of these descriptions means. So, neither of these two extracts from GNU manuals is helpful.
At the time of originally writing this question, I had not read the "Mastering Regular Expressions" section carefully enough, had not seen Figure 1-2, and thought that these symbols meant that there was a whitespace character either preceding or succeeding the "word". I now realise this is wrong. However, even this book description is incorrect/incomplete.
Consider these two examples:
grep --color -E -i '\<([a-z]+) +\1\>' <<< 'wibble someword someword-something else wibble'
"someword someword" is matched here.
grep --color -E -i '\<([a-z]+) +\1\>' <<< 'wibble someword someword_something else wibble'
Here nothing is matched.
The book doesn't explain this, since it says the "end of word" is where an alphanumeric sequence ends. Neither do the extracts from the GNU manuals.
A possible explanation is provided in https://www.grymoire.com/Unix/Regular.html#uh-9 (found in a random search) which says:
Searching for a word isn't quite as simple as it at first appears. The string "the" will match the word "other". You can put spaces before and after the letters and use this regular expression: " the ". However, this does not match words at the beginning or end of the line. And it does not match the case where there is a punctuation mark after the word.
The character before the "t" must be either a new line character, or anything except a letter, number, or underscore. The character after the "e" must also be a character other than a number, letter, or underscore or it could be the end of line character.
I don't know whether the author got this from, but assuming it's true, it would explain the behavior I'm seeing. But it still seems rather arbitrary. -
isn't a punctuation mark. Why can't it be included as part of a word? Or, to put it differently, why does the hyphen match
the end of a word, and the underscore doesn't? In fact, the hyphen is much more common in natural languages than the underscore is. Though perhaps the underscore is used in programming languages.
And if this is correct, then the GNU documentation really should document it properly. If I have this straight, would a bug report be in order?
Ideally this feature would be customizable. But perhaps that's asking too much.
This extract from the GNU grep
code, suggests that the description from www.grymoire.com may be correct. The relevant code is in function init_word_char
in 'lib/regcomp.c', and looks like
general_case:
for (; i < BITSET_WORDS; ++i)
for (j = 0; j < BITSET_WORD_BITS; ++j, ++ch)
if (isalnum (ch) || ch == '_')
dfa->word_char[i] |= (bitset_word_t) 1 << j;
The important line here is, of course line 983 of the file:
if (isalnum (ch) || ch == '_')
I.e. the character is either alphanumeric or an underscore.
I don't understand what most of this code means, of course.