The meaning of the \< and \> regex symbols in GNU grep

Question

The following regex example is from pg 22 of "Mastering Regular Expressions".

'\<([a-z]+) +\1\>'

I'm a novice in regular expressions, but my impression, though the book does not make this clear, is that the \< and the \> are not standard regex symbols. From searching, it seems these are GNU extensions to the grep regular expression syntax. Possibly it's not specific to grep - I'm not sure.

In any case, it's not clear what these symbols mean. The rest of this question quotes different definitions, or attempts at definitions. Some of which are clearly wrong, incomprehensible, or at least incomplete. If anyone can give me a precis definition I'd appreciate it. I'd say the last one quoted, in https://www.grymoire.com/Unix/Regular.html#uh-9, is the most likely to be correct. But official documentation it is not.

The book says

You can use the (perhaps odd looking) metasequences \< and \> [...]. You can think of them as word-based versions of ^ and $ that match the position at the start and end of a word, respectively.

and then later

The “start of a word” is simply the position where a sequence of alphanumeric characters begins; “end of word” is where such a sequence ends. Figure 1-2 on the next page shows a sample line with these positions marked.

Figure 1-2 is quite helpful.

It's also "documented" in The GNU findutils manual which says:

‘\<’ matches the beginning of a word
‘\>’ matches the end of a word

Also in The GNU grep manual it says:

'\<'
    Match the empty string at the beginning of word.
'\>'
    Match the empty string at the end of word.

I don't know what either of these descriptions means. So, neither of these two extracts from GNU manuals is helpful.

At the time of originally writing this question, I had not read the "Mastering Regular Expressions" section carefully enough, had not seen Figure 1-2, and thought that these symbols meant that there was a whitespace character either preceding or succeeding the "word". I now realise this is wrong. However, even this book description is incorrect/incomplete.

Consider these two examples:

grep --color -E -i '\<([a-z]+) +\1\>' <<< 'wibble someword someword-something else wibble'

"someword someword" is matched here.

grep --color -E -i '\<([a-z]+) +\1\>' <<< 'wibble someword someword_something else wibble'

Here nothing is matched.

The book doesn't explain this, since it says the "end of word" is where an alphanumeric sequence ends. Neither do the extracts from the GNU manuals.

A possible explanation is provided in https://www.grymoire.com/Unix/Regular.html#uh-9 (found in a random search) which says:

Searching for a word isn't quite as simple as it at first appears. The string "the" will match the word "other". You can put spaces before and after the letters and use this regular expression: " the ". However, this does not match words at the beginning or end of the line. And it does not match the case where there is a punctuation mark after the word.

The character before the "t" must be either a new line character, or anything except a letter, number, or underscore. The character after the "e" must also be a character other than a number, letter, or underscore or it could be the end of line character.

I don't know whether the author got this from, but assuming it's true, it would explain the behavior I'm seeing. But it still seems rather arbitrary. - isn't a punctuation mark. Why can't it be included as part of a word? Or, to put it differently, why does the hyphen match the end of a word, and the underscore doesn't? In fact, the hyphen is much more common in natural languages than the underscore is. Though perhaps the underscore is used in programming languages.

And if this is correct, then the GNU documentation really should document it properly. If I have this straight, would a bug report be in order?

Ideally this feature would be customizable. But perhaps that's asking too much.

This extract from the GNU grep code, suggests that the description from www.grymoire.com may be correct. The relevant code is in function init_word_char in 'lib/regcomp.c', and looks like

general_case:
  for (; i < BITSET_WORDS; ++i)
    for (j = 0; j < BITSET_WORD_BITS; ++j, ++ch)
      if (isalnum (ch) || ch == '_')
        dfa->word_char[i] |= (bitset_word_t) 1 << j;

The important line here is, of course line 983 of the file:

if (isalnum (ch) || ch == '_')

I.e. the character is either alphanumeric or an underscore.

I don't understand what most of this code means, of course.

Answer 1

You seem to be assuming a formal and precise definition of "word" in relation to regular expressions, while its meaning is actually implementation-dependent.

Indeed, in the paragraph "Regular Expression Nomenclature", "Flavor", the book you are quoting states that

Even if two programs both support ⌈\<···\>⌋, they might disagree on exactly what they do and don’t consider to be a word".

_{As an example of how variable the concept is, the Regular Expression page on Wikipedia defines the [:word:] non-standard character class as including the underscore, but the footnote associated with the [:word:] symbol links to the Emacs Lisp manual, where it is said that character class matches "any character that has word syntax", further linking to a Table of Syntax Classes that doesn't list the underscore among the "word constituents" (listing it among the "symbol constituents" instead—defined as "extra characters used in variable and command names along with word constituents").}

In this light, the apparently inaccurate statement

The “start of a word” is simply the position where a sequence of alphanumeric characters begins"

is likely to be taken as a simplification rather than a definition.

The expressions "matching the beginning of a word" and "matching the position at the start of a word" don't sound extremely formal either. The version involving the empty string, though less immediately clear, is more precise because the empty string is a formally defined concept¹.

Practically,

‘\<’
    Match the empty string at the beginning of word.

says that \< matches against a string only if that string includes a word-constituent character (according to the GNU grep definition, one in the [:alnum:] character class or a _) that is not immediately preceded by a word-constituent character.

The \<([a-z]+) +\1\> pattern in your example can then be read as "a sequence of one or more lowercase alphabetic characters (which collates between "a" and "z" in your locale), the first of whom is not immediately preceded by a word-constituent character, followed by one or more spaces, followed by the whole previous sequence of lowercase alphabetic characters, the last of whom is not followed by a word-constituent character.

---

¹ _{In the context of regular expressions, the string of length zero. It can be matched, and indeed every line contains it, including the empty line. Being the identity element of the concatenation operation, it can be matched before or after any literal character. It is, for instance, what X* matches in grep 'oX*o' <<<foo, or what the empty pattern '' matches in grep '' <<<'' or what is matched in echo "" | grep '^$'; and, also, what \< matches in grep '\<' <<<'a'.}

Answer 2

That part from the GNU grep manual says:

\< Match the empty string at the beginning of word.

\> Match the empty string at the end of word.

They match at the start and end of a "word", so \<bar matches the string foo bar, or just bar, but not foobar. The matches are described as matching an empty string, since when matching \<bar against foo bar, the match is just bar, not e.g. <space>bar, the \< doesn't add any characters to the matched string (which is relevant for e.g. grep -o).

They're not standard.

\w Match word constituent, it is a synonym for [_[:alnum:]].

This is what the manual says next. Note the small print. Word characters include alphanumerics (whatever that means in the current locale), and the underscore. So someword_something doesn't match someword\> which is effectively what your second grep tries to look for.

And yes, that's because in many programming languages alphanumerics and underscores are allowed in identifier names. The hyphen isn't, it's the minus operator.

Though of course in C and Javascript, $ is also valid in identifier names, and identifier names can't start with digits, but you can't have everything.