Punctuation problem using grep for getting n words around the token

Question

I am trying to process a file. My attempt did not work. The input file describes the desired output:

Input file:

This is a token, but when any punctuation is encountered, it stops the extraction.

I want to get n words around a specific token, meaning n words before the token and n words after the token. There is no fix pattern, as given in some other solutions.

Please help. Thank you.

Command used:

$ grep -io -E  '(\w+ ){0,5}\b(token)\b( \w+){0,5}' grepping-n-words-around-token 

Output:

This is a token
n words around a specific token
meaning n words before the token and n words after the
token

Desired Output:

This is a token, but when any punctuation is
n words around a specific token, meaning n words before the
meaning n words before the token and n words after the
and n words after the token. There is no fix pattern

Answer 1

You can't have GNU grep -o output the same text (like your meaning n words before the or and n words after the) twice. You could do it with pcregrep though by using -o<n> where n is the n^th capture group and capture what is matched in a look ahead operator (which doesn't advance the cursor for the next match):

$ pcregrep -o0 -o2  '(\w+\W+){0,5}token(?=((\W+\w+){0,5}))' file
This is a token, but when any punctuation is
n words around a specific token, meaning n words before the
meaning n words before the token and n words after the
and n words after the token. There is no fix pattern

-o0 is the whole text matched, -o1 is what is matched by the (....) inside the (?=(here)) look-ahead operator.

Note that on an input like:

6 5 4 3 2 1 token token 1 2 3 4 5 6

it would give:

5 4 3 2 1 token token 1 2 3 4
token 1 2 3 4 5

because it starts looking for the second match right after the first token, so only finds 0 words before the second token.

$ echo 6 5 4 3 2 1 token token 1 2 3 4 5 6 |
   pcregrep -o1  '(?=((\w+\W+){0,5}token(\W+\w+){0,5}))\w*'
5 4 3 2 1 token token 1 2 3 4
4 3 2 1 token token 1 2 3 4 5
3 2 1 token token 1 2 3 4 5
2 1 token token 1 2 3 4 5
1 token token 1 2 3 4 5
token token 1 2 3 4 5
token 1 2 3 4 5

Would probably not be what you want either (even if each is "token" preceded and followed by up to 5 words).

To get a line for each occurrence of "token" with up to 5 words on either side, I don't think you do it easily with pcregrep alone.

You'd need to record the position of each "token" word and then match the up-to-5-words<that-position>"token"up-to-5-words for each of those positions.

Something like:

$ echo 6 5 4 3 2 1 token token 1 2 3 4 5 6 | perl -lne '
    my @positions; push @positions, $-[0] while /\btoken\b/g;
    for $o (@positions) {
      print $& if /(\w+\W+){0,5}(?<=^.{$o})token(\W+\w+){0,5}/
    }'
5 4 3 2 1 token token 1 2 3 4
4 3 2 1 token token 1 2 3 4 5

Or to clarify which token is being matched in each case:

$ echo 6 5 4 3 2 1 token token 1 2 3 4 5 6 | perl -lne '
    my @positions; push @positions, $-[0] while /\btoken\b/g;
    for $o (@positions) {
      print "$1<token>$3" if /((\w+\W+){0,5})(?<=^.{$o})token((\W+\w+){0,5})/
    }'
5 4 3 2 1 <token> token 1 2 3 4
4 3 2 1 token <token> 1 2 3 4 5

(I'd expect it could be simplified/optimised).