2

I am trying to process a file. My attempt did not work. The input file describes the desired output:

Input file:

This is a token, but when any punctuation is encountered, it stops the extraction.

I want to get n words around a specific token, meaning n words before the token and n words after the token. There is no fix pattern, as given in some other solutions.

Please help. Thank you.

Command used:

$ grep -io -E  '(\w+ ){0,5}\b(token)\b( \w+){0,5}' grepping-n-words-around-token 

Output:

This is a token
n words around a specific token
meaning n words before the token and n words after the
token

Desired Output:

This is a token, but when any punctuation is
n words around a specific token, meaning n words before the
meaning n words before the token and n words after the
and n words after the token. There is no fix pattern
3

You can't have GNU grep -o output the same text (like your meaning n words before the or and n words after the) twice. You could do it with pcregrep though by using -o<n> where n is the nth capture group and capture what is matched in a look ahead operator (which doesn't advance the cursor for the next match):

$ pcregrep -o0 -o2  '(\w+\W+){0,5}token(?=((\W+\w+){0,5}))' file
This is a token, but when any punctuation is
n words around a specific token, meaning n words before the
meaning n words before the token and n words after the
and n words after the token. There is no fix pattern

-o0 is the whole text matched, -o1 is what is matched by the (....) inside the (?=(here)) look-ahead operator.

Note that on an input like:

6 5 4 3 2 1 token token 1 2 3 4 5 6

it would give:

5 4 3 2 1 token token 1 2 3 4
token 1 2 3 4 5

because it starts looking for the second match right after the first token, so only finds 0 words before the second token.

$ echo 6 5 4 3 2 1 token token 1 2 3 4 5 6 |
   pcregrep -o1  '(?=((\w+\W+){0,5}token(\W+\w+){0,5}))\w*'
5 4 3 2 1 token token 1 2 3 4
4 3 2 1 token token 1 2 3 4 5
3 2 1 token token 1 2 3 4 5
2 1 token token 1 2 3 4 5
1 token token 1 2 3 4 5
token token 1 2 3 4 5
token 1 2 3 4 5

Would probably not be what you want either (even if each is "token" preceded and followed by up to 5 words).

To get a line for each occurrence of "token" with up to 5 words on either side, I don't think you do it easily with pcregrep alone.

You'd need to record the position of each "token" word and then match the up-to-5-words<that-position>"token"up-to-5-words for each of those positions.

Something like:

$ echo 6 5 4 3 2 1 token token 1 2 3 4 5 6 | perl -lne '
    my @positions; push @positions, $-[0] while /\btoken\b/g;
    for $o (@positions) {
      print $& if /(\w+\W+){0,5}(?<=^.{$o})token(\W+\w+){0,5}/
    }'
5 4 3 2 1 token token 1 2 3 4
4 3 2 1 token token 1 2 3 4 5

Or to clarify which token is being matched in each case:

$ echo 6 5 4 3 2 1 token token 1 2 3 4 5 6 | perl -lne '
    my @positions; push @positions, $-[0] while /\btoken\b/g;
    for $o (@positions) {
      print "$1<token>$3" if /((\w+\W+){0,5})(?<=^.{$o})token((\W+\w+){0,5})/
    }'
5 4 3 2 1 <token> token 1 2 3 4
4 3 2 1 token <token> 1 2 3 4 5

(I'd expect it could be simplified/optimised).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.