2

I have a sample textfile(test_long_sentence.txt) below and I want to grep all the lines that contain test1 excluding unwanted data.

How do I grep the data before the quote closes?

test_long_sentence.txt

This is some unwanted data blah blah blah

20  /test1/catergory="Food"
20  /test1/target="Adults, \"Goblins\", Elderly,
Babies, \"Witch\",
Faries"
20  /test1/type="Western"

This is some unwanted data blah blah blah

20  /test1/theme="Halloween"

Command:

grep "test1" test_long_sentence.txt

Actual Output:

20  /test1/catergory="food"
20  /test1/target="Adults, \"Goblins\", Elderly,
20  /test1/type="Western"
20  /test1/theme="Halloween"

Expected Output:

20  /test1/catergory="food"
20  /test1/target="Adults, \"Goblins\", Elderly,
Babies, \"Witch\",
Faries"
20  /test1/type="Western"
20  /test1/theme="Halloween"

Ps: I have no control in editing the test_long_sentence.txt. So please, do not ask me to edit it to a single line.

  • you don't have to edit the input file, but you will have to at least pre-process it (e.g. with an awk or perl script) to join lines that don't start with a number and a space (that seems to be the common factor in continued lines from the small sample you've given). the pre-processing script will also have to ignore all lines from an empty line to the next line starting with a number. BTW, since you'll need perl or awk or similar to join the continued lines, you may as well make it also do the search & print. – cas Feb 12 '18 at 3:33
  • Is pcregrep an option? e.g. pcregrep -M '(?s)test1.*?"\n' test_long_sentence.txt – steeldriver Feb 12 '18 at 3:41
0

Using awk

$ awk '/test1/{line=$0; while (!(line ~ /[^\\]".*[^\\]"/)) {getline; line=line "\n" $0}; print line}' sentence.txt 
20  /test1/catergory="Food"
20  /test1/target="Adults, \"Goblins\", Elderly,
Babies, \"Witch\",
Faries"
20  /test1/type="Western"
20  /test1/theme="Halloween"

/test1/ is a condition. If the current line contains a match to the regex test1, then the commands in curly braces are performed. Those commands are:

  • line=$0

    The contents of the current line are saved in variable `line.

  • while (!(line ~ !/[^\\]".*[^\\]"/)) {getline; line=line "\n" $0}

    If the current contents of line do not contain two unescaped quotes, then get the next line, getline and append it to line via line=line "\n" $0

  • print line

    Now that the variable line contains two unescaped quotes, we print it.

For those who prefer their commands spread over multiple lines, the same command as above can be written as:

awk '
    /test1/{
        line=$0
        while (!(line ~ /[^\\]".*[^\\]"/)) {
            getline
            line=line "\n" $0
        }
        print line
    }' sentence.txt 

Using sed

$ sed -n '/test1/{:a; /[^\\]".*[^\\]"/{p;b}; N; ba}' sentence.txt 
20  /test1/catergory="Food"
20  /test1/target="Adults, \"Goblins\", Elderly,
Babies, \"Witch\",
Faries"
20  /test1/type="Western"
20  /test1/theme="Halloween"

How it works:

  • -n

    This tells sed not to print anything unless we explicitly ask it to.

  • /test1/{...}

    For any line containing test1, we perform the commands in curly braces which are:

    • :a

      This defines a label a.

    • /[^\\]".*[^\\]"/{p;b}

      If the pattern space currently contains two unescaped quotes, we print the pattern space, p, and then we skip the rest of the instructions and branch, b, to start over on the next line.

    • N

      If we get here, that means that the current did not have two unescaped quotes. We read in the next line into the pattern space.

    • ba

      We branch back to label a and repeat that commands which follow that label.

  • Thanks! Using sed method worked! I edited it a bit to accept variable input: sed -n '/$var/{:a; /[^\]".*[^\]"/{p;b}; N; ba}' sentence.txt. But I am curious to know how will the var work if I am using awk method? – Jojoleo Feb 12 '18 at 6:27
  • @Jojoleo To use the awk method with a variable: awk -v x="test1" '$0~x{line=$0; while (!(line ~ /[^\\]".*[^\\]"/)) {getline; line=line "\n" $0}; print line}' sentence.txt. If you are going to use a variable, the awk method is actually safer than sed. – John1024 Feb 12 '18 at 7:38
0

This will work with awk instead of grep for that particular file:

awk 'NR==3,NR==7;NR==11' test_long_sentence.txt

To help with this in the future, you can run cat -n on the file to see the lines that you want to include and exclude.

0

Here's a simple perl script to join the continued lines in your input. It assumes:

  • continued lines are to be joined with a single space character.

  • a "line" starts from any input line beginning with a number and a space, and continues until either:

    • a blank like
    • another line beginning with a number and a space
  • blank lines, and all lines between blank lines are to be ignored (i.e. discarded).

This may not be a perfect match for your actual input file, but does match the sample input provided in your question. Modify the code to suit your input as required.

#!/usr/bin/perl

my $skip=1;  # start with skip = true.
my $line='';

while(<>) {
  chomp;

  if (m/^\d+\s+/) { # / this comment is only here to fix SE\'s syntax highlighting
    $skip=0;
    print $line,"\n" if ($line);
    $line = $_;

 } elsif (m/^\s*$/) {
    if ($line ne '') { print $line, "\n"; $line = ''};
    $skip = 1 - $skip;

  } elsif (! $skip) {
    $line .= " $_";
 };
};

save as, e.g., ./join-lines.pl, make executable with chmod +x ./join-lines.pl, and run like:

$ ./join-lines.pl test_long_sentence.txt 
20  /test1/catergory="Food"
20  /test1/target="Adults, \"Goblins\", Elderly, Babies, \"Witch\", Faries"
20  /test1/type="Western"
20  /test1/theme="Halloween"

This can then be fed into grep or other tools as required.

Note that with the current sample input, the output of ./join-lines.pl test_long_sentence.txt | grep test1 is (apart from possible colourisation of matches by grep) identical to the output above because all of the sample lines contain "test1". You can more usefully grep for other matches, e.g.:

$ ./join-lines.pl test_long_sentence.txt | grep Witch
20  /test1/target="Adults, \"Goblins\", Elderly, Babies, \"Witch\", Faries"

If you need to be able to reproduce the exact same input (but without the blank and unwanted lines) then instead of joining the lines with a space, use a character (e.g. a TAB) or a longer string (e.g. " ===NL=== ") that you are absolutely certain will never appear in the input. e.g.

  } elsif (! $skip) {
    $line .= "\t$_";
 };

or

  } elsif (! $skip) {
    $line .= " ===NL=== $_ ";
 };

The joining string can then be converted back to a newline with, e.g., sed (using a tab character as the join character):

$ ./join-lines.pl test_long_sentence.txt  | grep Witch | sed -e 's/\t/\n/g'
20  /test1/target="Adults, \"Goblins\", Elderly,
Babies, \"Witch\",
Faries"

Another example, using a longer string for joining the lines:

./join-lines.pl test_long_sentence.txt  | grep Witch | sed -e 's/ ===NL=== /\n/g'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.