5

I have to parse huge text files where certain lines are of interest and others are not. Within those of interest I have to count the occurrences of a certain keyword.

Assumed the file is called input.txt and it looks like this:

format300,format250,format300
format250,ignore,format160,format300,format300
format250,format250,format300

I want to exclude the lines with ignore and count the number of format300, how do I do that?

What I've got so far is this command which only counts ONCE PER LINE (which is not yet good enough):

cat input.txt | grep -v ignore | grep 'format300' | wc -l

Any suggestions? If possible I want to avoid using perl.

7

You don't need the first cat, that it is known as a Useless use of cat (UUOC).

Also, very useful is grep -o, that only outputs the matching patterns, one per line.

And then, count lines with wc -l.

grep -v ignore YOUR_FILE | grep -o format300 | wc -l

This prints 3 for your small sample.

  • But it should print out 5, because my sample contains 5 times format300 – basZero Apr 11 '16 at 7:46
  • But there are 2 format300 on the line containing ignore, so these 2 shouldn't count, right? – Carlos Campderrós Apr 11 '16 at 7:59
  • thanks, this is the perfect answer to my question, very simple and it does exactly what I asked for. – basZero Apr 11 '16 at 10:49
  • and yes, you are right of course regarding the ignore. All perfect! – basZero Apr 11 '16 at 10:51
9

This one-liner should be able to do what you want:

grep -v ignore input.txt | sed 's/format300/format300\n/g' | grep -c "format300"

basically you are replacing each occurrence of your keyword with the keyword itself and a newline character, which effectively makes your input stream have the keyword only once on any given line. Then grep -c is counting lines with your keyword in them.

  • 3
    I would replace the sed with tr ',' '\n' otherwise you're going to count format3000s as well not just format300s – iruvar Apr 8 '16 at 15:18
  • 2
    @1_CR, agreed, but you would also have to use grep -xc format300 instead of grep -c format300 to avoid false positives on "format3000". So full solution is grep -v ignore input.txt | tr , '\n' | grep -xc format300 – Wildcard Apr 8 '16 at 18:57
  • @Wildcard, indeed – iruvar Apr 8 '16 at 19:01
  • Thanks guys, voted for this answer, but there was a simpler answer which I preferred over this one. – basZero Apr 11 '16 at 10:50
3

Input file may potentially contain partial matches that would invalidate the result, for example:

1 format300,format250,format300
2 format250,ignore,format160,format300,format300
3 format250,format250,format300
4 format999,format300000,format999
5 format999,ignore_me_not,format300

You don't want to count format300000 on line 4 or ignore line 5 because ignore_me_not contains the substring ignore.

This would do the trick:

grep -v "\bignore\b" FILE |grep -o "\bformat300\b"|wc -l

Correct output is

4

..becase line 2 is ignored, line 5 is not, and line 4 doesn't contain exactly format300.

If you let the wc -l part out, you can see what exactly is being matched:

enter image description here

  • Clever, but it goes beyond his actual stated requirements: "I want to exclude the lines with ignore..." Your point about format3000 et. al. is perfectly valid, though. (As I commented on another answer.) – Wildcard Apr 8 '16 at 19:00
  • This is the only solution that is considering the false positives, the others so far will all fail with this input. There is still a problem, though, because the word boundaries aren't quite what are needed. A character like - is not a word character, so ignore-me-not will match \bignore\b but shouldn't. – leftclickben Apr 8 '16 at 19:06
2

A Perl way:

perl -lne '$k+=(s/format300//g) unless /ignore/; }{ print $k' input.txt 

The s/format300//g will replace all occurrences of format300 with nothing, and returns the number of replacements. It's a simple way of counting the occurrences. The number is then added to $k and the whole thing only happens if the line doesn't match ignore. The }{ is perl shorthand for "do this after you've finished reading the file, so print $k will print the total number found.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.