5

I need to count the lines containing the words the and an in a text file (poem.txt), but not those containing both.

I've tried using

grep -c the poem.txt | grep -c an poem.txt

but this gives me the wrong answer of 6 when the total number of the and an is 9 lines.

I do want to count the lines containing the words and not the words themselves. Only the actual word should count, so the but not there, and an but not Pan.

Example file: poem.txt

Where is the misty shark?
Where is she?
The small reef roughly fights the mast.
Where is the small gull?
Where is he?
The gull grows like a clear pirate.
Clouds fall like old mainlands.

She will Rise calmly like a dead pirate.
Eat an orange.
Warm, sunny sharks quietly pull a cold, old breeze.
All ships command rough, rainy sails.

Elvis Aaron Presley also known simply as the Elvis
He is also referred to as the King
The best-selling solo music artist of all time
He was the most commercially successful artist in many genres

He has many awards including a Grammy lifetime achievement
Elvis in the 1970s has numerous jumpsuits including an eagle one.

To clarify further: how many lines in the poem contain the or an but you should not count the lines that include both the and an.

the car is red - this counted
an apple is in the corner - not counted
hello i am big - not counted
where is an apple - counted

So here the output should be 2.

Edit: I'm not worried about case sensitivity.

screenshot of highlighted words

Final edit: Thanks for all your help. i've managed to solve the problem. i used the one of the answer and changed it a little. i used cat poem.txt | grep -Evi -e '\<an .* the\>' -e '\<the .* an\>' | grep -Eci -e '\<(an|the)\> how ever i did change the -c in the second grep to a -n for some added info. Yet again thank you for all the help!! :)

6
  • 1
    Please edit your question and i) add an example input file and ii) show the output you expect. Clarify if we should count pan as an occurrence of an or not. Also clarify if you want to count the lines or the number of times each word appears. – terdon Feb 3 at 12:28
  • 1
    Does "but not both" mean a line that contains both should not count at all? Or should it add 1 to the count (as opposed to adding 2)? – Kamil Maciorowski Feb 3 at 12:30
  • grep -c the poem.txt | grep -c an poem.txt does not what you expect. It first counts the number of lines with the. The output, i.e. the number, is piped into a second grep, which ignores its input. It counts the lines containing an and outputs that. Besides, the two grep commands also count lines that contain there or demand. – berndbausch Feb 3 at 13:25
  • 1
    What about two occuences of the word? (e.g. The & the on line 3) How about case sensitivity? – FelixJN Feb 3 at 13:36
  • 1
    Is best-selling one word or 2 (or even 3 - best, selling, and best-selling)? As always with these sorts of questions the definition of a "word" is key. – Ed Morton Feb 5 at 2:20
7

With grep:

cat poem.txt \
  | grep -Evi -e '\<an\>.*\<the\>' -e '\<the\>.*\<an\>' \
  | grep -Eci -e '\<(an|the)\>'

This counts the matched lines. You can find an alternative syntax which counts the total number of matches down below.

Breakdown:

The frist grep command filters out all lines containing both 'an' and 'the'. The second grep command counts those lines, containing either 'an' or 'the'.

If you remove the c from the second grep's -Eci, you will see all matches highlighted.

Details:

  • The -E option enables extended expression syntax (ERE) for grep.

  • The -i option tells grep to match case-insensitive

  • The -v option tells grep to invert the result (i.e. match lines not containing the pattern)

  • The -c option tells grep to output the number of matched lines instead of the lines themselves

  • The patterns:

    1. \< matches the beginning of a word (thanks @glenn-jackman)
    2. \> matches the end of a word (thanks @glenn-jackman)

    --> That way we can make sure to not match words containing 'the' or 'an' (like 'pan')

    1. grep -Evi -e '\<an\>.*\<the\>' thus matches all lines not containing 'an ... the'

    2. Similarly, grep -Evi -e '\<the\>.*\<an\>' matches all lines not containing 'the ... an'

    3. grep -Evi -e '\<an\>.*\<the\>' -e '\<the.*an\>' is the combination of the 3. and 4.

    4. grep -Eci -e '\<(an|the)\>' matches all lines containing either 'an' or 'the' (surrounded by whitespace or start/end of line) and prints the number of matched lines

EDIT 1: Use \< and \> instead of ( |^) and ( |$), as suggested by @glenn-jackman

EDIT 2: In order to count the number of matches instead of the number of matched lines, use the following expression:

cat poem.txt \
  | grep -Evi -e '\<an\>.*\<the\>' -e '\<the\>.*\<an\>' \
  | grep -Eio -e '\<(an|the)\>' \
  | wc -l

This uses the -o option of grep, which prints every match in a separate line (and nothing else) and then wc -l to count the lines.

13
  • 4
    I'd recommend using \< instead of (^| ) and \> instead of ( |$) -- the angle brackets are the word boundaries for GNU extended regular expressions. After all, a non-word character can be something different from space. – glenn jackman Feb 3 at 15:22
  • 6
    Also why use cat why not filename with grep? – Prvt_Yadav Feb 3 at 15:36
  • 1
    I prefer this as the OP seems more.familiar with grep than awk, and may have hoped for, or be more at ease with, a grep based answer – Stilez Feb 4 at 0:36
  • 1
    @theCalcaholic apart from being slower, with cat you also lose all line number information – phuclv Feb 5 at 4:05
  • 1
    @theCalcaholic, that's all right. I used your second edit code, it gives another result. – TigerTV.ru Feb 5 at 11:58
10
perl -nE 'END {say $c+0} ++$c if /\bthe\b/i xor /\ban\b/i' file
gawk 'END {print c+0} /\<the\>/ != /\<an\>/ {++c}' IGNORECASE=1 file

Comparing the results from matching each expression can give the outcome you want.

For example, the result of matching \<the\> may be either 0 or 1. If the result of the other match is the same, then both regexps were either found or not found, and the line should not be counted. If they differ it means that one match was found and the other was not, so the counter is incremented.

gawk has a built-in xor() function:

gawk 'END {print c+0} xor(/\<the\>/,/\<an\>/) {++c}' IGNORECASE=1 file
2
5

The following GNU awk program should do the trick:

awk '(/(^|\W)[Tt]he(\W|$)/ && !/(^|\W)[Aa]n(\W|$)/) || (/(^|\W)[Aa]n(\W|$)/ && !/(^|\W)[Tt]he(\W|$)/) {c++} END{print c}' poem.txt

This will increase the counter c, if either

  • the line matches (^|\W)[Tt]he(\W|$) (first-letter-case-insensitive the, preceded by non-word constituent (\W) or begin of line (^), and followed by non-word constituent (\W) or end-of line ($)) but not (^|\W)[Aa]n(\W|$) (the isolated first-letter-case-insensitive an) - OR -
  • the line matches (^|\W)[Aa]n(\W|$) but not (^|\W)[Tt]he(\W|$)

In the end, print the value of c.

It can be formulated slightly shorter using \< and \> for "beginning-of-word" and "end-of-word":

awk '(/\<[Tt]he\>/ && !/\<[Aa]n\>/) || (/\<[Aa]n\>/ && !/\<[Tt]he\>/) {c++} END{print c}' poem.txt

Even shorter would be:

awk '/\<[Tt]he\>/ != /\<[Aa]n\>/ {c++} END{print c}' poem.txt

as the inequality is only ever true if either, but not both (nor none) of an and the are present on a line.

This approach requires GNU awk because the \W and \< / \> constructs are GNU extensions to the extended regular expression syntax (but \< / \> are also understood by BSD regexes).

Notice that the pipeline construct you showed in your own attempted solution won't work, as calling grep with a file as input parameter supersedes reading from stdin, so the first part of the pipeline would simply vanish unnoticed, with the output being entirely due to the last part (which looks for occurences of an, even those embedded in other words).

1
  • Once again, I would really be interested to know what is wrong with the answer that it merits a downvote ... perhaps I could then improve it! – AdminBee Feb 5 at 14:49
3

You can do it with GNU grep and PCRE zero-length assertions:

grep -iP '(?=.*\bthe\b)(?!.*\ban\b)|(?=.*\ban\b)(?!.*\bthe\b)' poem.txt

Where is the misty shark?
...
Eat an orange.
...

grep -ciP '(?=.*\bthe\b)(?!.*\ban\b)|(?=.*\ban\b)(?!.*\bthe\b)' poem.txt

9

The same feature is available in perl (where it originated), and perl may be present on machines where GNU grep isn't.

3

This will let you output lines containing any number of any set of words and will work using any awk in any shell on every Unix box:

$ cat tst.awk
BEGIN {
    FS = "[^[:alnum:]_]+"
    split(tolower(tgts),tmp)
    for (i in tmp) {
        targets[tmp[i]]
    }
    tgtsRequired = (tgtsRequired ? tgtsRequired : 1)
}
{
    delete present
    for (i=1; i<=NF; i++) {
        present[tolower($i)]
    }

    tgtsPresent = 0
    for (target in targets) {
        tgtsPresent += (target in present ? 1 : 0)
    }
}
tgtsPresent == tgtsRequired

$ awk -v tgts='the an' -f tst.awk poem.txt
Where is the misty shark?
The small reef roughly fights the mast.
Where is the small gull?
The gull grows like a clear pirate.
Eat an orange.
Elvis Aaron Presley also known simply as the Elvis
He is also referred to as the King
The best-selling solo music artist of all time
He was the most commercially successful artist in many genres

$ awk -v tgts='the an' -v tgtsRequired=2 -f tst.awk poem.txt
Elvis in the 1970s has numerous jumpsuits including an eagle one.

$ awk -v tgts='the an eagle' -v tgtsRequired=2 -f tst.awk poem.txt

$ awk -v tgts='the an eagle' -v tgtsRequired=3 -f tst.awk poem.txt
Elvis in the 1970s has numerous jumpsuits including an eagle one.

It assumes a "word" is any contiguous string of alpha-numeric-or-underscore characters.

3
  • 2
    I see you also got the causal drive-by downvote without explanation ... – AdminBee Feb 5 at 15:11
  • 1
    @AdminBee yeah, I get one on most posts. I flagged one for moderator attention one time but nothing changed. – Ed Morton Feb 5 at 15:39
  • 1
    @terdon can you see who the phantom-of-the-forum is who runs around downvoting everything for no reason and, if so, is there anything can be done about it? – Ed Morton Feb 5 at 15:52
2

This GNU awk command will do what you need:

$ awk '
        function lnxor(a,b) {return !(a&&b||(!a&&!b))};
        { str=tolower($0); a=(str~/\<the\>/); b=(str~/\<an\>/); 
          c=lnxor(a,b); count+=c 
        } END {print count}
' poem.txt

9

The lnxor is the logical not xor of a and b.

Where a is true if the line contains the and b is true if the line contains an.

In this way it is quite easy to extend the idea to more words.

In GNU awk it is also possible to use IGNORECASE (set it to non-zero), instead of tolower(), to ignore the case of the string being matched by the regex. However, that will change all string and regex operations, including the effects of FS and RS. Being, as it is, such an extensive change (which will work in this case), it should be used with more care. It is not specific to some particular test.

2

A method using perl wherein we sum the boolean values of the regexes and using that as an index select from an anonymous array the appropriate increment depending on whether none found, any one found, or both.

perl -lne '$k += (0,1,0)[/\ban\b/i+/\bthe\b/i]}{print 0+$k' poems.txt
9
2

Using gawk and wc:

gawk 'xor(/\<an\>/,/\<the\>/)' IGNORECASE=1 ./poem.txt | wc -l
6
  • 2
    It bears mentioning that this requires GNU Awk, as both the xor() function and the IGNORECASE variable are not provided natively by some other implementations, notably The One True Awk. – Greg A. Woods Feb 4 at 20:56
  • @GregA.Woods, yes, I used GNU Awk. – TigerTV.ru Feb 4 at 21:00
  • 1
    The \< and \> are also non-POSIX extensions. – Ed Morton Feb 5 at 2:42
  • @EdMorton, it's time to update POSIX :) – TigerTV.ru Feb 5 at 12:03
  • 1
    Then you'd have to update all of the software that claims to be POSIX compliant :-). It's easier just to say "using GNU awk for ...." in answers so people don't waste their time trying to make a script work when all they have is a POSIX awk. – Ed Morton Feb 5 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.