Count number of a substring repetition in a string

Question

I have a file which contains a gene sequence such as:

ATGTGGATGGTGGGTTACAATGAAGGTGGTGAGTTCAACATGGCTGATTATCCATTCAGTGGAAGGAAACTAAGGCCTCTCATTCCAAGACCAGTCCCAGTCCCTACTACTTCTCCTAACAGCACTTCAACTATAACTCCTTCCTTAAACCGCATTCATGGTGGCAATGATTTATTTTCACAATATCATCACAATCTGCAGCAGCAAGCATCAGTAGGAGATCATAGCAAGAGATCAGAGTTGAATAATAATAATAATCCATCTGCAGCAGTTGTGGTGAGTTCAAGATGGAATCCAACACCAGAACAGTTAAGAGCACTGGAAGAATTGTATAGAAGAGGAACAAGAACACCTTCTGCTGAGCAAATCCAACAAATAACTGCCCAGCTTAGAAAATTTGGAAAAATTGAAGGCAAAAATGTTTTCTATTGGTTTCAGAATCACAAAGCCAGAGAAAGGCAAAAACGACGGCGTCAAATGGAATCAGCAGCTGCTGAGTTTGATTCTGCTATTGAAAAGAAAGACTTAGGCGCAAGTAGG


ACAGTGTTTGAAGTTGAACACACTAAAAACTGGCTACCATCTACAAATTCCAGTACCAGTACTCTTCATCTTGCAGAGGAATCTGTTTCAATTCAAAGGTCAGCAGCAGCAAAAGCAGATGGATGGCTCCAATTCGATGAAGCAGAATTACAGCAAAGAAGAAACTTTATGGAAAGGAATGCCACGTGGCATATGATGCAGTTAACTTCTTCTTGTCCTACAGCTAGCATGTCCACCACAACCACAGTAACAACTAGACTTATGGACCCAAAACTCATCAAGACCCATGAACTCAACTTATTCATTTCACCTCACACATACAAAGAAAGAGAAAACGCTTTTATCCACTTAAATACTAGTAGTACTCATCAAAATGAATCTGATCAAACCCTTCAACTTTTCCCAATAAGGAATGGAGATCATGGATGCACTGATCATCATCATCATCATCATAACATTATCAAAGAGACACAGATATCAGCTTCAGCAATCAATGCACCCAACCAGTTTATTGAGTTTCTTCCCTTGAAAAACTGA

I am trying to count the number of occurrence of "ATG" substring in the above string (which is only one line without line breaks.) My file contains tens (10s) of these sequences and I want to be able to count how many "ATG" in each sequence. Each sequence is separated from others by an empty line.

I tried grep but did not know which options I should use (if at all grep can solve the problem) and I googled for any awk example but I did not find any.

Answer 1

Returns the number of occurrences of ATG in each line:

awk -F'ATG' 'NF{print NF-1}' testfile

This works for files with one or many lines.

Example 1

Consider this test file:

$ cat testfile
xxATGxxATG

ATGxxxATGxxx

xxATGxxxxATGxxATGxx

The code correctly counts the occurrences of ATG:

$ awk -F'ATG' 'NF{print NF-1}' testfile
2
2
3

Example 2

Using the example in the current version of the question:

$ cat >file1
ATGTGGATGGTGGGTTACAATGAAGGTGGTGAGTTCAACATGGCTGATTATCCATTCAGTGGAAGGAAACTAAGGCCTCTCATTCCAAGACCAGTCCCAGTCCCTACTACTTCTCCTAACAGCACTTCAACTATAACTCCTTCCTTAAACCGCATTCATGGTGGCAATGATTTATTTTCACAATATCATCACAATCTGCAGCAGCAAGCATCAGTAGGAGATCATAGCAAGAGATCAGAGTTGAATAATAATAATAATCCATCTGCAGCAGTTGTGGTGAGTTCAAGATGGAATCCAACACCAGAACAGTTAAGAGCACTGGAAGAATTGTATAGAAGAGGAACAAGAACACCTTCTGCTGAGCAAATCCAACAAATAACTGCCCAGCTTAGAAAATTTGGAAAAATTGAAGGCAAAAATGTTTTCTATTGGTTTCAGAATCACAAAGCCAGAGAAAGGCAAAAACGACGGCGTCAAATGGAATCAGCAGCTGCTGAGTTTGATTCTGCTATTGAAAAGAAAGACTTAGGCGCAAGTAGG


ACAGTGTTTGAAGTTGAACACACTAAAAACTGGCTACCATCTACAAATTCCAGTACCAGTACTCTTCATCTTGCAGAGGAATCTGTTTCAATTCAAAGGTCAGCAGCAGCAAAAGCAGATGGATGGCTCCAATTCGATGAAGCAGAATTACAGCAAAGAAGAAACTTTATGGAAAGGAATGCCACGTGGCATATGATGCAGTTAACTTCTTCTTGTCCTACAGCTAGCATGTCCACCACAACCACAGTAACAACTAGACTTATGGACCCAAAACTCATCAAGACCCATGAACTCAACTTATTCATTTCACCTCACACATACAAAGAAAGAGAAAACGCTTTTATCCACTTAAATACTAGTAGTACTCATCAAAATGAATCTGATCAAACCCTTCAACTTTTCCCAATAAGGAATGGAGATCATGGATGCACTGATCATCATCATCATCATCATAACATTATCAAAGAGACACAGATATCAGCTTCAGCAATCAATGCACCCAACCAGTTTATTGAGTTTCTTCCCTTGAAAAACTGA

This results in:

$ awk -F'ATG' 'NF{print NF-1}' file1
9
15

How it works

awk implicitly loops through every line of a file. Each line is divided into fields.

• -F'ATG'

  This tells awk to use ATG as the field separator.

• NF{print NF-1}

  For each non-empty line, this tells awk to print the number of fields minus 1.

  (On empty lines, the number of fields, NF, is zero. So, the condition NF evaluates to false on these lines, effectively skipping over them.)

Answer 2

From man grep

-o, --only-matching
    Print only the matched (non-empty) parts of a matching line, with each such part on a separate output line.

So you can try

$ grep -o 'ATG' file | wc -l

Answer 3

Improving tachomi's idea of grep -o, with John1024's testfile, here is a way to include both the count as well as indicating which line of the input file has that count, in case you are generating thousands of lines, and at a later point may need to retrace exactly which line a count came from.

Firstly a sample file, based on a slightly different version of John1024's testfile,

$ cat testfile2
xxATGxxATG

ATGxxxATGxxx

xxATGxxxxATGxxATG

If you add -n to show line number in original input, you see:

$ grep -no ATG testfile2
1:ATG
1:ATG
3:ATG
3:ATG
5:ATG
5:ATG
5:ATG

Finally, use uniq -c to do the counting:

$ grep -no ATG testfile2 | uniq -c
  2 1:ATG
  2 3:ATG
  3 5:ATG

So now you have both counts of ATG, as well as what line number (of the input file) that count came from.

You can also convert it to show only counts, by making use of awk:

$ grep -no ATG testfile2 | uniq -c | awk '{print $1}'
2
2
3

Awk's $1 refers to the first field.

Answer 4

You can use nl's section -delimiter to reset counts per set.

After copying your data to my clipboard...

xsel |
sed '/./!G;s/\n/::::::&::::/;s/ATG/&\
/g' | nl -d:: -v0

...yields...

 0  ATG
 1  TGGATG
 2  GTGGGTTACAATG
 3  AAGGTGGTGAGTTCAACATG
 4  GCTGATTATCCATTCAGTGGAAGGAAACTAAGGCCTCTCATTCCAAGACCAGTCCCAGTCCCTACTACTTCTCCTAACAGCACTTCAACTATAACTCCTTCCTTAAACCGCATTCATG
 5  GTGGCAATG
 6  ATTTATTTTCACAATATCATCACAATCTGCAGCAGCAAGCATCAGTAGGAGATCATAGCAAGAGATCAGAGTTGAATAATAATAATAATCCATCTGCAGCAGTTGTGGTGAGTTCAAGATG
 7  GAATCCAACACCAGAACAGTTAAGAGCACTGGAAGAATTGTATAGAAGAGGAACAAGAACACCTTCTGCTGAGCAAATCCAACAAATAACTGCCCAGCTTAGAAAATTTGGAAAAATTGAAGGCAAAAATG
 8  TTTTCTATTGGTTTCAGAATCACAAAGCCAGAGAAAGGCAAAAACGACGGCGTCAAATG
 9  GAATCAGCAGCTGCTGAGTTTGATTCTGCTATTGAAAAGAAAGACTTAGGCGCAAGTAGG


 0  ACAGTGTTTGAAGTTGAACACACTAAAAACTGGCTACCATCTACAAATTCCAGTACCAGTACTCTTCATCTTGCAGAGGAATCTGTTTCAATTCAAAGGTCAGCAGCAGCAAAAGCAGATG
 1  GATG
 2  GCTCCAATTCGATG
 3  AAGCAGAATTACAGCAAAGAAGAAACTTTATG
 4  GAAAGGAATG
 5  CCACGTGGCATATG
 6  ATG
 7  CAGTTAACTTCTTCTTGTCCTACAGCTAGCATG
 8  TCCACCACAACCACAGTAACAACTAGACTTATG
 9  GACCCAAAACTCATCAAGACCCATG
10  AACTCAACTTATTCATTTCACCTCACACATACAAAGAAAGAGAAAACGCTTTTATCCACTTAAATACTAGTAGTACTCATCAAAATG
11  AATCTGATCAAACCCTTCAACTTTTCCCAATAAGGAATG
12  GAGATCATG
13  GATG
14  CACTGATCATCATCATCATCATCATAACATTATCAAAGAGACACAGATATCAGCTTCAGCAATCAATG
15  CACCCAACCAGTTTATTGAGTTTCTTCCCTTGAAAAACTGA

sed just appends a newline after every occurrence of the string ATG it finds in input. sed also transforms blank-lines to look like:

::::::
::::

nl divides logical pages on its section -delimiter which is here set to ::. Three section -delimiters on a line by itself indicates the end of a logical page - which nl replaces with a blank-line - and two such the start of a new one. I start each count at -v0 zero because the last sequence in any chain will not contain the ATG string.

For less descriptive output, you can simply transform every ATG sequence to a character that doesn't otherwise appear in your input, and -delete the difference.

xsel |
sed 's/ATG/./g' | tr -dc . | wc -c

...which swaps every ATG in input with a dot, deletes all bytes which are not dots, then counts bytes:

24

Answer 5

If you wanna have some real awk fun, you could do this:

echo -e "test one - test two - test three\ntest four - test five\nnope six\ntest seven"
test one - test two - test three
test four - test five
nope six
test seven
...
echo -e "test one - test two - test three\ntest four - test five\nnope six\ntest seven" | awk -v myVar="test" 'BEGIN{count=0}; {thisLine=gsub(myVar,"&"); count=count+thisLine; print "\"" myVar "\" in line " NR ": " thisLine}; END{print "Total number of \"" myVar "\": " count}'
"test" in line 1: 3
"test" in line 2: 2
"test" in line 3: 0
"test" in line 4: 1
Total number of "test": 6

Breakdown:

echo -e "test one - test two - test three\ntest four - test five\nnope six\ntest seven" |\  ## echo -e tells bash that '\n' is a new line
awk -v myVar="test" '                                  ## -v set an awk variable
  BEGIN{
    count=0
  };                                                   ## Begins with a 'count' variable set to zero
  {                                                    ## Now, for each line...
    thisLine=gsub(myVar,"&");                          ## Set var for number of myVar in the line, since we're using it twice
    count=count+thisLine;                              ## Add number in line to total count
    print "\"" myVar "\" in line " NR ": " thisLine    ## print for the line: myVar in quotes, line number, then count in line
  };
  END{
    print "Total number of \"" myVar "\": " count      ## End with total count
  }
'