Count occurrences of line in multiple files

Question

I have log files that I need to grep and count the number of occurrences of a specific line.

Issue is that I must start at a specific line of the first file and only count occurrences after that line. All other files should be searched completely. Only the first file needs to be limited to the last x lines.

Then I need to multiply the result by 200. I'd like to do this with a single command.

Here is a working example that works with 2 log files but I will need to expand this to 3 files soon and was hoping someone could come up with a better way to do this:

expr `tail -n 5147 Log1.log.2013-11-18 | grep "The line I need to count" | wc -l` + `cat Log2.log | grep "The line I need to count" | wc -l` | awk '{print $1"*200"}' | bc

Answer 1

I think you should be able to just add the third file after the second cat:

expr `tail -n 5147 Log1.log.2013-11-18 | grep "The line I need to count" | wc -l` + `cat Log2.log Log3.log | grep "The line I need to count" | wc -l` | awk '{print $1"*200"}' | bc

example:

$ echo abc > 1.txt 
$ echo cde > 2.txt 
$ echo def > 3.txt 
$ cat 1.txt 2.txt 3.txt | grep d 
cde 
def

Answer 2

A general solution for arbitrary numbers of files:

 for file1 in log1.txt log2.txt logN.txt; do
   for file2 in logA.txt logB.txt logC.txt; do
     for file3 in logD.txt logE.txt logF.txt; do
     match1=$(tail -n 5147 $file1 | grep -c 'The line');
     match2=$(grep -c "the line" $file2);
     match3=$(grep -c "the line" $file3);
     echo "($match1 + $match2 + $match3) * 200" | bc;
   done;
 done;
done

This will count the occurrences of the line in the last 5147 lines of each of $file1s and save that as $match1. It will then count the occurrences of the line in each of the $file2s and save it as match2 and the number of occurrences in each file3 as $match3. Then, it will calculate ($match1 + $match2 + $match3) * 200.

Simply adding one more file to what you are doing and fixing up your syntax:

echo "$(tail -n 5147 Log1.log.2013-11-18 | grep -c 'The line') \ 
      $(grep -c 'The line' Log2.log) $(grep -c 'The line' Log3.log)" | 
        awk '{print ($1+$2+$3)*200}' 

There are a couple of tricks here. grep -c counts matches instead of printing them, so no need for wc. awk can do math, so no need for bc. I use echo instead of expr because I am not doing arithmetic in the shell anymore, just echoing the results of the three searches and passing them to awk. This should result in something like:

echo "10 4 12" | awk '{print ($1+$2+$3)*200}' 

A more concise version:

echo $((($(tail -n 5147 Log1.log.2013-11-18 | grep -c "The line) + \ 
         $(grep -c "The line" Log2.log) + $(grep -c "The lne" Log3.log)) * 200))

This does all the math in the shell, using the $(( )) notation. You can try it with echo $((2*4)). It's not very easy to read, it expands to:

echo "$(( ($file1_matches + $file2_matches + $file3_matches) * 200 ))"

Answer 3

I solved a similar issue of counting lines in multiple CSV files.

I have a large list of CSV files - the output of a script, so they have the same fields. I was interested in the number of lines with a valid value.

The command that did the trick is

awk -F, '{if ($5>0.0) print $5}' va.2021-04-11.153123.csv va.2021-04-11.111709.csv | wc -l

• -F, sets the delimiter to , (my CSV files are , separated)
• '{if ($5>0.0) print $5}' is the "script" which tells awk what to do:
  • if the 5th field is larger than 0.0 then print the 5th field
  • then pipe that to wc which with the -l flag will count the lines in the output
• va.2021-04-11.153123.csv va.2021-04-11.111709.csv is a space-separated list of two CSV files