0

I have log files that I need to grep and count the number of occurrences of a specific line.

Issue is that I must start at a specific line of the first file and only count occurrences after that line. All other files should be searched completely. Only the first file needs to be limited to the last x lines.

Then I need to multiply the result by 200. I'd like to do this with a single command.

Here is a working example that works with 2 log files but I will need to expand this to 3 files soon and was hoping someone could come up with a better way to do this:

expr `tail -n 5147 Log1.log.2013-11-18 | grep "The line I need to count" | wc -l` + `cat Log2.log | grep "The line I need to count" | wc -l` | awk '{print $1"*200"}' | bc
  • So, you want (number of matches in last x lines of file1 + number of matches in file2 + number of matches in fileN) * 200? – terdon Nov 20 '13 at 1:47
  • @terdon Yes that seems to be a much more succinct way of putting it. – FGreg Nov 20 '13 at 1:51
1

A general solution for arbitrary numbers of files:

 for file1 in log1.txt log2.txt logN.txt; do
   for file2 in logA.txt logB.txt logC.txt; do
     for file3 in logD.txt logE.txt logF.txt; do
     match1=$(tail -n 5147 $file1 | grep -c 'The line');
     match2=$(grep -c "the line" $file2);
     match3=$(grep -c "the line" $file3);
     echo "($match1 + $match2 + $match3) * 200" | bc;
   done;
 done;
done

This will count the occurrences of the line in the last 5147 lines of each of $file1s and save that as $match1. It will then count the occurrences of the line in each of the $file2s and save it as match2 and the number of occurrences in each file3 as $match3. Then, it will calculate ($match1 + $match2 + $match3) * 200.

Simply adding one more file to what you are doing and fixing up your syntax:

echo "$(tail -n 5147 Log1.log.2013-11-18 | grep -c 'The line') \ 
      $(grep -c 'The line' Log2.log) $(grep -c 'The line' Log3.log)" | 
        awk '{print ($1+$2+$3)*200}' 

There are a couple of tricks here. grep -c counts matches instead of printing them, so no need for wc. awk can do math, so no need for bc. I use echo instead of expr because I am not doing arithmetic in the shell anymore, just echoing the results of the three searches and passing them to awk. This should result in something like:

echo "10 4 12" | awk '{print ($1+$2+$3)*200}' 

A more concise version:

echo $((($(tail -n 5147 Log1.log.2013-11-18 | grep -c "The line) + \ 
         $(grep -c "The line" Log2.log) + $(grep -c "The lne" Log3.log)) * 200))

This does all the math in the shell, using the $(( )) notation. You can try it with echo $((2*4)). It's not very easy to read, it expands to:

echo "$(( ($file1_matches + $file2_matches + $file3_matches) * 200 ))"
  • I edited my question to clarify only the first file needs to be restricted to the last x lines. Your edit is looking like what I was getting at... I felt my original command was way more complex than it needed to be. – FGreg Nov 20 '13 at 1:49
  • @FGreg see updated answer, I kept the general case for as many files as you wish and gave a more succinct version of your command + one more file. – terdon Nov 20 '13 at 1:58
  • Thank you. Your third suggestion is working and makes much more sense than mine. Your second command seems off though... what is the NR>5146? Does that take only the last 5146 lines of the file? – FGreg Nov 20 '13 at 2:19
  • @FGreg yes, NR is a special awk variable that holds the current line number. And as I write this comment, I realize that is not at all what you asked for, sorry. Have a look at the updated answer, all versions should do what you want now and I've also added a much shorter one. – terdon Nov 20 '13 at 3:02
2

I think you should be able to just add the third file after the second cat:

expr `tail -n 5147 Log1.log.2013-11-18 | grep "The line I need to count" | wc -l` + `cat Log2.log Log3.log | grep "The line I need to count" | wc -l` | awk '{print $1"*200"}' | bc

example:

$ echo abc > 1.txt 
$ echo cde > 2.txt 
$ echo def > 3.txt 
$ cat 1.txt 2.txt 3.txt | grep d 
cde 
def
  • Thank you for the input. Your suggestion is accurate but I was looking for a cleaner version of my command as well. – FGreg Nov 20 '13 at 2:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.