How to grep the logs for a list of patterns and make a report out of it?

Question

I have a C++ executable which scan some of my files and those files has some user id in it. After scanning is completed it generates below log file (abc.log) like this -

INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:110) - checking file: "p1_weekly_1980_32_200003_5.data" with path: "/database/batch/p1_snapshot/p1_weekly_1980_32_200003_5.data"
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 45576752 with value badge_leaf_cat and status -2
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 870206432 with value badge_leaf_cat and status -2
INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:120) - golden_file: /database/batch/p1_snapshot//p1_weekly_1980_32_200003_5.data is valid
INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:110) - checking file: "p1_weekly_1980_13_200003_5.data" with path: "/database/batch/p1_snapshot/p1_weekly_1980_13_200003_5.data"
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 876269533 with value badge_leaf_cat and status -2
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 17256973 with value badge_leaf_cat and status -2
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 830173693 with value badge_leaf_cat and status -2
INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:120) - golden_file: /database/batch/p1_snapshot//p1_weekly_1980_13_200003_5.data is valid
INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:110) - checking file: "p1_weekly_1980_0_200003_5.data" with path: "/database/batch/p1_snapshot/p1_weekly_1980_0_200003_5.data"
ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgrUtil.cc:493) - failed reading user id: 18446744073135142816 num attributes: 0 seeing 1 bad records from 365 records
ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgrUtil.cc:493) - failed reading user id: 18446744073698151136 num attributes: 0 seeing 2 bad records from 595 records
ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgrUtil.cc:493) - failed reading user id: 18446744072929739296 num attributes: 0 seeing 3 bad records from 1214 records
ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:117) - golden_file: /database/batch/p1_snapshot//p1_weekly_1980_0_200003_5.data is corrupt

Now I need to grep the above logs and find out how many files it has been scanned, how many user id failed, and which files are corrupt.

• For how many files it has scanned, I need to look for checking file words in each line and basis on that increase the count for number of files it has scanned.
• For how many unique user id failed, I need to look for failed reading user id words in a line and basis on that increase the count and then provide the list of user id's which failed.
• If any user id failed, then it means the file will be corrupt which was holding that user id, so I need to look for is corrupt word in each line and find the file name which is corrupt. In general this file database/batch/p1_snapshot//p1_weekly_1980_0_200003_5.data is corrupted.

And below is the response I wanted to see after scanning the above logs -

Total Number of Files Scanned - 1000
Total Number of Unique User ID failed - 10000
Total Number of Files Corrupted - 5

List of Unique User Id's which are corrupt - 
UserId-A
UserId-B

Files which are corrupted - 
FileName-A
FileName-B

How would I go ahead and get the above result after scanning the logs using grep?

Answer 1

Try the following script:

#!/bin/bash

logfile="$1"

nfiles=$(grep -c 'checking file' "$logfile")
failed_userid=($(grep -oP 'failed reading user id: \K[^ ]*' "$logfile"))
corrupted_files=($(grep -oP '[^ ]*(?= is corrupt)' "$logfile"))


echo "Total Number of Files Scanned - $nfiles"
echo "Total Number of Unique User ID failed - ${#failed_userid[@]}"
echo "Total Number of Files Corrupted - ${#corrupted_files[@]}"
echo

echo "List of Unique User Id's which are corrupt - "
for uid in "${failed_userid[@]}"; do
   echo "$uid"
done

echo

echo "Files which are corrupted - "
for corf in "${corrupted_files[@]}"; do
   echo "$corf"
done

Run it with

$ ./script file.log

The result for input from your question looks like

Total Number of Files Scanned - 3
Total Number of Unique User ID failed - 3
Total Number of Files Corrupted - 1

List of Unique User Id's which are corrupt - 
18446744073135142816
18446744073698151136
18446744072929739296

Files which are corrupted - 
/database/batch/p1_snapshot//p1_weekly_1980_0_200003_5.data

Short explanation:

• -c option of grep counts the matching lines
• -P enables perl regular expresions syntax
• -o matches only part of lines
• (?= construct is the so called positive look-ahead (take it as pattern, but do not include to the output)
• \K is look-behind assertion (take whole pattern, but throw away from result everything up to this point)

The rest should be obvious. Be aware however that I've assumed there are no whitespaces in file names!