I have about 800 log files and I'm trying to write a script that reports the counts of lines per second or "requests per second" in each log file and reports the output which includes the timestamp for the highest lines per second count and the log file name and the highest number per second from that file and then go to the second log file and do the same thing then append the results to the same output file.

The log file snippet is shown below.

I'm currently using this command to get the results from each log but doing this for 800 log files is not practical

grep "2017-02-22" "LogFile.log" | cut -c1-20 | uniq -c | sort -n | tail -n1

This command reports a list of lines count per 10 seconds but it's not very efficient

So the output file should look like this:

Date/time    "1st LogFileName"    "highest requests per second for this log"
Date/time    "2nd LogFileName"    "highest requests per second for this log"

#Fields: date time time-taken c-ip cs-username cs-auth-group x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action cs-method
 rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes x-virus-id x-bl
uecoat-application-name x-bluecoat-application-operation
#Remark: 1412140034 "lofnetsg1" "" "main"
2017-02-15 17:49:06 8 - - - OBSERVED "Technology/Internet" -  404 TCP_NC_MISS GET text/html;%20charset=iso-8859-1 http dci.sophosupd.c
om 80 /osxhe/d/16/d16e39472e867bf7f8407db13d3502b5.dat - dat "Sophos%20AutoUpdate/ CFNetwork/760.6.3 Darwin/15.6.0 (x86_64)" 437 243
- "none" "none"
2017-02-15 17:49:06 578 - - - OBSERVED "Web Ads/Analytics" -  200 TCP_TUNNELED CONNECT - tcp tpc.googlesyndication.com 443 / - - "Mozill
a/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" 21106 2873 - "none" "none"
2017-02-15 17:49:06 17 - - - OBSERVED "Web Ads/Analytics" -  200 TCP_NC_MISS GET - http t.lkqd.net 80 /t ?ev=147&tsid=132117&env=3&cb=7
&lkqdtagtype=platform-connection&tlr=0&adid=127634&asrc=15817&dealid=117723&c1=&c2=&c3=&tsia=8945&tssa=&rnd=157089284&cs=3668523497 - "Mozilla/5.0
(Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 290 1534 - "none" "none"
2017-02-15 17:49:06 669 - - - OBSERVED "Radio/Audio Streams" -  200 TCP_MISS GET audio/x-aac http primary.hls-streaming.production.str
eaming.siriusxm.com 80 /AAC_Data/broadwaysbest/HLS_broadwaysbest_64k_v3/broadwaysbest_64k_1_021560803727_00310913_v3.aac ?consumer=k2&gupId=BE4A750
32FEC2FF7293A7F5C0FAEFEA3 aac "SXMLiveAudioPlayer/1485824608 CFNetwork/808.3 Darwin/16.3.0" 79798 1119 - "none" "none"
2017-02-15 17:49:06 32 - - - OBSERVED "Web Ads/Analytics" Error
.html  200 TCP_NC_MISS GET text/javascript http beacon.krxd.net 80 /optout_check ?callback=Krux.ns._default.kxjsonp_optOutCheck - "Mozilla/5.0 (Win
dows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 635 451 - "none" "none" 
  • post some log file contents and the expected output – Kamaraj Feb 17 '17 at 4:55

You need a position file. The position file should store the time and the size (in byte) of the file in initial read or the position of the file in last read.

You'd then use tail -c +bytepos to seek to that position, and then restrict the output using sed '/PATTERN/!q' FILE which should copy its stdout until it no longer matches PATTERN. You'd want to generate a negative pattern though. From there just pipe wc to count lines.

So something like (untested):

tail -c +bytepos | sed '/timestamp/!q' | wc
  • That didn't do it.. Sorry – Katkota Feb 19 '17 at 4:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.