2

I'm using the following command to get the highest number of requests per second in a log file and it works well.

grep "2017-02-22" "LogFile.log" | cut -c1-20 | uniq -c | sort -n | tail -n1

Now I would like to also get the smallest requests per second and the amount of time the highest number stayed for. For example: let's say the results from the command was 2000 which is the highest requests per second inside the log file, I want to find out how long the 2000 requests lasted for? in another word: if the peak of 2000 happened, I'd like to find out how long it took before this went down.

Here's a portion of the log file:

#Start-Date: 2017-02-16 19:49:06
#Date: 2016-10-11 15:16:48
#Fields: date time time-taken c-ip cs-username cs-auth-group x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action c
s-method rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes
 x-virus-id x-bluecoat-application-name x-bluecoat-application-operation
#Remark: 1412140034 "lofnetsg1" "192.168.13.14" "main"
2017-02-16 19:49:06 116154 10.5.13.149 - - - OBSERVED "Non-Viewable/Infrastructure" -  200 TCP_TUNNELED CONNECT - tcp u-amvx4npjuy.wc.yahoo
dns.net 443 / - - "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 660
3 1036 - "none" "none"
2017-02-16 19:49:06 1 10.2.29.41 - - - OBSERVED "Technology/Internet" -  304 TCP_HIT GET application/pkix-crl http www.microsoft.com 80 /pk
i/CRL/products/Microsoft%20Windows%20Hardware%20Compatibility%20PCA(1).crl - crl "Microsoft-CryptoAPI/6.1" 192.168.13.14 568 338 - "none" "
none"
2017-02-16 19:49:06 18 10.1.15.166 - - - OBSERVED "Content Servers" http://www.foxnews.com/  304 TCP_CLIENT_REFRESH GET text/javascript;cha
rset=UTF-8 http widget-cdn.rpxnow.com 80 /translations/share/en - - "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Ge
cko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 487 417 - "none" "none"
2017-02-16 19:49:06 6677 172.16.121.69 - - - OBSERVED "Social Networking;Content Servers" -  200 TCP_TUNNELED CONNECT - tcp pbs.twimg.com 4
43 / - - "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 10020
3 1241 - "Twitter" "none"
2017-02-16 19:49:06 1664 10.14.16.67 - - - OBSERVED "Informational;Health" -  200 TCP_TUNNELED CONNECT - tcp www.drugs.com 443 / - - "Mozil
la/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" 192.168.13.14 6313 2281 - "none" "none"
2017-02-16 19:49:06 1095 172.16.121.69 - - - OBSERVED "Web Ads/Analytics" -  200 TCP_TUNNELED CONNECT - tcp as-sec.casalemedia.com 443 / -
- "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 1058 2818 -
"none" "none"
2017-02-16 19:49:06 24282 172.16.121.69 - - - OBSERVED "Web Ads/Analytics" -  200 TCP_TUNNELED CONNECT - tcp dt.adsafeprotected.com 443 / -
 - "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 3687 3007 -
 "none" "none"
2017-02-16 19:49:06 1 10.2.29.41 - - - OBSERVED "Non-Viewable/Infrastructure" -  304 TCP_HIT GET application/pkix-crl http crl.microsoft.co
m 80 /pki/crl/products/MicrosoftTimeStampPCA.crl - crl "Microsoft-CryptoAPI/6.1" 192.168.13.14 500 304 - "none" "none"
2017-02-16 19:49:06 48 10.2.50.46 - - - OBSERVED "Web Ads/Analytics" -  200 TCP_TUNNELED CONNECT - tcp x.bidswitch.net 443 / - - "Mozilla/5
.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 39 219 - "none" "none"
2017-02-16 19:49:06 26855 172.16.121.69 - - - OBSERVED "Web Ads/Analytics" -  200 TCP_TUNNELED CONNECT - tcp ping.chartbeat.net 443 / - - "
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 699 2727 - "non
e" "none"
2017-02-16 19:49:06 22 10.2.10.172 - - - OBSERVED "Web Ads/Analytics" http://player.radio.com/listen/station/985-the-sports-hub  200 TCP_NC
_MISS GET application/javascript;%20charset=utf-8 http ib.adnxs.com 80 /ttj ?id=10203641&size=300x250&pagetype=ros&promo_sizes=&cb=14872745
46795 - "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 11458 3251 -
"none" "none"
2017-02-16 19:49:06 965 10.32.14.38 - - - OBSERVED "Technology/Internet" -  200 TCP_TUNNELED CONNECT - tcp clients4.google.com 443 / - - "C
hrome WIN 56.0.2924.87 (0e9a9a6f3676ae439b78cd9b3f62b4193c3ac7d5-refs/branch-heads/2924@{#895}) channel(stable)" 192.168.13.14 1455 3073 -
"none" "none"
2017-02-16 19:49:06 939 10.7.18.97 - - - OBSERVED "Health" http://cmri.in/cmri-doctors/  200 TCP_NC_MISS GET text/html;%20charset=UTF-8 htt
p cmri.in 80 /doctor/dr-mahesh-chowdhury/ - - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:51.0) Gecko/20100101 Firefox/51.0" 192.168.
13.14 7501 573 - "none" "none"
2
  • Please add a few sample lines of Logfile.log
    – MikeD
    Mar 2, 2017 at 15:54
  • Please let me know if the answer worked for you or, if not, what remaining issues you are seeing. Thanks!
    – MikeD
    Mar 10, 2017 at 23:53

1 Answer 1

1

Here is one way obtain how long the most hits lasted:

Assign a variable, highest to your original command: *adding ^ in the grep pattern, restricts matches to dates at the beginning of a line.

  highest=$(grep "^2017-02-22" "LogFile.log" | cut -c1-20 | uniq -c | sort -n | tail -n1)  

Again using your original command, up to uniq, redirect the full uniq count list to a tempfile

  grep "^2017-02-22" "LogFile.log" | cut -c1-20 | uniq -c >tempfile  

Use grep to search for the highest number in the new tempfile using -A1 to get the first line after:

  grep -A1 "^$highest" tempfile | tail -n1  

You may combine these into one line using semi-colons (;) like so:

  highest=$(grep "^2017-02-22" "LogFile.log" | cut -c1-20 | uniq -c | sort -n | tail -n1);grep "^2017-02-22" "LogFile.log" | cut -c1-20 | uniq -c >tempfile;grep -A1 "^$highest" tempfile | tail -n1

If you need to do the date-time math for output of the actual time difference, you may capture the result so far to a variable nexttime

  nexttime=$(highest=$(grep "^2017-02-22" "LogFile.log" | cut -c1-20 | uniq -c | sort -n | tail -n1);grep "^2017-02-22" "LogFile.log" | cut -c1-20 | uniq -c >tempfile;grep -A1 "^$highest" tempfile | tail -n1)

and use this to calculate the difference:

  highest=$(echo $highest | cut -d' ' -f 3)              # get the time field (#3)
  nexttime=$(echo $nexttime | cut -d' ' -f 3)            # get the time field (#3)
  logStart=$(date -u -d "$highest" +"%s")                # convert to seconds
  logEnd=$(date -u -d "$nexttime" +"%s")                 # convert to seconds
  date -u -d "0 $logEnd sec - $logStart sec" +"%H:%M:%S" # display time difference

And, likewise, you may combine these statements with the others above, using semi-colons (;), or put it all in a script.

3
  • Thanks for the hint but how can i find out how long the high number lasted for?
    – Katkota
    Mar 2, 2017 at 14:37
  • I have updated my answer based on your added log file sample.
    – MikeD
    Mar 3, 2017 at 20:36
  • Thanks Mike for all the help. I will try it this week and let you know
    – Katkota
    Mar 5, 2017 at 1:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .