Cfrtjryk

I'm using the following command to get the highest number of requests per second in a log file and it works well.

grep "2017-02-22" "LogFile.log" | cut -c1-20 | uniq -c | sort -n | tail -n1

Now I would like to also get the smallest requests per second and the amount of time the highest number stayed for. For example: let's say the results from the command was 2000 which is the highest requests per second inside the log file, I want to find out how long the 2000 requests lasted for? in another word: if the peak of 2000 happened, I'd like to find out how long it took before this went down.

Here's a portion of the log file:

#Start-Date: 2017-02-16 19:49:06

#Date: 2016-10-11 15:16:48

#Fields: date time time-taken c-ip cs-username cs-auth-group x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action c

s-method rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes

 x-virus-id x-bluecoat-application-name x-bluecoat-application-operation

#Remark: 1412140034 "lofnetsg1" "192.168.13.14" "main"

2017-02-16 19:49:06 116154 10.5.13.149 - - - OBSERVED "Non-Viewable/Infrastructure" -  200 TCP_TUNNELED CONNECT - tcp u-amvx4npjuy.wc.yahoo

dns.net 443 / - - "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 660

3 1036 - "none" "none"

2017-02-16 19:49:06 1 10.2.29.41 - - - OBSERVED "Technology/Internet" -  304 TCP_HIT GET application/pkix-crl http www.microsoft.com 80 /pk

i/CRL/products/Microsoft%20Windows%20Hardware%20Compatibility%20PCA(1).crl - crl "Microsoft-CryptoAPI/6.1" 192.168.13.14 568 338 - "none" "

none"

2017-02-16 19:49:06 18 10.1.15.166 - - - OBSERVED "Content Servers" http://www.foxnews.com/  304 TCP_CLIENT_REFRESH GET text/javascript;cha

rset=UTF-8 http widget-cdn.rpxnow.com 80 /translations/share/en - - "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Ge

cko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 487 417 - "none" "none"

2017-02-16 19:49:06 6677 172.16.121.69 - - - OBSERVED "Social Networking;Content Servers" -  200 TCP_TUNNELED CONNECT - tcp pbs.twimg.com 4

43 / - - "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 10020

3 1241 - "Twitter" "none"

2017-02-16 19:49:06 1664 10.14.16.67 - - - OBSERVED "Informational;Health" -  200 TCP_TUNNELED CONNECT - tcp www.drugs.com 443 / - - "Mozil

la/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" 192.168.13.14 6313 2281 - "none" "none"

2017-02-16 19:49:06 1095 172.16.121.69 - - - OBSERVED "Web Ads/Analytics" -  200 TCP_TUNNELED CONNECT - tcp as-sec.casalemedia.com 443 / -

- "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 1058 2818 -

"none" "none"

2017-02-16 19:49:06 24282 172.16.121.69 - - - OBSERVED "Web Ads/Analytics" -  200 TCP_TUNNELED CONNECT - tcp dt.adsafeprotected.com 443 / -

 - "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 3687 3007 -

 "none" "none"

2017-02-16 19:49:06 1 10.2.29.41 - - - OBSERVED "Non-Viewable/Infrastructure" -  304 TCP_HIT GET application/pkix-crl http crl.microsoft.co

m 80 /pki/crl/products/MicrosoftTimeStampPCA.crl - crl "Microsoft-CryptoAPI/6.1" 192.168.13.14 500 304 - "none" "none"

2017-02-16 19:49:06 48 10.2.50.46 - - - OBSERVED "Web Ads/Analytics" -  200 TCP_TUNNELED CONNECT - tcp x.bidswitch.net 443 / - - "Mozilla/5

.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 39 219 - "none" "none"

2017-02-16 19:49:06 26855 172.16.121.69 - - - OBSERVED "Web Ads/Analytics" -  200 TCP_TUNNELED CONNECT - tcp ping.chartbeat.net 443 / - - "

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 699 2727 - "non

e" "none"

2017-02-16 19:49:06 22 10.2.10.172 - - - OBSERVED "Web Ads/Analytics" http://player.radio.com/listen/station/985-the-sports-hub  200 TCP_NC

_MISS GET application/javascript;%20charset=utf-8 http ib.adnxs.com 80 /ttj ?id=10203641&size=300x250&pagetype=ros&promo_sizes=&cb=14872745

46795 - "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 11458 3251 -

"none" "none"

2017-02-16 19:49:06 965 10.32.14.38 - - - OBSERVED "Technology/Internet" -  200 TCP_TUNNELED CONNECT - tcp clients4.google.com 443 / - - "C

hrome WIN 56.0.2924.87 (0e9a9a6f3676ae439b78cd9b3f62b4193c3ac7d5-refs/branch-heads/2924@{#895}) channel(stable)" 192.168.13.14 1455 3073 -

"none" "none"

2017-02-16 19:49:06 939 10.7.18.97 - - - OBSERVED "Health" http://cmri.in/cmri-doctors/  200 TCP_NC_MISS GET text/html;%20charset=UTF-8 htt

p cmri.in 80 /doctor/dr-mahesh-chowdhury/ - - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:51.0) Gecko/20100101 Firefox/51.0" 192.168.

13.14 7501 573 - "none" "none"

Here is one way obtain how long the most hits lasted:

Assign a variable, highest to your original command: *adding ^ in the grep pattern, restricts matches to dates at the beginning of a line.

  highest=$(grep "^2017-02-22" "LogFile.log" | cut -c1-20 | uniq -c | sort -n | tail -n1)  

Again using your original command, up to uniq, redirect the full uniq count list to a tempfile

  grep "^2017-02-22" "LogFile.log" | cut -c1-20 | uniq -c >tempfile  

Use grep to search for the highest number in the new tempfile using -A1 to get the first line after:

  grep -A1 "^$highest" tempfile | tail -n1  

You may combine these into one line using semi-colons (;) like so:

  highest=$(grep "^2017-02-22" "LogFile.log" | cut -c1-20 | uniq -c | sort -n | tail -n1);grep "^2017-02-22" "LogFile.log" | cut -c1-20 | uniq -c >tempfile;grep -A1 "^$highest" tempfile | tail -n1

If you need to do the date-time math for output of the actual time difference, you may capture the result so far to a variable nexttime

  nexttime=$(highest=$(grep "^2017-02-22" "LogFile.log" | cut -c1-20 | uniq -c | sort -n | tail -n1);grep "^2017-02-22" "LogFile.log" | cut -c1-20 | uniq -c >tempfile;grep -A1 "^$highest" tempfile | tail -n1)

and use this to calculate the difference:

  highest=$(echo $highest | cut -d' ' -f 3)              # get the time field (#3)

  nexttime=$(echo $nexttime | cut -d' ' -f 3)            # get the time field (#3)

  logStart=$(date -u -d "$highest" +"%s")                # convert to seconds

  logEnd=$(date -u -d "$nexttime" +"%s")                 # convert to seconds

  date -u -d "0 $logEnd sec - $logStart sec" +"%H:%M:%S" # display time difference

And, likewise, you may combine these statements with the others above, using semi-colons (;), or put it all in a script.