1

I'm trying to print a list of the most frequent visits from User Agents from the Nginx access log.

I have so far come up with this:

cat /var/log/nginx/access.log | awk -F\" '{print $6}' | sort | uniq -c | sort -nr | head -20

And this does indeed give me the top 20 User Agents sorted by how many times they have visited the website.

I would like to further modify this to only count visits in the last 30 days though and I'm struggling to get a working command.

cat from=$(date -d'now-30 days' +[%d/%b:%H:%M:%S) to==$(date +[%d/%b:%H:%M:%S) /var/log/nginx/access.log | awk -F\" '{print $6}' | sort | uniq -c | sort -nr | head -20

The purpose of this is to identify search engine crawlers/bots visiting the website very frequently so if you can recommend anything else for this, please let me know.

I include a few example lines from the log file below:-

157.55.39.46 - - [11/Jun/2018:13:23:20 +0100] "GET /accessories/antique-gold--pine--wall-mounted?width=1308%2C1334 HTTP/1.1 "200 27726 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"[RT:3.041] [C:995725]
157.55.39.46 - - [11/Jun/2018:13:23:24 +0100] "GET /accessories/antique-gold--chrome--pine--wall-mounted?limit=25&mode=list HTTP/1.1 "200 34206 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"[RT:3.412] [C:995725]
54.36.148.11 - - [11/Jun/2018:13:23:29 +0100] "GET /bathroom-mirrors/mirrors/700 HTTP/1.1 "404 10570 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)" "-"[RT:0.393] [C:995737]

Currently, the command prints the following:-

8603 Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
4051 Mozilla/5.0 (compatible; SemrushBot/2~bl; +http://www.semrush.com/bot.html)
1707 Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
1585 Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.79 Safari/537.36
1519 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
1368 Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
1185 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36
 993 Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)
 903 Mozilla/5.0 (iPhone; CPU iPhone OS 11_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.0 Mobile/15E148 Safari/604.1
 658 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
 648 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1 Safari/605.1.15
 572 Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0
 435 Mozilla/5.0 (iPhone; CPU iPhone OS 11_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.0 Mobile/15E148 Safari/604.1
 336 Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
 324 Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0
 324 Mozilla/5.0 (iPad; CPU OS 10_3_3 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) CriOS/67.0.3396.69 Mobile/14G60 Safari/602.1
 319 Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0
 309 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
 295 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.79 Safari/537.36
 280 Mozilla/5.0 (iPad; CPU OS 11_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.0 Mobile/15E148 Safari/604.1
3
  • could we have one or two lines of your logs ? whit false ip and domain name if necessary it's hard to tell without the source
    – Kiwy
    Commented Jun 11, 2018 at 12:13
  • Yes sure, I've added to the original question. Thanks.
    – zigojacko
    Commented Jun 11, 2018 at 12:28
  • If your access logs have more than 30 days of material, probably worth checking out logrotate. Then you can just run this on the most recent file. Commented Jul 11 at 13:26

0

You must log in to answer this question.

Browse other questions tagged .