I am looking for a way to the following.

Here is one line from nginx's error log.

2014/12/29 21:17:57 [error] 30078#0: *311826 openat() "/var/www/html/images/images/Outlet-Celine-Boston-Square-Calfskin-Bags-Red_celine_2140_1.jpg" failed (2: No such file or directory), client:, server: server.domain.tld, request: "GET /images/images/Outlet-Celine-Boston-Square-Calfskin-Bags-Red_celine_2140_1.jpg HTTP/1.1", host: "www.buylvneverfullpm.net"

There are thousands more, so I need a way to :

  1. find if a line contains 'host: "*"', where * is the url
  2. get the value of host: "www.xxx.yyy"
  3. strip the www at the beginning
  4. count how many times each xxx.yyy occurs
  5. sort the number of occurrences for each unique host value, from highest to lowest.

I have been using the following:

awk '($20 ~ /GET/)' /var/log/nginx/error_log | awk '{print $24}' | sort | uniq -c | sort -rn

but since it doesn't strip the "www", there are repeated entries.

1 Answer 1


Just get awk to handle everything for you

awk '$20 ~ /GET/{gsub(/"/, "", $24); sub(/[^.]*\./, "", $24); a[$24]++};
END{for (k in a)print k, a[k]}' /var/log/nginx/error_log
  • I like your approach, but I would like to take this a notch further. In my question above, I did not account for TLDs of the form "co.uk". I should also mention that the position of "host: *" varies within the log. It's not always $24 with awk. I am accepting your answer for now, since you've made my life easier. Thank you very much. Commented Dec 30, 2014 at 17:47
  • @Pomegranate, you are welcome. Please ask another question with those specifics; this is the right forum
    – iruvar
    Commented Dec 30, 2014 at 17:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .