0

I have two web servers, and I want to know the top 10 ips sorted by the number of requests.

The webservers are Apache based, so I need to look at an access.log file.

The problem is since these files are huge, I really wish to not transfer them locally, thus I would like to know if there is a way to do this in streaming. I do have ssh access to these machines.

One way I though, would be running something like this:

awk "{ print $1 }" access.log | sort | uniq -c | sort -n | tail 

On both the machines, and then somehow locally combine the results but this is obviously wrong.

  • How accurate does this have to be? For example if the first webserver has received 1000 requests from 10 hosts and 999 from an 11th (total 10999) and the second webserver has received the same number of requests but from 10000 hosts and no more than 2 requests from each one then you need to combine the full information to determine if you have 1000, 1001 or 1002 requests for the first 10 and 999, 1000 or 1001 from the 11th. – icarus Dec 19 '19 at 12:46
  • Is the concern about the size of the access.log files related to the disk space to store a copy or the network bandwidth (and hence time) to transfer them? – icarus Dec 19 '19 at 12:55
  • The only concern is with store of both files, not with the bandwidth. – Federico Ponzi Dec 19 '19 at 13:47
1

Since the concern is only with the filesize conceptually all that is needed is

{
ssh server2 cat /path/to/access.log
cat /local/path/to/access.log
} | awk '{print $1}' | sort | uniq -c | sort -n | tail

however there are a number of things that can be done to improve the speed.

First only send the ip addresses across the network, to reduce the bandwidth.

{
ssh server2 awk '{print $1}' /path/to/access.log
cat /local/path/to/access.log
} | awk '{print $1}' | sort | uniq -c | sort -n | tail

Second take advantage of awk's hashing to remove the need to sort. This replaces an order n*lg(n) with an order n. This uses an associative array called seen to count how many times each ip address is seen, and at the end print out the count and the address.

{
ssh server2 awk '{print $1}' /path/to/access.log
cat /local/path/to/access.log
} | awk '{seen[$1]++} END {for (i in seen){print seen[i],i}}' | sort -n | tail

Third reverse the sort, again to reduce the amount of data that needs to flow

{
ssh server2 awk '{print $1}' /path/to/access.log
cat /local/path/to/access.log
} | awk '{seen[$1]++} END {for (i in seen){print seen[i],i}}' | sort -rn | head

Depending on the data, it would probably make sense to pre-process the data on the remote web server. (seen array renamed to s to save typing). Here the data being sent is count and address pairs. We then add them up locally in a third awk process.

{
ssh server2 awk '{s[$1]++}END{for (i in s){print s[i],i}}' /path/to/access.log
awk '{s[$1]++}END{for (i in s){print s[i],i}}' /local/path/to/access.log
} | awk '{s[$2]+=$1}END{for (i in s){print s[i],i}}' | sort -rn | head

Untested of course.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.