4

I'm trying to trigger a beep on the PC speaker for every unique visitor of a website.

After some brainstorming, it seemed to be possible with one line:

for e in `ssh me@mymachine "tail -n 1 -f /var/log/apache2/test.log | awk '{print $1}' | uniq"`; do beep; done

However uniq doesn't output anything as long as stdin is open (seems to wait for EOF). The same goes for the for loop. If I remove the uniq from the chain, I still get no output with tail keeping the pipe open.

This seems not to be because of buffering. Even if I write >100.000 lines into the test file with this command running, there's no output on the other end.

Is there a way to get that working without completely killing the beauty (simplicity) of the solution?

Update

I solved the first part. uniq is unblocked by prefixing the tail command with stdbuf -oL -eL (see https://unix.stackexchange.com/a/25378/109296). The same doesn't work for the loop.

Update 2

I got it working - but not exactly according to my spec and with 2 lines:

while [ 1 -eq 1 ]; do ssh root@speedy "stdbuf -oL -eL tail -n 1 -f /var/log/apache2/www.access.log | stdbuf -oL -eL grep 'GET / '"; sleep 60; done > www.log

awk '{print $1}' is missing because it didn't work inside this construct (just passed through the whole line). I don't know why. But I can live without, because anyway uniq turned out not to be so useful after all, because it only looks at adjacent lines, which means that the requests patterns ip1, ip2, ip1 would still let ip1 through twice. uniq -u would do what I expect, but it has the same problem like sort: doesn't output anything as long as stdin is open (not even with stdbuf -oL.

This command just writes all requests for the base URL (/) to another file. I wrapped it into a loop (and wait) in order to have it automatically retry if for some reason the pipe or connection interrupts.

while inotifywait -e modify www.log; do beep -f 250; done makes the sound! I could not get the bash for loop to process line by line unbuffered, also tried while read with the same result. Thus I gave up and went on with inotifywait which however means that I need an intermediate file (maybe a named pipe would also work, didn't try. Doesn't really make a difference for me).

I'd still be thankful to contributions which help to make the filtering of unique visitors work (without escalating complexity).

This will be a nice surprise for my team members when they return to the office :-)

I plan to extend this notification system to monitor several events, using different audio frequencies. That's the best job I've found so far for an old server collecting dust...

12
  • uniq won't work because the input is unsorted.
    – badp
    Apr 6, 2015 at 1:37
  • 1
    Yes, I noticed that. However sort blocks the chain, I couldn't yet find a solution to this problem.
    – didi_X8
    Apr 6, 2015 at 2:01
  • it's actually not about stdin at all - but about stdout. they're waiting until they have enough output to fill their outbuffers probably. Did your 100k test lines contain uniqe entries? If not, that's all data they would be eliding anyway - and so not printing - and it probably made no effect. I would think, though, that the pipeline wouldn't be the greatest way to go longterm - rather you'd want a database. You have to keep resorting the same input over and over or else keep it all in memory. You might hack up a table w/ ar. tsort could be worthwhile as well.
    – mikeserv
    Apr 6, 2015 at 5:54
  • 1
    Oh, so you're not actually checking for uniques or maybe the log only gets updated when there are uniques? In either case, (while running an interactive POSIX shell) you can use MAILFILE=/var/log/apache2/www.access.log%$(beep) which will get the shell to watch the file for modification and expand the % user-specified message every time it does.
    – mikeserv
    Apr 6, 2015 at 8:35
  • @mikeserv: Thx for contributing.
    – didi_X8
    Apr 6, 2015 at 14:26

2 Answers 2

1

I think I understand what you are trying to accomplish:

  1. For each hit to the web site, which is logged by the web server:
  2. If the visit is "unique" (how do you define this??) log the entry and send an audible notification.

The trick is how you define "unique". Is it by URL, by IP address, by cookie? Your approach with awk was arguably the right way to go, but you got snagged by shell-escaping rules.

So here is something that sort of combines your approaches. First, you really need a script on the web server to do this. Otherwise you're going to be lost in complex quotation-escaping rules. Second, I'm assuming your web-server is using the "common-log format", which frankly, sucks for this kind of work, but we can work with it.

while true; do 
  ssh root@speedy remote-log-capturing-script
done > unique-visits.log

Use mikeserv's excellent suggestion about MAILFILE. The script on speedy should look like this:

#!/bin/sh
tail -1f /var/log/apache2/www.access.log | 
awk '$(NF-1) == 200' | 
grep --line-buffered -o '"GET [^"]*"' |
awk '!url[$1]{ print; url[$1]=1 }'

Awk is always line-buffered. The first awk ensures you're only getting actual successful hits, not cached-hits or 404s. The grep -o prints out only the matching part of the input, in this case, the URL. (This is GNU grep, which I assume you are using. If not, use the stdbuf trick.) The next awk uses a little expression to conditionally print out the input line -- only if that input line was never before seen.

You can also do this with perl to achieve more complexity within one fork:

#!/bin/sh
tail -1f /var/log/apache2/www.access.log | 
perl -lane '$|=1;' \
  -e 'if ($F[$#F-1] eq "200" and ' \
  -e ' /\s"GET\s([^"]*)"\s/ and !$url{$1}) { '\
  -e '  print $1;$url{$1}=undef; }'

Now both of these will only print unique URLs. What if two web clients from different IPs hit the same page? You only get one output. To change that, with the perl solutions, this is easy: modify the key that goes into url.

 $url{$F[0],$1}

When using perl -a, $F[0] represents the first white-space-delimited field of input, just like awk's $1 -- ie, the connecting hostname/IP address. And perl's $1 represents the first matching subexpression of the regular-expression /\s"GET\s([^"]*)"\s/, ie, just the URL itself. The cryptic $F[$#F-1] means 2nd-to-last field of the input line.

1
  • 1
    Thanks for the detailled explanation. I didn't try it, but have no reason to believe it doesn't work. It seems to be quite similar to the latest iteration I posted in the initial post, just more verbose. The compact perl command contributed by JJoao makes it possible to avoid the escape hell you mention and thus doesn't require a server side script.
    – didi_X8
    Aug 6, 2015 at 23:27
0

This is what I came finally up with, thanks to the neat Perl command contributed by JJoao:

# kill everything on termination
trap "kill 0" SIGINT SIGTERM
# Make sure the remote processes are killed on exit, see http://unix.stackexchange.com/questions/103699/kill-process-spawned-by-ssh-when-ssh-dies
shopt -s huponexit
( while [ 1 -eq 1 ]; do ssh -t -t root@speedy "stdbuf -oL -eL tail -n 1 -f /var/log/apache2/www.access.log | stdbuf -oL -eL grep 'GET / ' |  stdbuf -oL -eL perl -naE '($a{$F[0]}++ == 0) and say $F[0]'"; sleep 60; done > www.log ) &
( while inotifywait -e modify www.log; do beep -f 250; done ) &

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .