22

I'm trying to see how many times foo bar appears in /var/log/foo.log within an arbitrary amount of time on a remote server, but nothing that I've tried so far has worked.

I've already got a timer script that I use to keep track of how long it has been since I started tailing /var/log/foo.log, and now I'd just like a way to tell how many times foo bar has appeared in the tailed output.

I searched google, but I didn't find anything pertinent within the first 10 pages of results.

Here's what I've tried with frustrating results:

## works on local machine, but doesn't work as expected on remote
tail -f /var/log/foo.log | grep foo\ bar | sed '='

## works on local, but not remote
tail -f /var/log/foo.log | grep foo\ bar | cat -n -

##  works on local, but not remote
tail -f /var/log/foo.log | grep foo\ bar | awk -F'\n' '{printf "[%d]> ", NR; print $1}'

I even tried to write a sed script that'd act like tail -f, but I made limited-to-no headway with that.

NOTE

the remote server is running an older version of coreutils, and upgrading is an option, but is NOT in any way the desired solution.

  • 2
    In which way does it not work? Try the --line-buffered option to grep. Or tail -f ... | awk '/foo bar/{print ++n, $0}' – Stéphane Chazelas Jan 17 '14 at 22:44
  • Why it doesn't work on remote? Example: tail -f /var/log/log.log | awk '{ printf "[%d]> %s\n", NR+1 ,$0; fflush(stdout); }' – user55518 Jan 17 '14 at 22:46
29
tail -f | nl

works for me and is the first what I thought of - that is if you really want the lines numbered from 1 and not with the real line number from the file watched. Optionally add grep if needed to the appropriate place (either before or after nl). However, remember that buffering may occur. In my particular case, grep has the --line-buffered option, but nl buffers it's output and doesn't have an option to switch that off. Hence the tail | nl | grep combo doesn't really flow nicely.

That said,

tail -f | grep -n pattern

works for me as well. Numbering starts again from the beginning of the "tailing" rather than beginning of the whole log file.

  • the version of grep running on the server doesn't have a -n option. – Alexej Magura Jan 20 '14 at 15:56
  • it does, however, have the long option --line-number: tail -f /var/log/foo.log | grep foo\ bar --line-number works! – Alexej Magura Jan 20 '14 at 16:56
  • 1
    That's interesting - I haven't checked POSIX as such, but the GNU grep manpage says: -n is specified by POSIX. – peterph Jan 20 '14 at 20:06
16

I think this is better..

less -N +F <filepath>
  • 2
    Could you explain why you think its better? – Navigatron Nov 18 '15 at 20:16
  • This is a large edit, I'm reverting. – Adam Eberlin May 26 '16 at 22:21
  • 3
    Shows the line number taking as reference the entire file. tail -f | nl shows the line number taking as reference the first output of tail. – rafaelvalle Dec 26 '17 at 13:48
  • This is very handy and addresses the OP's Title but not their question. They wanted to know how many times X appears in a file :P – Timmah Jun 18 at 4:02
6

You can also pipe the output to less, it has a line number feature, -N which would allow you to scroll back and forth through the log.

$ tail -f /var/log/foo.log | less -N

Example

  1 Jan 17 22:11:58 greeneggs fprintd[4323]: ** Message: entering main loop
  2 Jan 17 22:12:01 greeneggs su: (to root) saml on pts/5
  3 Jan 17 22:12:28 greeneggs fprintd[4323]: ** Message: No devices in use, exit
  4 Jan 17 22:12:56 greeneggs gnome-session[1876]: 22:12:56 | Git | personal_repo | Checking for remote changes...
  5 Jan 17 22:12:56 greeneggs gnome-session[1876]: 22:12:56 | Cmd | personal_repo | git rev-parse HEAD
  6 Jan 17 22:12:56 greeneggs gnome-session[1876]: 22:12:56 | Cmd | personal_repo | git ls-remote --heads --exit-code "ssh://sam@sparkleshare.jake      
  6 8us.org/home/sam/SparkleShare/personal_repo.git" master
  7 Jan 17 22:12:58 greeneggs gnome-session[1876]: X11 forwarding request failed on channel 1
  8 Jan 17 22:12:58 greeneggs gnome-session[1876]: 22:12:58 | Git | personal_repo | No remote changes, local+remote: 532213be48cce3b93cb177d409faa      
  8 03b71d0cfa5
  9 Jan 17 22:13:35 greeneggs gnome-session[1876]: 22:13:35 | ListenerTcp | Pinging tcp://notifications.sparkleshare.org:443/
 10 Jan 17 22:13:35 greeneggs gnome-session[1876]: 22:13:35 | ListenerTcp | Received pong from tcp://notifications.sparkleshare.org:443/

NOTE: Take notice of the output. You may or may not like this feature, but it will take long lines and chop them so that they continue on the next line, but still maintain the same corresponding line number. I find this feature invaluable when parsing log files that are wide! You can see the effect of this feature on lines 6 & 8.

  • This is unclear. Where is the filename specified? It should be clarified, noting that the output is numbered from 1 starting with the last 10 lines of the filename, as that is the default behavior of tail. As for the long lines, that behavior is toggled within less using -S. – ILMostro_7 May 23 '16 at 6:35
1

To grep new lines only in the log file as they come with their line number, you could do:

{
  initial_lines=$(wc -l)
  tail -n +1 -f | awk -v NR="$initial_lines" '/pattern/{print NR": "$0}'
} < file.log

(with mawk, you'll want to add the -Winteractive option to prevent its input (!) buffering).

wc -l reads the lines that were already there and counts them (the newline characters which means it still works even if the last line is not full yet), and then we tail -f the rest (starting from where wc stopped reading) and tell awk what is the line number of the first one it sees.

  • choosing l as the variable name made me squint my eyes on $l, thinking it was $1 ^^ (but as I know (and 100% trust) you, I reread and saw the truth). Just for curiosity: to avoid some "race condition" between the wc -l and the tail -f (if the file grows fast, one may discard some lines and thus the NR starts from the wrong number), is it possible to skip $l lines instead? (and what limit is there to tail's -n in posix & in gnu?). Maybe with a temporary intermediate file? – Olivier Dulac Nov 27 '17 at 13:10
  • @OlivierDulac, tail -n +1 (read anything from the start position) addresses the race condition concerns. It will read the lines that were not in the file at the time wc -l terminated, from the exact position wc left it. So NR will have the right position regardless of how many lines have been written in between wc ending and tail starting. It's if you told tail to start from some position relative to the end of the file that you'd have issues. – Stéphane Chazelas Nov 27 '17 at 13:35
  • oh, interresting: indeed, the data accumulate into stdin while nothing reads it (between the end of the wc until the start of the head)... I should have realised that. Thx. Now I see why you "<file". clever, as usual :) – Olivier Dulac Nov 27 '17 at 13:45
  • 1
    @OlivierDulac, about limitations (which don't apply to tail -n +1 here), for regular files, most implementations don't have one as they can start from the end and seek back until they find the nth newline without having to store more than one buf worth of data in memory. For non-seekable input, that's where you can run into limits. POSIX requires implementations to be able to store at least 10 x LINE_MAX bytes (LINE_MAX being at least 2048). GNU tail has no limit other than memory AFAIK – Stéphane Chazelas Nov 27 '17 at 13:54
0

If you wanted to number from the beginning you'd need grep -n to apply to all lines.

 tail -f -n100000000 filename.log | grep -n '' 

If you then only wanted to show the last 10 I'd have thought you could re-tail the file:

 tail -f -n100000000 filename.log | grep -n '' | tail -n10

The first is handy, but shows too much output. I don't know why the second one doesn't work.

  • Tail doesn't have a "show all lines" hence my 100000000 – Martin Cleaver Nov 21 '17 at 17:18
  • 1
    tail -n +1 -f to tail from the beginning. – Stéphane Chazelas Nov 21 '17 at 17:29
  • 1
    The second one doesn't work because the right-most tail can't output anything until it has seen the last line of its input (how would it know which is the 10th last line?) which will never happen as tail -f never stops. – Stéphane Chazelas Nov 21 '17 at 17:31
-1

The command cat -n [filename] | tail will get a quick count and display of the most recent records if that's what you're looking for.

The -f switch makes it persistent until escape - which really doesn't sound applicable in your scenario or is redundant.

wc -l [filename] will get a count of the lines in the target

wc -l [filenameprefix]* will count all lines in all files matching the pattern and even report a summary total at the end.

More complete detail can yield more complete responses.

-1

It's the argument n or --lines (used slightly different ways, see below):

$ tail -f -n 25 /path/to/file.txt

$ tail -f --lines=25 /path/to/file.txt

See also the help:

$ tail --help
  • The OP wants line numbers, not a number of lines. – robbat2 Jan 18 at 23:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.