0

I've got a command that outputs a few different lines constantly (a stream), I'd like a live summary of the number of times each line has occurred (with a few seconds latency being acceptable).

for example if my command outputs the following:

apple
apple
apple
apple
banana
orange
banana

I'd like something like:

4 apple
2 banana
1 orange

and for the output to refresh every few seconds.

How might I achieve this? (rereading an entire log file will take too long, it has to be the output of a live pipe)

2

You could use awk:

... | awk '{seen[$0]++} !(NR % 10) {print "======"; for (i in seen) print seen[i], i}'

This will keep a count of duplicates (seen[$0]++), and print them every ten input lines (!(NR % 10)). For example:

% % while sleep 1; do echo $((RANDOM % 10)); done | awk '{seen[$0]++} !(NR % 10) {print "======"; for (i in seen) print seen[i], i}'
======
1 0
1 1
3 3
1 6
2 7
2 9
======
3 0
3 1
1 2
3 3
1 4
2 5
2 6
3 7
2 9
======
3 0
3 1
2 2
3 3
2 4
5 5
4 6
3 7
2 8
3 9
...
| improve this answer | |
  • This is awesome, thanks so much! – Camden Narzt May 13 at 14:40
2

Could be done with a short perl script like:

#! /usr/bin/perl
system qw(tput sc); # save cursor
$rc = `tput rc; tput ed`; # restore cursor and erase down
sub report {
  print $rc;
  print "$_: $c{$_}\n" for sort {
    ($c{$b} <=> $c{$a}) || ($a cmp $b)
  } keys %c;
  STDOUT->flush;
  alarm 1;
}
$SIG{ALRM} = \&report;
alarm 1;
while (<>) {
  chomp;
  $c{$_}++;
}
report;
| improve this answer | |
  • Thanks for this, I accepted the awk answer because it's just a bit more readable than perl, but your answer is great too. – Camden Narzt May 13 at 14:43
0

watch -n <seconds> "sort <file> | uniq -c"

Should be about what you want. Runs sort | uniq every <seconds>.

| improve this answer | |
  • 1
    Note the (rereading an entire log file will take too long, it has to be the output of a live pipe) in the OP's question. – Stéphane Chazelas May 12 at 11:37
  • hmm, that makes matters more complex. you will need to read the file in chunks and keep a rolling count of entries to be updated. – Pedro May 13 at 13:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.