1

I have a program that writes on the standard output a list of string values, one per line, and I would like to display in real time the list of distinct values along with the number of occurences for each one.

For example, from this output:

apple
carrot
pear
carrot
apple

I need a command which generates this output (ideally updated in real time):

apple: 2
carrot: 2
pear: 1

Any idea?

  • 1
    Of course, there's the usual sort | uniq -c which will do that in non-real-time. – Celada Jan 29 '15 at 2:59
  • Would something like yourprog | awk '{print $0": "++a[$0]}' suffice? – steeldriver Jan 29 '15 at 3:33
  • @Celada you could probably do something like that if the program's output were directed to a file and you just did a watch "cat file.txt | sort | uniq -c" – Bratchley Jan 29 '15 at 4:00
  • @Bratchley sure, but that's O(m·n log n) where m is the number of times you recalculate the result, whereas O(n log n) should be achievable. Oh, too much computer science for a simple shell script, eh? Yeah, never mind :-) – Celada Jan 29 '15 at 7:58
3

To expand on what @Bratchley said in the comments, if you have your program's output printing to a file, then you can run then watch command in the terminal to get near-real-time view of the output by including the -n flag like so:

watch -n 0.1 "cat yourprograms.log | sort | uniq -c | sort -rn"

Note: The ' -n ' flag sets the refresh interval. The minimum time ' watch ' can take is 0.1 second (or 1/10 a second) and nothing smaller.

Example Output:

Every 0.1s: cat yourprograms.log | sort | uniq -c | sort -rn

      6 things
      4 mine
      3 dot
      1 below

Including | sort -rn allows for a better sort view. sort -rn sorts the output of uniq -c in reverse numeric order.

If you only care about say the top 10, you can include the head command like so:

watch -n 0.1 "cat yourprograms.log | sort | uniq -c | sort -rn | head"
1

Here's a small Python 2 program that does what you want. The words are listed in chronological order of first appearance, i.e., each new word is added to the bottom of the list, but it would be easy to sort words alphabetically, or in order of number of occurrences.

The output could be a little neater, eg if you know the maximum string length then the occurence numbers could be aligned in a column.

wordcount.py

#! /usr/bin/env python

''' Real-time word counting

    Written by PM 2Ring 2015.01.29
    From http://unix.stackexchange.com/q/181722/88378
'''

CSI = '\x1b['
clear = CSI + '2J' + CSI + 'H'

def main():
    words = []
    wordcount = {}
    while True:
        try:
            word = raw_input()
        except (KeyboardInterrupt, EOFError):
            print 
            break

        if word not in wordcount:
            words.append(word)
            wordcount[word] = 1
        else:
            wordcount[word] += 1

        print clear
        for word in words:
            print '%s: %d' % (word, wordcount[word])

if __name__ == '__main__':
    main()

To pipe the output of yourprog into this program you can do this:

yourprog | python wordcount.py

assuming that wordcount.py is in the current directory.

Alternatively, give wordcount.py execute permission (eg chmod a+x wordcount.py) and put it into a directory in your command PATH list (i.e. a directory listed when you do echo "$PATH"). Then you can do

yourprog | wordcount.py

and you can run it anywhere.

If you only have Python 3 and not Python 2, the program will require some minor changes to make it run.

FWIW, here's another Python script I wrote to test the above code. It prints a random word, one word per line, every 0.5 seconds.

randwords.py

#! /usr/bin/env python

import random, time

wordlist = [
    'apple',
    'carrot',
    'pear',
    'orange',
    'banana',
    'cabbage',
    'potato'
]

def main():
    #Time delay between words, in seconds
    delay = 0.5
    while True:
        try:
            print random.choice(wordlist)
            time.sleep(delay)
        except KeyboardInterrupt:
            print 
            break

if __name__ == '__main__':
    main()

You can run it, piping its output into wordcount.py like this:

python -u randwords.py | python wordcount.py

The -u flag tells the Python interpreter to use unbuffered I/O.

1

Simple awk thing. Using some ANSI sequences for movement and clearing...

#!/usr/bin/awk -f

BEGIN {
    print "Stats:\n---------------------------------"
}
function clear() {
    for (k in ar)
        printf "\r\033[K\033[1A"
}
function stats() {
    for (k in ar)
        printf "%-10s: %d\n", k, ar[k]
}
/./{
    clear()
    if (!ar[$0])
        ar[$0]=1
    else
        ++ar[$0]
    stats()
}

Sample output generator:

#!/bin/bash

declare -a fr=(
    apple
    carrot
    pear
)

range=${#fr[@]}

while ((1)); do
    ((x = RANDOM % range))
    printf "%s\n" "${fr[$x]}"
    sleep .5
done

Run as:

$ ./fruit_script | ./awk_script

Extended with colors, hidden cursor etc.:

#!/bin/bash

quit()
{
    printf "\r\033[K\033[?25h"
}

trap quit SIGINT

awk '
    BEGIN {
        width=3
        printf "\033[?25l"
        print "Stats: (Ctrl-c to quit)\n---------------------------------"
    }
    function clear() {
        for (k in ar)
            printf "\r\033[K\033[1A"
    }
    function stats() {
        for (k in ar)
            printf "\033[0;34m%-*s\033[0m: \033[1;31m%d\033[0m\n", width, k, ar[k]
    }
    /./{
        if (length($0) + 1 > width)
            width=length($0) + 1
        clear()
        if (!ar[$0])
            ar[$0]=1
        else
            ++ar[$0]

        stats()
    }
'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.