I have a log file I am trying to pull data from. It is formatted like so, and lists when certain modules are checked in and out:

19:50:26 (license_manager) OUT: "certain_module" doej@server.domain
19:50:28 (license_manager) IN: "rarely_used_module" doej@server.domain
19:50:28 (license_manager) IN: "certain_module" smithj@server.domain
19:50:28 (license_manager) IN: "different_module" doej@server.domain
19:50:38 (license_manager) OUT: "certain_module" smithj@server.domain
19:50:38 (license_manager) OUT: "different_module" userg@server.domain

So far I have the following. I am specifically interested in when the "rarely_used_module" is getting checked out and by whom:

cd /path/to/script && cat logfile.txt | grep -c "rarely_used_module" 

That just gives me the count, though, and nothing else. I would like to make this more sophisticated for two reasons: to learn more about shell scripting and to implement a greater amount of bash concepts into this script, secondly, I would like it to be able to tell me what time the license was checked out and what user checked it out.

The absolute ideal situation would be to count the number of times this license was checked out and format it into some sort of table for reference. Is this possible?

Update #1

Desired output, something like the following. I'm going to pose a theoretical example below in which the rarely_used_module has been checked out 4 times by 2 unique users, 2 separate times:

Number of license checkouts for rarely_used_module: 4 
User : doej@server.domain (2)
User : smitht@server.domain (2)

Essentially I want the number of times the module has been checked out, total, and I want the names of the users that have checked this license out. I know I can grab the OUT: line for "rarely_used_module" but I'm not sure how to work that in.

  • 1
    "I would like it to be able to tell me what time the license was checked out and what user checked it out." You can get that from: grep "rarely_used_module" logfile.txt. If that is not what you are looking for, you need to explain your goals in more detail. Just asking for "some sort of table" is a bit vague.
    – John1024
    May 27 '15 at 21:50
  • It would appear that you have accidentally created two accounts.  You should use the contact form and select “I need to merge user profiles” to have your accounts merged.  In order to merge them, you will need to provide links to the two accounts.  For your information these are unix.stackexchange.com/users/117058/user117058 and unix.stackexchange.com/users/117059/linux-noob.  You’ll then be able to edit your question (without going through peer review), and comment on and accept answers to this question. May 27 '15 at 22:35

For your updated question:

awk '
/"rarely_used_module"/ && /OUT:/ { nc[$NF]++ ; c++ }
    printf "Number of license checkouts for rarely_used_module: %d\n", c
    for (i in nc) printf "User: %s (%d)\n", i, nc[i]
' logfile.txt

creates this output:

Number of license checkouts for rarely_used_module: 4
User: doej@server.domain (2)
User: smithj@server.domain (2)

I keep the original answer below, to show ways how to extend the code in case you have increased requirements.

Here is a sample how one could approach to such tasks using awk:

awk '
BEGIN { SUBSEP = ", " ; OFS = ": " }
{ m[$(NF-1)]++ }
{ n[$(NF-1)] = n[$(NF-1)] " " $NF }
{ nc[$(NF-1),$NF]++ }
    print "\n=== count modules:"
    for (i in m) print i, m[i]
    print "\n=== collect names using modules:"
    for (i in n) print i, n[i]
    print "\n=== count names using modules:"
    for (i in nc) print i, nc[i]
' logfile.txt


  • { m[$(NF-1)]++ } - increments counter for second last field (modules) in the input data
  • { n[$(NF-1)] = n[$(NF-1)] " " $NF } - concatenates the last field (names) for each key (modules)
  • { nc[$(NF-1),$NF]++ } - increments counter for a key-tuple of (name, module)

With your sample data it would produce this output:

=== count modules:
"rarely_used_module": 1
"different_module": 2
"certain_module": 3

=== collect names using modules:
"rarely_used_module":  doej@server.domain
"different_module":  doej@server.domain userg@server.domain
"certain_module":  doej@server.domain smithj@server.domain smithj@server.domain

=== count names using modules:
"different_module", userg@server.domain: 1
"different_module", doej@server.domain: 1
"certain_module", smithj@server.domain: 2
"rarely_used_module", doej@server.domain: 1
"certain_module", doej@server.domain: 1

When I need something more complex than change or match each line, I use Python, because it is general-purpose language. It may be more verbose than awk (BTW, there is a pawk, Python awk), but it also gives you well-documented and easily extensible code.

Here are Python 2 script for your task:

from collections import defaultdict

FILE = 'module.txt'

# Global table of usages is 
# dict [ module_name ] -> dict [ user_name ] -> count
usage = defaultdict(lambda : defaultdict(int))

# Read, parse data and add usage count where needed
with open(FILE) as f:
    for line in f:
        # Split using spaces and pick last 2 fields, 
        # strip unncessary characters
        fields = line.split()     
        user = fields[-1].rstrip()
        module_name = fields[-2].strip('"')

        usage[module_name][user] += 1

# Now print pretty results
for module_name, module_usage in usage.items():
    print '====> ', module_name
    for user, count in module_usage.items():
        print '\t', user, count

It will print following data for your sample:

====>  different_module                                                                                                                                                        
        doej@server.domain 1
        userg@server.domain 1
====>  rarely_used_module
        doej@server.domain 1
====>  certain_module
        smithj@server.domain 2
        doej@server.domain 1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.