6

I have a file, let's call it filename.log, in it I have something like this

(2014-11-18 14:09:21,766), , xxxxxx.local, EventSystem, DEBUG FtpsFile delay secs is 5 [pool-3-thread-7] 
(2014-11-18 14:09:21,781), , xxxxxx.local, EventSystem, DEBUG FtpsFile disconnected from ftp server [pool-3-thread-7] 
(2014-11-18 14:09:21,798), , xxxxxx.local, EventSystem, DEBUG FtpsFile FTP File  Process@serverStatus on exit  - 113 [pool-3-thread-7] 
(2014-11-18 14:09:21,798), , xxxxxx.local, EventSystem, DEBUG FtpsFile FTP File  Process@serverStatus on exit  - 114 [pool-3-thread-7] 
(2014-11-18 14:09:21,799), , xxxxxx.local, EventSystem, DEBUG JobQueue $_Runnable Finally of consume() :: [pool-3-thread-7] 

I am trying to find the classes the produce the most frequent DEBUG messages.

In this example you can see FtpsFile and JobQueue are two of the classes producing a message.

I have this

cat filename.log | sed -n -e 's/^.*\(DEBUG \)/\1/p' | sort | uniq -c | sort -rn | head -10

This will produce the class name and show me the most frequent classes as a top 10.

The problem is this does not give me the count of the class FtpsFile as 4. It counts each FtpsFile log file as a different unique entity.

How do I change the command above to basically say grab the first word after DEBUG and ignore the rest for your count?

Ideally I should get 4 FtpsFile 1 JobQueue

2
  • With GNU sed:

    sed 's/.*DEBUG \(\w*\).*/\1/' | uniq -c
          4 FtpsFile
          1 JobQueue
    
  • With grep:

    grep -Po 'DEBUG \K\w+' | uniq -c
          4 FtpsFile
          1 JobQueue
    
  • With awk:

    awk '$6=="DEBUG"{print $7}' | uniq -c
          4 FtpsFile
          1 JobQueue
    

The last one can be done in pure awk, but for a sake of similarity I piped it to uniq.

  • cat filename.log | sed -n -e 's/.*DEBUG \(\w*\).*/\1/p' | sort | uniq -c | sort -rn | head -10 Is what did it. The first sort put all the classes together and then counted the uniques, sorted them and printed the top 10. Without the first sort I was getting the result but it looked like this: 3078 XXXHandler 3075 XXXHandler 3027 XXXXEventHandler 36 YYYYEventHandler 26 LLLEventHandler With the double short all XXXHandler where counted together – wiredniko Nov 18 '14 at 21:15
  • 1
    @wiredniko yes, if your "classes" don't come grouped together as in the example from the question then yes - first sort before uniq is absolutely necessary. – jimmij Nov 18 '14 at 21:26
1

Quick fix - I added the following cut command to single out that field:

[host:~]$ cat logfile | cut -d" " -f7 | sort | uniq -c | sort -rn | head -10
      4 FtpsFile
      1 JobQueue

In my eagerness to K.I.S.S., this doesn't apply to classes with a space in the name.

  • Where is the sed expression part? Is it before the cut -d" " -f7? cat filename.log | sed -n -e 's/^.*\(DEBUG \)/\1/p' | cut -d" " -f7 | sort | uniq -c | sort -rn | head -10 Gives me the number of unique words but it does not look at the class names – wiredniko Nov 18 '14 at 20:48
  • err, instead of "added" I should have said "replaced". – AGHz Nov 18 '14 at 20:54
0

You can use awk(instead of sed) to avoid looking at fields before the one you are interested in, then cut the section you want to see out:

[hunter@apollo: ~]$ cat filename.log | awk -F, '{ print $6 }' | cut -c 1-15 | uniq -c | sort -rn | head -10
      4  DEBUG FtpsFile
      1  DEBUG JobQueue

(Note: you were also sorting twice, which seems unneeded)

EDIT: If you don't know how long the classes will be, you can add an additional awk command (instead of cut):

[hunter@apollo: ~]$ cat filename.log | awk -F, '{ print $6 }' | awk '{ print $1, $2 }' | uniq -c | sort -rn | head -10
      4 DEBUG FtpsFile
      1 DEBUG JobQueue
  • The problem with this is I don't know how long my class name is going to be. Also It won't always be the 6th word – wiredniko Nov 18 '14 at 20:53
  • See my edit above. – Hunter McMillen Nov 18 '14 at 20:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.