Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have large file containing logs similar to one shown below. I would like to find all transactions(TR#) that were affected with the error. I need to extract one occurrence of each TR# ID.

How could I go about it?

    Apr 30 16:51:29.574 application.crit: [6104]:TR#14. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#14. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#14. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#14. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704

Required output:

    Apr 30 16:51:29.574 application.crit: [6104]:TR#14. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704
share|improve this question
    
Did the final line of input get truncated? is the error code meant to be 704 there as well not 70? –  steeldriver May 13 at 10:10
    
@steeldriver. That is typo. It is always 704. –  Prabhu May 13 at 10:12
    
Are the timestamps really all the same? if not, which one do you want to print (if any)? –  steeldriver May 13 at 10:22
    
@steeldriver No. The date is same but not the time stamp. Getting out the first one should do. Thanks! –  Prabhu May 13 at 10:24

5 Answers 5

up vote 3 down vote accepted

This is very simple to do in awk:

$ awk 'c[$5]++==1' file 
Apr 30 16:51:29.574 application.crit: [6104]:TR#14. Transaction send can not be sent. Error Code: 704
Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704

Or, in Perl:

$ perl -ane '$k{$F[4]}++==1 && print' file 
Apr 30 16:51:29.574 application.crit: [6104]:TR#14. Transaction send can not be sent. Error Code: 704
Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704

The above assume that the number before each TR#ID is part of the ID. If the numbers can change but you only need one of them, use this instead:

$ awk -F'[:.]' 'c[$7]++==1' file 

or

$ perl -F'[:.]' -ane '$k{$F[6]}++==1 && print' file 
share|improve this answer

To get and print the first occurrence of each message, try

awk '! m[$5] {m[$5]=$0} END{for (e in m) print m[e]}'

I made the timestamps in your example sequential in order to test it (and also corrected the final truncated error value):

$ awk '! m[$5] {m[$5]=$0} END{for (e in m) print m[e]}' tr2.log
Apr 30 16:51:27.574 application.crit: [6104]:TR#14. Transaction send can not be sent. Error Code: 704
Apr 30 16:51:31.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704

With thanks to @terdon

share|improve this answer
    
Why not just awk 'c[$5]++==1' file ? Am I missing something? –  terdon May 13 at 14:16
    
Doh! thanks - I will simplify my answer accordingly. My brain hurts (the OP kept changing the question...) –  steeldriver May 13 at 14:33
    
OK, I thought you might be anticipating a complication that my answer would miss :) Oh man, I just saw the edit history. Poor you. –  terdon May 13 at 14:34
    
Actually yours is still simpler - why not post it as a separate answer? I always forget about the 1 default print action. –  steeldriver May 13 at 14:37
    
I did post it. It's right here. –  terdon May 13 at 14:38

Here's a perl script that does what you want:

#!/usr/bin/perl

#Read each line
while ($line = <>) {
  # Extract the transaction ID by looking for the text TR followed by digits
  ($trid) = $line =~ /.*(TR#\d+).*/ ;
  # If we've not seen the ID before, print it out
  unless ($trids{$trid}) {
    print $line;
  }
  # Remember the ID so we don't print it out again
  $trids{$trid} = 1;
}

When I call it using your input, this is what I get:

temeraire:ul jenny$ ./extract.pl in.txt 
    Apr 30 16:51:29.574 application.crit: [6104]:TR#14. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704
share|improve this answer

I would do it by piping easy tasks to be done:

grep 'Error Code: 704' test_log | uniq | less

The grep command will only select the lines containing the string "Error Code: 704". The uniq command will only keep one of each element. The sign ">" will use the standard output and send it to the file named "new_file"

Probably there are better ways of doing it which I don't know yet

share|improve this answer
    
The basic idea is sound, but the output doesn't match what the poster asked for. –  Jenny D May 13 at 10:06
    
@3nirque. Thanks for your answer. Actually each log statement as some details like time and date precede 'TR#' I have edited the post with that detail. –  Prabhu May 13 at 10:10
    
I just removed the awk and sed commands. It works nicely in your exemple. In this case if the time stamp is different you might find some lines that contain the same error code –  3nrique0 May 13 at 11:08

Through GNU sed, stolen from this SO answer,

sed '$!N; /^\(.*\)\n\1$/!P; D' file
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.