1

I have large file containing logs similar to one shown below. I would like to find all transactions(TR#) that were affected with the error. I need to extract one occurrence of each TR# ID.

How could I go about it?

    Apr 30 16:51:29.574 application.crit: [6104]:TR#14. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#14. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#14. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#14. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704

Required output:

    Apr 30 16:51:29.574 application.crit: [6104]:TR#14. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704
Improve this question
4
  • Did the final line of input get truncated? is the error code meant to be 704 there as well not 70?
    – steeldriver
    Commented May 13, 2014 at 10:10
  • @steeldriver. That is typo. It is always 704.
    – Prabhu
    Commented May 13, 2014 at 10:12
  • Are the timestamps really all the same? if not, which one do you want to print (if any)?
    – steeldriver
    Commented May 13, 2014 at 10:22
  • @steeldriver No. The date is same but not the time stamp. Getting out the first one should do. Thanks!
    – Prabhu
    Commented May 13, 2014 at 10:24

5 Answers 5

Reset to default
3

This is very simple to do in awk:

$ awk 'c[$5]++==1' file 
Apr 30 16:51:29.574 application.crit: [6104]:TR#14. Transaction send can not be sent. Error Code: 704
Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704

Or, in Perl:

$ perl -ane '$k{$F[4]}++==1 && print' file 
Apr 30 16:51:29.574 application.crit: [6104]:TR#14. Transaction send can not be sent. Error Code: 704
Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704

The above assume that the number before each TR#ID is part of the ID. If the numbers can change but you only need one of them, use this instead:

$ awk -F'[:.]' 'c[$7]++==1' file

or

$ perl -F'[:.]' -ane '$k{$F[6]}++==1 && print' file
Improve this answer
3

To get and print the first occurrence of each message, try

awk '! m[$5] {m[$5]=$0} END{for (e in m) print m[e]}'

I made the timestamps in your example sequential in order to test it (and also corrected the final truncated error value):

$ awk '! m[$5] {m[$5]=$0} END{for (e in m) print m[e]}' tr2.log
Apr 30 16:51:27.574 application.crit: [6104]:TR#14. Transaction send can not be sent. Error Code: 704
Apr 30 16:51:31.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704

With thanks to @terdon

Improve this answer
8
  • Why not just awk 'c[$5]++==1' file ? Am I missing something?
    – terdon ♦
    Commented May 13, 2014 at 14:16
  • Doh! thanks - I will simplify my answer accordingly. My brain hurts (the OP kept changing the question...)
    – steeldriver
    Commented May 13, 2014 at 14:33
  • OK, I thought you might be anticipating a complication that my answer would miss :) Oh man, I just saw the edit history. Poor you.
    – terdon ♦
    Commented May 13, 2014 at 14:34
  • Actually yours is still simpler - why not post it as a separate answer? I always forget about the 1 default print action.
    – steeldriver
    Commented May 13, 2014 at 14:37
  • I did post it. It's right here.
    – terdon ♦
    Commented May 13, 2014 at 14:38
2

Here's a perl script that does what you want:

#!/usr/bin/perl

#Read each line
while ($line = <>) {
  # Extract the transaction ID by looking for the text TR followed by digits
  ($trid) = $line =~ /.*(TR#\d+).*/ ;
  # If we've not seen the ID before, print it out
  unless ($trids{$trid}) {
    print $line;
  }
  # Remember the ID so we don't print it out again
  $trids{$trid} = 1;
}

When I call it using your input, this is what I get:

temeraire:ul jenny$ ./extract.pl in.txt 
    Apr 30 16:51:29.574 application.crit: [6104]:TR#14. Transaction send can not be sent. Error Code: 704
    Apr 30 16:51:29.574 application.crit: [6104]:TR#238. Transaction send can not be sent. Error Code: 704
Improve this answer
2

I would do it by piping easy tasks to be done:

grep 'Error Code: 704' test_log | uniq | less

The grep command will only select the lines containing the string "Error Code: 704". The uniq command will only keep one of each element. The sign ">" will use the standard output and send it to the file named "new_file"

Probably there are better ways of doing it which I don't know yet

Improve this answer
3
  • The basic idea is sound, but the output doesn't match what the poster asked for.
    – Jenny D
    Commented May 13, 2014 at 10:06
  • @3nirque. Thanks for your answer. Actually each log statement as some details like time and date precede 'TR#' I have edited the post with that detail.
    – Prabhu
    Commented May 13, 2014 at 10:10
  • I just removed the awk and sed commands. It works nicely in your exemple. In this case if the time stamp is different you might find some lines that contain the same error code
    – 3nrique0
    Commented May 13, 2014 at 11:08
1

Through GNU sed, stolen from this SO answer,

sed '$!N; /^\(.*\)\n\1$/!P; D' file
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .