Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a log file and I'm making a script to do some actions. An action is to print a specific area of the log.

Every block at the log starts with a specific time stamp and inside the block may have other dates etc.

I want to get the block that inside it there is the word exception. Tried with sed but as I know process line by line, also tried with awk and FS \n but again nothing....

A part of the log file,

06:14:27.9 starting web server
06:14:33.3 Initializing Spring framework Logs
Oct 18, 2013 6:14:33 AM org.apache.catalina.startup.Embedded start
INFO: Starting tomcat server
Oct 18, 2013 6:14:34 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.32
Oct 18, 2013 6:14:35 AM org.apache.catalina.startup.ContextConfig DefaultWebConfig
INFO: No default web.xml
Oct 18, 2013 6:14:38 AM org.apache.catalina.session.StandardManager doLoad
SEVERE: IOException while loading persisted sessions: java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: ads.doc.backoffice.StoreInfos
java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: ads.doc.backoffice.StoreInfos
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1354)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
......
......
......
......
    at ads.tools.AppServerMain.main(AppServerMain.java:83)
Caused by: java.io.NotSerializableException: ads.doc.backoffice.StoreInfos
INFO: Jk running ID=0 time=0/105  config=null
06:14:48.6 Starting exporter server
06:14:48.6 starting cron server

another part of a log

03:19:13.4 Begin summary update for ads.doc.inventory.InventoryItemSummary
03:19:33.9 CronServer:: DailyJob ads.tools.UpdateSummaries@17c5d6cf failed with exception ads.util.AppError: Cannot create UnitName from keys: Each
ads.util.AppError: Cannot create UnitName from keys: Each
at ads.db.DBObjectDefault.createFromKeys(DBObjectDefault.java:42)
at ads.db.DBTable.createFromKeys(DBTable.java:227)
at ads.db.DBValue.getValue(DBValue.java:621)
at ads.dbmanager.DBObjectsManager.initObjects(DBObjectsManager.java:400)
at ads.dbmanager.DBObjectsManager.reload(DBObjectsManager.java:447)
at ads.dbmanager.DBObjectsManager.loadFromStore(DBObjectsManager.java:497)
at ads.doc.inventory.InventoryItemSummary.refreshSince(InventoryItemSummary.java:173)
at ads.db.DBSummaryTable.refreshAll(DBSummaryTable.java:67)
at ads.tools.CronServer$DailyThread.run(CronServer.java:271)
[SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.net.ConnectException: Connection refused; targetException=java.lang.IllegalArgumentException: Error opening socket: java.net.ConnectException: Connection refused]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(SOAPHTTPConnection.java:354)
at org.apache.soap.rpc.Call.invoke(Call.java:248)
at ads.support.SupportCall.call(SupportCall.java:56)
at ads.tools.SupportThread.run(SupportThread.java:101)
03:46:42.5 Periodic support request failed: ads.support.SupportException: Error opening socket: java.net.ConnectException: Connection refused
06:31:36.1 Upload failed: java.io.FileNotFoundException: c:/tmp/cygwin1.dll (No such file or directory)
08:01:08.0 connect from /172.22.3.28

I want to print from the first till the second last (06:14:33.3 till 06:14:48.6). And inside the log there are multiple blocks as this.

Tried this: awk '/^[0-9][0-9]\:[0-9][0-9]\:[0-9][0-9]\.[0-9].*[e|E]xception.*[0-9][0-9]\:[0-9][0-9]\:[0-9][0-9]\.[0-9]/ {print}' FS="\n" RS="" log.txt

and also this:

sed '/^[0-9][0-9]\:[0-9][0-9]\:[0-9][0-9]\.[0-9].*[e|E]xception.*/,/^[0-9][0-9]\:[0-9][0-9]\:[0-9][0-9]\.[0-9]/!d' log.txt

but I can't get the result I want.

share|improve this question
1  
You don't have to deal with the whole thing in one regexp. Look for the first line, then when that's found, branch to examine the subsequent lines. –  goldilocks Dec 6 '13 at 14:59
    
Every line is possible to be the start of the block. Can you give me an example? –  C Ts Dec 6 '13 at 15:02
    
Could you clarify a bit more. How is a block defined? Is it the text between two timestamps? –  Ketan Dec 6 '13 at 15:49
    
Added some more log, when there is no error the block/timestamp is a single line but when there are more information the timestamp it's multiline. –  C Ts Dec 6 '13 at 16:19
    
The first timestamp (just time) is the actual logger's stamp. The second timestamp (starting with a date) is part of the message it was passed. –  goldilocks Dec 6 '13 at 17:29

2 Answers 2

Since no awk wizards have chimed in yet, I'll demonstrate what I meant about using more than one search capture in perl (since I'm no awk wizard and much prefer perl to bash -- but you could translate this into bash). Sans comments and the test data, it's ~15 lines:

#!/usr/bin/perl
use strict;
use warnings FATAL => qw(all);

my $timestamp = qr/^\d+:\d+:\d+/;
# Alternate version, see discussion below:
#my $timestamp = qr/^([a-zA-Z]+\s+\d+,\s+\d+\s+)?\d+:\d+:\d+/;

my $block = "";
my $exception = 0;
while (<DATA>) {
# Note that perl considers an empty string false.
    if ($block) {
    # $_ is the current input line in perl.
        if ($_ =~ $timestamp) {
            print "\n\n***EXCEPTION***\n$block" if $exception;
            $block = "";
            $exception = 0;
        }
    # This isn't actually a second regexp, it's a substring search.
    # But you could use a regexp.
        $exception = 1 if (index($_,"Exception") >= 0);
        $block .= $_;
        next;
    }
    $block = $_ if ($_ =~ $timestamp);
}

# Everything after this is read in via <DATA> for testing.
__DATA__
06:14:27.9 starting web server
06:14:33.3 Initializing Spring framework Logs
Oct 18, 2013 6:14:33 AM org.apache.catalina.startup.Embedded start
INFO: Starting tomcat server
Oct 18, 2013 6:14:34 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.32
Oct 18, 2013 6:14:35 AM org.apache.catalina.startup.ContextConfig DefaultWebConfig
INFO: No default web.xml
Oct 18, 2013 6:14:38 AM org.apache.catalina.session.StandardManager doLoad
SEVERE: IOException while loading persisted sessions: java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: ads.doc.backoffice.StoreInfos
java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: ads.doc.backoffice.StoreInfos
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1354)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at ads.tools.AppServerMain.main(AppServerMain.java:83)
Caused by: java.io.NotSerializableException: ads.doc.backoffice.StoreInfos
INFO: Jk running ID=0 time=0/105  config=null
06:14:48.6 Starting exporter server
06:14:48.6 starting cron server
03:19:13.4 Begin summary update for ads.doc.inventory.InventoryItemSummary
03:19:33.9 CronServer:: DailyJob ads.tools.UpdateSummaries@17c5d6cf failed with exception ads.util.AppError: Cannot create UnitName from keys: Each
ads.util.AppError: Cannot create UnitName from keys: Each
at ads.db.DBObjectDefault.createFromKeys(DBObjectDefault.java:42)
at ads.db.DBTable.createFromKeys(DBTable.java:227)
at ads.db.DBValue.getValue(DBValue.java:621)
at ads.dbmanager.DBObjectsManager.initObjects(DBObjectsManager.java:400)
at ads.dbmanager.DBObjectsManager.reload(DBObjectsManager.java:447)
at ads.dbmanager.DBObjectsManager.loadFromStore(DBObjectsManager.java:497)
at ads.doc.inventory.InventoryItemSummary.refreshSince(InventoryItemSummary.java:173)
at ads.db.DBSummaryTable.refreshAll(DBSummaryTable.java:67)
at ads.tools.CronServer$DailyThread.run(CronServer.java:271)
[SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.net.ConnectException: Connection refused; targetException=java.lang.IllegalArgumentException: Error opening socket: java.net.ConnectException: Connection refused]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(SOAPHTTPConnection.java:354)
at org.apache.soap.rpc.Call.invoke(Call.java:248)
at ads.support.SupportCall.call(SupportCall.java:56)
at ads.tools.SupportThread.run(SupportThread.java:101)
03:46:42.5 Periodic support request failed: ads.support.SupportException: Error opening socket: java.net.ConnectException: Connection refused
06:31:36.1 Upload failed: java.io.FileNotFoundException: c:/tmp/cygwin1.dll (No such file or directory)
08:01:08.0 connect from /172.22.3.28        

The DATA stuff is all your log examples put together. The point of the two searches is to first identify the beginning of a block and record it, and then to determine if the block contains an Exception.

As is, the script finds the following four blocks:

***EXCEPTION***
06:14:33.3 Initializing Spring framework Logs
[...]
INFO: Jk running ID=0 time=0/105  config=null

***EXCEPTION***
03:19:33.9 CronServer:: DailyJob ads.tools.UpdateSummaries@17c5d6cf failed with exception ads.util.AppError: Cannot create UnitName from keys: Each
[...]
at ads.tools.SupportThread.run(SupportThread.java:101)

***EXCEPTION***
03:46:42.5 Periodic support request failed: ads.support.SupportException: Error opening socket: java.net.ConnectException: Connection refused

***EXCEPTION***
06:31:36.1 Upload failed: java.io.FileNotFoundException: c:/tmp/cygwin1.dll (No such file or directory)

That's based on interpreting each block as beginning with a 06:31:36 style timestamp; the stuff with Oct 18, 2013 6:14:33 is obviously JRE log stuff passed to the server log, and so part of one entry. If you want to do this differently, by separating the individual JRE exceptions, use the "Alternate version" of $timestamp, which regards both styles as starting an entry.

share|improve this answer
    
If you want to try the perl script with your actual logs, replace while (<DATA>) with while (<STDIN>) and pipe the log in (whatever.pl < log.txt). –  goldilocks Dec 6 '13 at 20:41
    
Sorry my comment got deleted. I tried with but no success. Need to spend a bit more time. –  Ketan Dec 6 '13 at 20:50
    
Although it's in perl and I don't have any idea I'll try to convert it to bash (hope to succeed) because it works perfectly!!! :D As DATA is the $*? "block" is necessary for bash? –  C Ts Dec 9 '13 at 15:30
    
I'm not sure what you mean. The DATA section is just a perl feature I used for testing, so it's not necessary no -- the <STDIN> version will work without it. As for $block, the idea is to accumulate the lines in a string; if the block turns out to be an Exception, you then print it, if not, throw it away and start accumulating again. I guess in bash if you were using while read line, $block .= $_ (perl) would be block="$block$line". Note the use of the sigil ($) is slightly different in perl vs. bash. –  goldilocks Dec 9 '13 at 15:45
    
I know it's been a while, but can you help me with this? #!/bin/bash timestamp=^[0-9][0-9]\:[0-9][0-9]\:[0-9][0-9]\.[0-9] block="" exception=false except=.*Exception.* while read line do if [[ -z $block ]] ; then if [[ $line =~ $timestamp ]] then if [[ $exception=false ]] then echo "\n\n***EXCEPTION***\n$block" fi block="" exception=false fi if [[ $line =~ $except ]] then exception=true fi block=$block$line fi if [[ $line =~ $timestamp ]] then block=$line fi done –  C Ts Dec 23 '13 at 17:13

Alright, this is a close one:

awk -F "[0-9][0-9]:[0-9][0-9]:[0-9][0-9].[0-9]" '{for (i=1; i<= NF; i++) if ( $i ~ /[e|E]xception/ ) print $i} RS=""' t.txt

I use the timestamp regex as FS and search for the Exception in the fields. Not perfect but close.

After cating both pieces of logs OP provided, the output I got is as follows:

Initializing Spring framework Logs<br>
Oct 18, 2013 6:14:33 AM org.apache.catalina.startup.Embedded start<br>
INFO: Starting tomcat server<br>
Oct 18, 2013 6:14:34 AM org.apache.catalina.core.StandardEngine start<br>
INFO: Starting Servlet Engine: Apache Tomcat/6.0.32<br>
Oct 18, 2013 6:14:35 AM org.apache.catalina.startup.ContextConfig <br>efaultWebConfig
INFO: No default web.xml<br>
Oct 18, 2013 6:14:38 AM org.apache.catalina.session.StandardManager doLoad<br>
SEVERE: IOException while loading persisted sessions: <br>java.io.WriteAbortedException: writing aborted; <br>java.io.NotSerializableException: ads.doc.backoffice.StoreInfos<br>
java.io.WriteAbortedException: writing aborted; <br>java.io.NotSerializableException: ads.doc.backoffice.StoreInfos
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1354)<br>
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
...... <br>
...... <br>
...... <br>
...... <br>
    at ads.tools.AppServerMain.main(AppServerMain.java:83)<br>
Caused by: java.io.NotSerializableException: ads.doc.backoffice.StoreInfos<br>
INFO: Jk running ID=0 time=0/105  config=null<br>

 CronServer:: DailyJob ads.tools.UpdateSummaries@17c5d6cf failed with exception ads.util.AppError: Cannot create UnitName from keys: Each
ads.util.AppError: Cannot create UnitName from keys: Each
at ads.db.DBObjectDefault.createFromKeys(DBObjectDefault.java:42)
at ads.db.DBTable.createFromKeys(DBTable.java:227)
at ads.db.DBValue.getValue(DBValue.java:621)
at ads.dbmanager.DBObjectsManager.initObjects(DBObjectsManager.java:400)
at ads.dbmanager.DBObjectsManager.reload(DBObjectsManager.java:447)
at ads.dbmanager.DBObjectsManager.loadFromStore(DBObjectsManager.java:497)
at ads.doc.inventory.InventoryItemSummary.refreshSince(InventoryItemSummary.java:173)
at ads.db.DBSummaryTable.refreshAll(DBSummaryTable.java:67)
at ads.tools.CronServer$DailyThread.run(CronServer.java:271)
[SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.net.ConnectException: Connection refused; targetException=java.lang.IllegalArgumentException: Error opening socket: java.net.ConnectException: Connection refused]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(SOAPHTTPConnection.java:354)
at org.apache.soap.rpc.Call.invoke(Call.java:248)
at ads.support.SupportCall.call(SupportCall.java:56)
at ads.tools.SupportThread.run(SupportThread.java:101)

 Periodic support request failed: ads.support.SupportException: Error opening socket: java.net.ConnectException: Connection refused

 Upload failed: java.io.FileNotFoundException: c:/tmp/cygwin1.dll (No such file or directory)
share|improve this answer
    
Worked, but once the FS is the regex I'm missing the timestamp, is there any way to also print the FS? –  C Ts Dec 9 '13 at 15:04

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.