bash script, printing multiple lines that matching to a specific patern

Question

I have a log file and I'm making a script to do some actions. An action is to print a specific area of the log.

Every block at the log starts with a specific time stamp and inside the block may have other dates etc.

I want to get the block that inside it there is the word exception. Tried with sed but as I know process line by line, also tried with awk and FS \n but again nothing....

A part of the log file,

06:14:27.9 starting web server
06:14:33.3 Initializing Spring framework Logs
Oct 18, 2013 6:14:33 AM org.apache.catalina.startup.Embedded start
INFO: Starting tomcat server
Oct 18, 2013 6:14:34 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.32
Oct 18, 2013 6:14:35 AM org.apache.catalina.startup.ContextConfig DefaultWebConfig
INFO: No default web.xml
Oct 18, 2013 6:14:38 AM org.apache.catalina.session.StandardManager doLoad
SEVERE: IOException while loading persisted sessions: java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: ads.doc.backoffice.StoreInfos
java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: ads.doc.backoffice.StoreInfos
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1354)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
......
......
......
......
    at ads.tools.AppServerMain.main(AppServerMain.java:83)
Caused by: java.io.NotSerializableException: ads.doc.backoffice.StoreInfos
INFO: Jk running ID=0 time=0/105  config=null
06:14:48.6 Starting exporter server
06:14:48.6 starting cron server

another part of a log

03:19:13.4 Begin summary update for ads.doc.inventory.InventoryItemSummary
03:19:33.9 CronServer:: DailyJob ads.tools.UpdateSummaries@17c5d6cf failed with exception ads.util.AppError: Cannot create UnitName from keys: Each
ads.util.AppError: Cannot create UnitName from keys: Each
at ads.db.DBObjectDefault.createFromKeys(DBObjectDefault.java:42)
at ads.db.DBTable.createFromKeys(DBTable.java:227)
at ads.db.DBValue.getValue(DBValue.java:621)
at ads.dbmanager.DBObjectsManager.initObjects(DBObjectsManager.java:400)
at ads.dbmanager.DBObjectsManager.reload(DBObjectsManager.java:447)
at ads.dbmanager.DBObjectsManager.loadFromStore(DBObjectsManager.java:497)
at ads.doc.inventory.InventoryItemSummary.refreshSince(InventoryItemSummary.java:173)
at ads.db.DBSummaryTable.refreshAll(DBSummaryTable.java:67)
at ads.tools.CronServer$DailyThread.run(CronServer.java:271)
[SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.net.ConnectException: Connection refused; targetException=java.lang.IllegalArgumentException: Error opening socket: java.net.ConnectException: Connection refused]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(SOAPHTTPConnection.java:354)
at org.apache.soap.rpc.Call.invoke(Call.java:248)
at ads.support.SupportCall.call(SupportCall.java:56)
at ads.tools.SupportThread.run(SupportThread.java:101)
03:46:42.5 Periodic support request failed: ads.support.SupportException: Error opening socket: java.net.ConnectException: Connection refused
06:31:36.1 Upload failed: java.io.FileNotFoundException: c:/tmp/cygwin1.dll (No such file or directory)
08:01:08.0 connect from /172.22.3.28

I want to print from the first till the second last (06:14:33.3 till 06:14:48.6). And inside the log there are multiple blocks as this.

Tried this: awk '/^[0-9][0-9]\:[0-9][0-9]\:[0-9][0-9]\.[0-9].*[e|E]xception.*[0-9][0-9]\:[0-9][0-9]\:[0-9][0-9]\.[0-9]/ {print}' FS="\n" RS="" log.txt

and also this:

sed '/^[0-9][0-9]\:[0-9][0-9]\:[0-9][0-9]\.[0-9].*[e|E]xception.*/,/^[0-9][0-9]\:[0-9][0-9]\:[0-9][0-9]\.[0-9]/!d' log.txt

but I can't get the result I want.

You don't have to deal with the whole thing in one regexp. Look for the first line, then when that's found, branch to examine the subsequent lines. — goldilocks, Commented Dec 6, 2013 at 14:59
Every line is possible to be the start of the block. Can you give me an example? — C Ts, Commented Dec 6, 2013 at 15:02
Could you clarify a bit more. How is a block defined? Is it the text between two timestamps? — Ketan, Commented Dec 6, 2013 at 15:49
Added some more log, when there is no error the block/timestamp is a single line but when there are more information the timestamp it's multiline. — C Ts, Commented Dec 6, 2013 at 16:19
The first timestamp (just time) is the actual logger's stamp. The second timestamp (starting with a date) is part of the message it was passed. — goldilocks, Commented Dec 6, 2013 at 17:29

goldilocks · Accepted Answer · 2013-12-06 17:28:26Z

Since no awk wizards have chimed in yet, I'll demonstrate what I meant about using more than one search capture in perl (since I'm no awk wizard and much prefer perl to bash -- but you could translate this into bash). Sans comments and the test data, it's ~15 lines:

#!/usr/bin/perl
use strict;
use warnings FATAL => qw(all);

my $timestamp = qr/^\d+:\d+:\d+/;
# Alternate version, see discussion below:
#my $timestamp = qr/^([a-zA-Z]+\s+\d+,\s+\d+\s+)?\d+:\d+:\d+/;

my $block = "";
my $exception = 0;
while (<DATA>) {
# Note that perl considers an empty string false.
    if ($block) {
    # $_ is the current input line in perl.
        if ($_ =~ $timestamp) {
            print "\n\n***EXCEPTION***\n$block" if $exception;
            $block = "";
            $exception = 0;
        }
    # This isn't actually a second regexp, it's a substring search.
    # But you could use a regexp.
        $exception = 1 if (index($_,"Exception") >= 0);
        $block .= $_;
        next;
    }
    $block = $_ if ($_ =~ $timestamp);
}

# Everything after this is read in via <DATA> for testing.
__DATA__
06:14:27.9 starting web server
06:14:33.3 Initializing Spring framework Logs
Oct 18, 2013 6:14:33 AM org.apache.catalina.startup.Embedded start
INFO: Starting tomcat server
Oct 18, 2013 6:14:34 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.32
Oct 18, 2013 6:14:35 AM org.apache.catalina.startup.ContextConfig DefaultWebConfig
INFO: No default web.xml
Oct 18, 2013 6:14:38 AM org.apache.catalina.session.StandardManager doLoad
SEVERE: IOException while loading persisted sessions: java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: ads.doc.backoffice.StoreInfos
java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: ads.doc.backoffice.StoreInfos
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1354)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
    at ads.tools.AppServerMain.main(AppServerMain.java:83)
Caused by: java.io.NotSerializableException: ads.doc.backoffice.StoreInfos
INFO: Jk running ID=0 time=0/105  config=null
06:14:48.6 Starting exporter server
06:14:48.6 starting cron server
03:19:13.4 Begin summary update for ads.doc.inventory.InventoryItemSummary
03:19:33.9 CronServer:: DailyJob ads.tools.UpdateSummaries@17c5d6cf failed with exception ads.util.AppError: Cannot create UnitName from keys: Each
ads.util.AppError: Cannot create UnitName from keys: Each
at ads.db.DBObjectDefault.createFromKeys(DBObjectDefault.java:42)
at ads.db.DBTable.createFromKeys(DBTable.java:227)
at ads.db.DBValue.getValue(DBValue.java:621)
at ads.dbmanager.DBObjectsManager.initObjects(DBObjectsManager.java:400)
at ads.dbmanager.DBObjectsManager.reload(DBObjectsManager.java:447)
at ads.dbmanager.DBObjectsManager.loadFromStore(DBObjectsManager.java:497)
at ads.doc.inventory.InventoryItemSummary.refreshSince(InventoryItemSummary.java:173)
at ads.db.DBSummaryTable.refreshAll(DBSummaryTable.java:67)
at ads.tools.CronServer$DailyThread.run(CronServer.java:271)
[SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.net.ConnectException: Connection refused; targetException=java.lang.IllegalArgumentException: Error opening socket: java.net.ConnectException: Connection refused]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(SOAPHTTPConnection.java:354)
at org.apache.soap.rpc.Call.invoke(Call.java:248)
at ads.support.SupportCall.call(SupportCall.java:56)
at ads.tools.SupportThread.run(SupportThread.java:101)
03:46:42.5 Periodic support request failed: ads.support.SupportException: Error opening socket: java.net.ConnectException: Connection refused
06:31:36.1 Upload failed: java.io.FileNotFoundException: c:/tmp/cygwin1.dll (No such file or directory)
08:01:08.0 connect from /172.22.3.28

The DATA stuff is all your log examples put together. The point of the two searches is to first identify the beginning of a block and record it, and then to determine if the block contains an Exception.

As is, the script finds the following four blocks:

***EXCEPTION***
06:14:33.3 Initializing Spring framework Logs
[...]
INFO: Jk running ID=0 time=0/105  config=null

***EXCEPTION***
03:19:33.9 CronServer:: DailyJob ads.tools.UpdateSummaries@17c5d6cf failed with exception ads.util.AppError: Cannot create UnitName from keys: Each
[...]
at ads.tools.SupportThread.run(SupportThread.java:101)

***EXCEPTION***
03:46:42.5 Periodic support request failed: ads.support.SupportException: Error opening socket: java.net.ConnectException: Connection refused

***EXCEPTION***
06:31:36.1 Upload failed: java.io.FileNotFoundException: c:/tmp/cygwin1.dll (No such file or directory)

That's based on interpreting each block as beginning with a 06:31:36 style timestamp; the stuff with Oct 18, 2013 6:14:33 is obviously JRE log stuff passed to the server log, and so part of one entry. If you want to do this differently, by separating the individual JRE exceptions, use the "Alternate version" of $timestamp, which regards both styles as starting an entry.

If you want to try the perl script with your actual logs, replace while (<DATA>) with while (<STDIN>) and pipe the log in (whatever.pl < log.txt). — goldilocks, Commented Dec 6, 2013 at 20:41
Sorry my comment got deleted. I tried with but no success. Need to spend a bit more time. — Ketan, Commented Dec 6, 2013 at 20:50
Although it's in perl and I don't have any idea I'll try to convert it to bash (hope to succeed) because it works perfectly!!! :D As DATA is the $*? "block" is necessary for bash? — C Ts, Commented Dec 9, 2013 at 15:30
I'm not sure what you mean. The DATA section is just a perl feature I used for testing, so it's not necessary no -- the <STDIN> version will work without it. As for $block, the idea is to accumulate the lines in a string; if the block turns out to be an Exception, you then print it, if not, throw it away and start accumulating again. I guess in bash if you were using while read line, $block .= $_ (perl) would be block="$block$line". Note the use of the sigil ($) is slightly different in perl vs. bash. — goldilocks, Commented Dec 9, 2013 at 15:45
I know it's been a while, but can you help me with this? #!/bin/bash timestamp=^[0-9][0-9]\:[0-9][0-9]\:[0-9][0-9]\.[0-9] block="" exception=false except=.*Exception.* while read line do if [[ -z $block ]] ; then if [[ $line =~ $timestamp ]] then if [[ $exception=false ]] then echo "\n\n***EXCEPTION***\n$block" fi block="" exception=false fi if [[ $line =~ $except ]] then exception=true fi block=$block$line fi if [[ $line =~ $timestamp ]] then block=$line fi done — C Ts, Commented Dec 23, 2013 at 17:13

Ketan · Accepted Answer · 2013-12-06 21:22:12Z

Alright, this is a close one:

awk -F "[0-9][0-9]:[0-9][0-9]:[0-9][0-9].[0-9]" '{for (i=1; i<= NF; i++) if ( $i ~ /[e|E]xception/ ) print $i} RS=""' t.txt

I use the timestamp regex as FS and search for the Exception in the fields. Not perfect but close.

After cating both pieces of logs OP provided, the output I got is as follows:

Initializing Spring framework Logs<br>
Oct 18, 2013 6:14:33 AM org.apache.catalina.startup.Embedded start<br>
INFO: Starting tomcat server<br>
Oct 18, 2013 6:14:34 AM org.apache.catalina.core.StandardEngine start<br>
INFO: Starting Servlet Engine: Apache Tomcat/6.0.32<br>
Oct 18, 2013 6:14:35 AM org.apache.catalina.startup.ContextConfig <br>efaultWebConfig
INFO: No default web.xml<br>
Oct 18, 2013 6:14:38 AM org.apache.catalina.session.StandardManager doLoad<br>
SEVERE: IOException while loading persisted sessions: <br>java.io.WriteAbortedException: writing aborted; <br>java.io.NotSerializableException: ads.doc.backoffice.StoreInfos<br>
java.io.WriteAbortedException: writing aborted; <br>java.io.NotSerializableException: ads.doc.backoffice.StoreInfos
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1354)<br>
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
...... <br>
...... <br>
...... <br>
...... <br>
    at ads.tools.AppServerMain.main(AppServerMain.java:83)<br>
Caused by: java.io.NotSerializableException: ads.doc.backoffice.StoreInfos<br>
INFO: Jk running ID=0 time=0/105  config=null<br>

 CronServer:: DailyJob ads.tools.UpdateSummaries@17c5d6cf failed with exception ads.util.AppError: Cannot create UnitName from keys: Each
ads.util.AppError: Cannot create UnitName from keys: Each
at ads.db.DBObjectDefault.createFromKeys(DBObjectDefault.java:42)
at ads.db.DBTable.createFromKeys(DBTable.java:227)
at ads.db.DBValue.getValue(DBValue.java:621)
at ads.dbmanager.DBObjectsManager.initObjects(DBObjectsManager.java:400)
at ads.dbmanager.DBObjectsManager.reload(DBObjectsManager.java:447)
at ads.dbmanager.DBObjectsManager.loadFromStore(DBObjectsManager.java:497)
at ads.doc.inventory.InventoryItemSummary.refreshSince(InventoryItemSummary.java:173)
at ads.db.DBSummaryTable.refreshAll(DBSummaryTable.java:67)
at ads.tools.CronServer$DailyThread.run(CronServer.java:271)
[SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.net.ConnectException: Connection refused; targetException=java.lang.IllegalArgumentException: Error opening socket: java.net.ConnectException: Connection refused]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(SOAPHTTPConnection.java:354)
at org.apache.soap.rpc.Call.invoke(Call.java:248)
at ads.support.SupportCall.call(SupportCall.java:56)
at ads.tools.SupportThread.run(SupportThread.java:101)

 Periodic support request failed: ads.support.SupportException: Error opening socket: java.net.ConnectException: Connection refused

 Upload failed: java.io.FileNotFoundException: c:/tmp/cygwin1.dll (No such file or directory)

Worked, but once the FS is the regex I'm missing the timestamp, is there any way to also print the FS? — C Ts, Commented Dec 9, 2013 at 15:04

Stack Exchange Network

bash script, printing multiple lines that matching to a specific patern

2 Answers 2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
bash
sed
awk
regular-expression
.

Hot Network Questions

bash script, printing multiple lines that matching to a specific patern

2 Answers 2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged bashsedawkregular-expression.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
bash
sed
awk
regular-expression
.