2

I have multiline log file and I want to convert it to single line log.

Multiline example:

6/13/2015 12:00:47 AM - {   562} START Web 
6/13/2015 12:00:47 AM - Requested Web connection from 123.125.71.103 [123.125.71.103], ID=562 
6/13/2015 12:01:24 AM - {   563} START POP3 
6/13/2015 12:01:24 AM - Requested POP3 connection from 10.127.251.37 [10.127.251.37], ID=563 
6/13/2015 12:01:24 AM - (   563) USER test.mail@test.me 
6/13/2015 12:01:24 AM - POP3 connection with 10.127.251.37 [10.127.251.37] ended. ID=563 
6/13/2015 12:01:24 AM - {   563} END POP3
6/13/2015 12:01:24 AM - {   564} START POP3 
6/13/2015 12:01:24 AM - Requested POP3 connection from 10.127.251.37 [10.127.251.37], ID=564 
6/13/2015 12:01:24 AM - (   564) USER test.mail@test.me 
6/13/2015 12:01:24 AM - POP3 connection with 10.127.251.37 [10.127.251.37] ended. ID=564 
6/13/2015 12:01:24 AM - {   564} END POP3
6/13/2015 12:01:40 AM - Web connection with 123.125.71.103 [123.125.71.103] ended. ID=562 
6/13/2015 12:01:40 AM - {   562} END Web

For start I would like single line output like this, where I match same log IDs (for example "562").

6/13/2015 12:00:47 AM - {   562} START Web 6/13/2015 12:00:47 AM - Requested Web connection from 123.125.71.103 [123.125.71.103], ID=562 6/13/2015 12:01:40 AM - Web connection with 123.125.71.103 [123.125.71.103] ended. ID=562 6/13/2015 12:01:40 AM - {   562} END Web
6/13/2015 12:01:24 AM - {   563} START POP3 6/13/2015 12:01:24 AM - Requested POP3 connection from 10.127.251.37 [10.127.251.37], ID=563 6/13/2015 12:01:24 AM - (   563) USER test.mail@test.me  6/13/2015 12:01:24 AM - POP3 connection with 10.127.251.37 [10.127.251.37] ended. ID=563  6/13/2015 12:01:24 AM - {   563} END POP3
6/13/2015 12:01:24 AM - {   564} START POP3 6/13/2015 12:01:24 AM - Requested POP3 connection from 10.127.251.37 [10.127.251.37], ID=564 6/13/2015 12:01:24 AM - (   564) USER test.mail@test.me  6/13/2015 12:01:24 AM - POP3 connection with 10.127.251.37 [10.127.251.37] ended. ID=564  6/13/2015 12:01:24 AM - {   564} END POP3

I have done following bash script which is not working as expected since it's merging all "POP3" or "Web" messages to single line and not sepparating them based on message ID.

Script:

#!/bin/bash

HOME=/var/tmp/test.txt

ID=`((awk '$6 ~/[0-9]\W/ {print $6}' $HOME | awk '{gsub (/)/, ""); print}' | awk '{gsub (/}/, ""); print}') && (awk '$11 ~/[0-9]/ {print $11}' $HOME | awk '{gsub ("ID=", ""); print}'))`


for ID in $HOME
do
        awk '!/Web/' $HOME | xargs >> final.txt
        awk '/Web/' $HOME | xargs >> final.txt
done

Any suggestion how I should create loop to merge only same IDs?

0

If you can get by on just the timestamps, the following is enough:

sed -e:n -e'$!N;s/^\(\([^-]*-\).*\)\n *\2/\1:::/;tn' -eP\;D <in >out

It recursively appends the Next line to the current one, and, if all of the characters at the head of the current line up to and including the first - dash can match the head of the appended line, the two are joined and the appended timestamp is removed. If that s///ubstitution tests successful, sed will branch back to the :n label for another New line, else all pending merged data is Printed to stdout before it is Deleted and sed starts over from the top to try again.

Given your sample data, it prints:

6/13/2015 12:00:47 AM - {   562} START Web ::: Requested Web connection from 123.125.71.103 [123.125.71.103], ID=562
6/13/2015 12:01:24 AM - {   563} START POP3 ::: Requested POP3 connection from 10.127.251.37 [10.127.251.37], ID=563 ::: (   563) USER test.mail@test.me ::: POP3 connection with 10.127.251.37 [10.127.251.37] ended. ID=563 ::: {   563} END POP3::: {   564} START POP3 ::: Requested POP3 connection from 10.127.251.37 [10.127.251.37], ID=564 ::: (   564) USER test.mail@test.me ::: POP3 connection with 10.127.251.37 [10.127.251.37] ended. ID=564 ::: {   564} END POP3
6/13/2015 12:01:40 AM - Web connection with 123.125.71.103 [123.125.71.103] ended. ID=562 ::: {   562} END Web

But that's not very good, apparently. It appears you want to merge on ID - sorry about that. The following does work - it also clears repetitive timestamps and IDS as the occur in input.

sed   -e'y/)},={/(((((/' \
      -e's/-\([^(I]*\)[^0-9]*\([0-9]*\)[( ]*/- \2 -\1/;=' |
paste -d- - - |
sort -t- -nk3,3 -nk1,1 |
sed   -e's/^[^-]*-//;:n' -e'h;$!N' \
      -e's/\(-\([^-]*-\).*[^ ]\) *\n\([^-]*-\)\{2\}\2/\1 - \3/;tn' \
      -ex\;:t -e's/\(\([^-]*-\)[^/]*\)- *\2/\1:::/;tt'   -e'p;g;D'

6/13/2015 12:00:47 AM - 562 - START Web ::: Requested Web connection from 123.125.71.103 [123.125.71.103] - 6/13/2015 12:01:40 AM - Web connection with 123.125.71.103 [123.125.71.103] ended. ::: END Web
6/13/2015 12:01:24 AM - 563 - START POP3 ::: Requested POP3 connection from 10.127.251.37 [10.127.251.37] ::: USER test.mail@test.me ::: POP3 connection with 10.127.251.37 [10.127.251.37] ended. ::: END POP3
6/13/2015 12:01:24 AM - 564 - START POP3 ::: Requested POP3 connection from 10.127.251.37 [10.127.251.37] ::: USER test.mail@test.me ::: POP3 connection with 10.127.251.37 [10.127.251.37] ended. ::: END POP3
  • Thanks mikeserv, I was thinking to remove timestamps as well, since they are repeating several times. Your approach is not merging all same IDs into single line. For example ID 562 in your print is present in first and third line. I would like to merge all same IDs into single line. I provide only sample of logs. As well you can see in your printed output that timestamp in first line is different than in third line, but ID is same (this apply only for logs with "START Web" or "END Web"). – mmesojedec Jul 26 '15 at 18:14
  • @mmesojedec - ok. Well, ^that^ does it.... I dunno what that means? You want to split on IDS and not on timestamps? – mikeserv Jul 26 '15 at 18:14
  • Actually suggested solution above from DarkHeart is working as expected. If repeated timestamp can be removed, that would be great. – mmesojedec Jul 26 '15 at 18:16
  • I want to join all lines with same ID. there can be 2 or n lines with same ID and I want to join them together. of course if timestamp can be removed would be great as well. – mmesojedec Jul 26 '15 at 18:28
  • @mmesojedec - sorry it took me awhile to get it. But I think I do now. – mikeserv Jul 26 '15 at 19:35
2

You could just do the whole thing in awk. The following combines the IDs as they were read.

awk  '{
    line = $0;
    # ID is { XXX } or ( XXX )
    if ( /[{(] *[0-9]+[})]/ ) {
        id = $0;
        sub(/ *[})].*/,"", id);
        sub(/.*[({] */,"", id);
    }
    # ID is ID=XXX
    else if ( $NF ~ /ID=/ ) {
        id = $NF;
        sub(/[^=]*=/,"",id);
    }
    # else ID= previous value

    # save line into a assoc. array of IDs
    final[id] = final[id]""line" "; # add space between lines
}
END {
    # print foreach id
    for ( id in final ) {
        print final[id];
    }
}
' /var/tmp/text.txt

You could reduce superfluous information, such as printing of the IDs and instead just use a prefix e.g.

     # remove ID
     sub(/ID=[0-9]/,"",id);
     sub(/[({] *[0-9]+[})]/,"",id);

     END {
         # print foreach id
         for ( id in final ) {
             #Print ID then the rest of the line
             printf("[ID=%d]: %s\n", id, final[id]);
         }
     }
1

Based on @mikeserv approach I'm getting following output.

SCRIPT:

(   sed   -e'y/)},={/(((((/' \
          -e's/-\([^(I]*\)[^0-9]*\([0-9]*\)[( ]*/- \2 -\1/;=' |
    paste -d- - - |
    sort  -t- -nk3,3 -nk1,1 |
    sed   -e's/^[^-]*-//;:n' -e'h;$!N' \
          -e's/\(-\([^-]*-\).*[^ ]\) *\n\([^-]*-\)\{2\}\2/\1 - \3/;tn' \
          -ex\;:t -e's/\(\([^-]*-\)[^/]*\) - *\2/\1,/;tt'   -e'p;g;D'
)   < in.txt > out.txt

6/13/2015 12:00:47 AM - 562 - START Web, Requested Web connection from 123.125.71.103 [123.125.71.103] - 6/13/2015 12:01:40 AM - Web connection with 123.125.71.103 [123.125.71.103] ended., END Web
6/13/2015 12:01:24 AM - 563 - START POP3, Requested POP3 connection from 10.127.251.37 [10.127.251.37], +OK ArGoSoft Mail Server Pro for WinNT/2000/XP( Version 1.8 (1.8.9.6( - 6/13/2015 12:01:24 AM - CAPA, -ERR Unknown command, USER test.mail@test.me, +OK Password required for test.mail@test.me, PASS XXXXXXXXX, +OK Mailbox locked and ready, Adding address to POP Before SMTP manager, STAT, +OK 178 97537344, UIDL, +OK, ., LIST, +OK, ., QUIT, +OK Aba he, POP3 connection with 10.127.251.37 [10.127.251.37] ended., END POP3
6/13/2015 12:04:25 AM - 564 - START POP3, Requested POP3 connection from 10.127.251.37 [10.127.251.37], +OK ArGoSoft Mail Server Pro for WinNT/2000/XP( Version 1.8 (1.8.9.6( - 6/13/2015 12:04:25 AM - CAPA, -ERR Unknown command, USER test.mail@test.me, +OK Password required for test.mail@test.me, PASS XXXXXXXXX, +OK Mailbox locked and ready, Adding address to POP Before SMTP manager, STAT, +OK 178 97537344, UIDL, +OK, ., LIST, +OK, . - 6/13/2015 12:04:26 AM - QUIT, +OK Aba he, POP3 connection with 10.127.251.37 [10.127.251.37] ended., END POP3
6/13/2015 12:04:36 AM - 565 - START Web, Requested Web connection from 31.133.9.16 [31.133.9.16], Web connection with 31.133.9.16 [31.133.9.16] ended., END Web
6/13/2015 12:07:26 AM - 566 - START POP3, Requested POP3 connection from 10.127.251.37 [10.127.251.37], +OK ArGoSoft Mail Server Pro for WinNT/2000/XP( Version 1.8 (1.8.9.6( - 6/13/2015 12:04:25 AM - CAPA, -ERR Unknown command, USER test.mail@test.me, +OK Password required for test.mail@test.me, PASS XXXXXXXXX, +OK Mailbox locked and ready, Adding address to POP Before SMTP manager, STAT, +OK 178 97537344, UIDL, +OK, ., LIST, +OK, . - 6/13/2015 12:04:26 AM - QUIT, +OK Aba he, POP3 connection with 10.127.251.37 [10.127.251.37] ended., END POP3

You can see that in this example in line 4 we are missing timestamp before "Web connection with 31.133.9.16 [31.133.9.16] ended.". This will be same issue for all similar logs starting with "Web connection.....". For all other log containing POP3 messages everything is OK.

How should I modify sed command to include also timestamp for all remaining "Web connection...." messages and not only first one?

  • You don't need the third sed - just to remove the space before the - in the first place. So, I helped a little.... Really though, this is no the place for a question. If you have another question, you should ask another question. Anyway, it's supposed to remove duplicate timestamps. Does it not do that? – mikeserv Jul 27 '15 at 8:42
  • @mikeserv - Thanks for help. Yes, it remove duplicate timestamps. I just realized that timestamp for example above is same for ID 565, so timestamp before "Web connection....." is removed and this is expected output. – mmesojedec Jul 27 '15 at 9:22
  • @mikeserv - how should I include also IDs which are square brackets. Example like this "6/13/2015 2:15:09 AM - { 667} START Delivery 6/13/2015 2:15:09 AM - Active delivery threads (ID=667 start): 0 6/13/2015 2:15:09 AM - [ 667] Delivering to test.mail@test.com 6/13/2015 2:15:09 AM - [ 667] Forwarding mail... 6/13/2015 2:15:09 AM - [ 667] Forwarding message to 1 address(es) 6/13/2015 2:15:09 AM - Delivery thread, ID=667, terminated after running for 0.031 seconds. 6/13/2015 2:15:09 AM - { 667} END Delivery 6/13/2015 2:15:09 AM - Active delivery threads (ID=667 end): 0". – mmesojedec Jul 27 '15 at 20:53
  • @mikeserv - I tried to add " ([0-9]*)[[ ]* " into second line of SED command. – mmesojedec Jul 27 '15 at 20:55
0

So as IDs in each line in 6th field or in beforelast field collect all lines by ID is possible w/out sub

awk -F"[ }=)]+" '
NF{
    if($6 ~ "[0-9]{3}")
        ids=$6
    else 
        ids=$(NF-1)

    if(!M[ids])
        M[ids]=$0
    else
        M[ids]=M[ids] " " $0
}
END{
    for(i in M)
        print M[i]
}' /var/tmp/text.txt
  • I'm getting following error when executing this: "awk: cmd. line:5: (FILENAME=/var/tmp/multilog.txt FNR=54) fatal: attempt to access field -1". I modify only text location and output (like this: /var/tmp/multilog.txt > singlelog.txt). – mmesojedec Jul 26 '15 at 18:21
  • @mmesojedec That's mean the 54th row in /var/tmp/multilog.txt file is empty. Have edited script to avoid the problem. – Costas Jul 27 '15 at 7:36
  • OK. let me check. – mmesojedec Jul 27 '15 at 7:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.