1

I have two types of log files

Standard Syslog format

The timestamp looks as below

5:2015 Dec 21 07:35:06:ABC:foo1:1559: common.c:946:Enabling filter
6:2015 Dec 21 07:35:08:ABC:bar1:1461: api.c:124:Trigger activated
6:2015 Dec 21 07:35:16:BMC:kernel:-:<6>drivers/usb1_1.c:598:Error processing request on endpoint 0

Custom Log format

The date time is given in first line of the file. following lines are "relative" to the first line. example as below:

Timestamp H:M:S 15:4:1 D:M:Y 16:1:2015
Firmware Version: 121020150140
[04:01]------------[ Logs ]------------
[04:03]Device Data: -> Supported Attributes -> 0x8033B
[04:01]Device Cleanup

[04:19]SendClearMsg ...
[04:23]Param:GetData failed
[04:51]Current Update Count:7
[05:01]MODECHK:Normal mode

a timestamp of 4:01 is 4 minutes, 1 second since 15:4:1 which should be translated as 15:08:2.

Any suggestions to achieve this ?

  • translate relative times into absolute time
  • Merge with syslog into one big file, sorted by time
  • What is the cost of changing the way the second timestamp is displayed ? – netmonk Dec 21 '15 at 9:16
  • 1
    @netmonk okay..i got what you are asking. We don't have the source for second log, it is from a different module from third party. – rajaganesh87 Dec 21 '15 at 9:52
  • ok so we need to find a solution :) – netmonk Dec 21 '15 at 10:13
  • @netmonk i am exploring with "date "-d +5 min 3 sec",it works but i am having trouble replacing it back in same line number in log – rajaganesh87 Dec 21 '15 at 10:32
  • What are those leading 5:, 6:, do you want to keep them? Before the timestamp? Do you want to keep the Timestamp and Firmware Version line? Do you mind the final format for the timestamp? Is 2015-12-21 13:12:02 OK? – Stéphane Chazelas Dec 21 '15 at 12:29
2

This solution uses an awk script to convert the dates in the first file to the number of seconds since the epoch and preprends this number to the output. We use date +%s --date to do the hard work, and capture the output of the command into awk variable secs by calling awk's getline function. (The awk syntax is: command | getline variable).

awk <log1 >log1.new '
{ y = substr($0,3,4); m = $2; d = $3; hms = substr($0,15,8)
  "date \"+%s\" --date \"" d " " m " " y " " hms "\"" | getline secs
  print secs " " $0
}'

A second awk script does the same for the second file, but only for the first line with the Timestamp, which is saved in awk variable base. On the other lines we simply add the minutes and seconds offset to this base and use date to convert the seconds since the epoch into a real date, in the format of the first file.

awk <log2 >log2.new '
/^Timestamp/{ split($5,x,":"); dmy = sprintf("%04d/%02d/%02d",x[3],x[2],x[1])
              split($3,x,":"); hms = sprintf("%02d:%02d:%02d",x[1],x[2],x[3])
              "date \"+%s\" --date \"" dmy " " hms "\"" | getline base
}
/^\[/ { mins = substr($0,2,2); secs = substr($0,5,2);
        tot = base + mins*60+secs
        "date \"+%Y %b %d %H:%M:%S\" --date @" tot | getline date
        print tot " -:" date " " substr($0,8)
}'

The two files are then merged by a sort on the number field, and finally the number is removed by a sed.

sort -m -n -k1,1 log1.new log2.new |
sed 's/^[^ ]* //'
1

This adds the relative numbers produced by your custom log file to the original base number and updates all lines

#!/usr/bin/env bash

f=$(head -n 1 custom_log_format.log)
base=$(sed 's/.*H:M:S \(.*\) D:M:Y.*/\1/' <<<$f)

OLDIFS=$IFS
IFS=$'n' 
readarray lines < custom_log_format.log
IFS=$OLDIFS
for i in ${!lines[@]}
do
    b="${lines[$i]}"
    if [[ $b == "["* ]]
    then
        rel_time=$(sed 's/^\[\(.*[^ ]\)\].*/\1/' <<<$b)
        time=$(echo $rel_time | awk -F: '{ print ($1 * 60) + $2 * 60 }')
        # convert base to seconds
        base_seconds=$(date -d"$base" +"%s")
        new_time_seconds=$(( base_seconds + time ))

        new_time=$(date -d"@$new_time_seconds" +"%H:%M:%S")
    fi
    echo ${b/$rel_time/$new_time}
done

output

Timestamp H:M:S 15:4:1 D:M:Y 16:1:2015
Firmware Version: 121020150140
[15:09:01]------------[ Logs ]------------
[15:11:01]Device Data: -> Supported Attributes -> 0x8033B
[15:09:01]Device Cleanup

[15:27:01]SendClearMsg ...
[15:31:01]Param:GetData failed
[15:59:01]Current Update Count:7
[15:10:01]MODECHK:Normal mode

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.