I have two types of log files

Standard Syslog format

The timestamp looks as below

5:2015 Dec 21 07:35:06:ABC:foo1:1559: common.c:946:Enabling filter
6:2015 Dec 21 07:35:08:ABC:bar1:1461: api.c:124:Trigger activated
6:2015 Dec 21 07:35:16:BMC:kernel:-:<6>drivers/usb1_1.c:598:Error processing request on endpoint 0

Custom Log format

The date time is given in first line of the file. following lines are "relative" to the first line. example as below:

Timestamp H:M:S 15:4:1 D:M:Y 16:1:2015
Firmware Version: 121020150140
[04:01]------------[ Logs ]------------
[04:03]Device Data: -> Supported Attributes -> 0x8033B
[04:01]Device Cleanup

[04:19]SendClearMsg ...
[04:23]Param:GetData failed
[04:51]Current Update Count:7
[05:01]MODECHK:Normal mode

a timestamp of 4:01 is 4 minutes, 1 second since 15:4:1 which should be translated as 15:08:2.

Any suggestions to achieve this ?

  • translate relative times into absolute time
  • Merge with syslog into one big file, sorted by time
  • What is the cost of changing the way the second timestamp is displayed ?
    – netmonk
    Commented Dec 21, 2015 at 9:16
  • 1
    @netmonk okay..i got what you are asking. We don't have the source for second log, it is from a different module from third party. Commented Dec 21, 2015 at 9:52
  • ok so we need to find a solution :)
    – netmonk
    Commented Dec 21, 2015 at 10:13
  • @netmonk i am exploring with "date "-d +5 min 3 sec",it works but i am having trouble replacing it back in same line number in log Commented Dec 21, 2015 at 10:32
  • What are those leading 5:, 6:, do you want to keep them? Before the timestamp? Do you want to keep the Timestamp and Firmware Version line? Do you mind the final format for the timestamp? Is 2015-12-21 13:12:02 OK? Commented Dec 21, 2015 at 12:29

3 Answers 3


This solution uses an awk script to convert the dates in the first file to the number of seconds since the epoch and preprends this number to the output. We use date +%s --date to do the hard work, and capture the output of the command into awk variable secs by calling awk's getline function. (The awk syntax is: command | getline variable).

awk <log1 >log1.new '
{ y = substr($0,3,4); m = $2; d = $3; hms = substr($0,15,8)
  "date \"+%s\" --date \"" d " " m " " y " " hms "\"" | getline secs
  print secs " " $0

A second awk script does the same for the second file, but only for the first line with the Timestamp, which is saved in awk variable base. On the other lines we simply add the minutes and seconds offset to this base and use date to convert the seconds since the epoch into a real date, in the format of the first file.

awk <log2 >log2.new '
/^Timestamp/{ split($5,x,":"); dmy = sprintf("%04d/%02d/%02d",x[3],x[2],x[1])
              split($3,x,":"); hms = sprintf("%02d:%02d:%02d",x[1],x[2],x[3])
              "date \"+%s\" --date \"" dmy " " hms "\"" | getline base
/^\[/ { mins = substr($0,2,2); secs = substr($0,5,2);
        tot = base + mins*60+secs
        "date \"+%Y %b %d %H:%M:%S\" --date @" tot | getline date
        print tot " -:" date " " substr($0,8)

The two files are then merged by a sort on the number field, and finally the number is removed by a sed.

sort -m -n -k1,1 log1.new log2.new |
sed 's/^[^ ]* //'

This adds the relative numbers produced by your custom log file to the original base number and updates all lines

#!/usr/bin/env bash

f=$(head -n 1 custom_log_format.log)
base=$(sed 's/.*H:M:S \(.*\) D:M:Y.*/\1/' <<<$f)

readarray lines < custom_log_format.log
for i in ${!lines[@]}
    if [[ $b == "["* ]]
        rel_time=$(sed 's/^\[\(.*[^ ]\)\].*/\1/' <<<$b)
        time=$(echo $rel_time | awk -F: '{ print ($1 * 60) + $2 * 60 }')
        # convert base to seconds
        base_seconds=$(date -d"$base" +"%s")
        new_time_seconds=$(( base_seconds + time ))

        new_time=$(date -d"@$new_time_seconds" +"%H:%M:%S")
    echo ${b/$rel_time/$new_time}


Timestamp H:M:S 15:4:1 D:M:Y 16:1:2015
Firmware Version: 121020150140
[15:09:01]------------[ Logs ]------------
[15:11:01]Device Data: -> Supported Attributes -> 0x8033B
[15:09:01]Device Cleanup

[15:27:01]SendClearMsg ...
[15:31:01]Param:GetData failed
[15:59:01]Current Update Count:7
[15:10:01]MODECHK:Normal mode

Try rust-based tool Super Speedy Syslog Searcher

(assuming you have rust installed)

cargo install super_speedy_syslog_searcher


s4 log1 log2

Super Speedy Syslog Searcher will sort varying log message datetime formats by the interpreted datetime.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .