How to extract logs between two time stamps, line may or may not start with datetime

Question

How to extract logs between two time stamps, line may or may not start with datetime. I am trying below and it is only extracting the lines that starts with datetime only. Datetime format is 2014-04-07 23:00

$ awk \
  '$0 ~ /^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-2][0-9]:[0-5][0-9]/
      {
        if ($1" "$2 >= "2014-04-07 23:00")     p=1;
        if ($1" "$2 >= "2014-04-08 02:00")  p=0;
      }
    p { print $0 }' log

Here is my sample data:

2014-04-07 22:59:10.001 agaggagag  
gagagg 
2014-04-07 23:40:33.345 aegsgssdh  
wqtqttqtqtq  
post  
agggsdgg  
2014-04-08 01:00:54.777 ggsdgwettwetewt  
cvdgwetegdkiytitityi  
error 
2014-04-08 02:02:22.009 qwwqtwtwebbcbewdhshsdh  
asgsaftewtewt  
1253536443755475  
2014-04-08 10:55:34.934 etwtewtewppip  
jklhlljkjvncncmmm  
sghywywywywyw  

I want to extract all data between 2014-04-07 23:00 and 2014-04-08 02:00

Answer 1

The RE in the pattern match starts with ^ which binds the expression to the start of the line. If you want match your expression anywhere you need to remove it.

Your if... statements assume that the date/time fields are in $1 and $2, which are also (by definition) not necessarily true. Try this instead (it's untested because I don't have a sample of your data)

awk '
    {
        if (match($0, /\<[0-9]{4}-[0-9]{2}-[0-9]{2} [0-2][0-9]:[0-5][0-9]\>/))
        {
            s = substr($0, RSTART, RLENGTH)            
            if (s >= "2014-04-07 23:00") p=1
            if (s >= "2014-04-08 02:00") p=0
        }
    }
    p { print $0 }
' log

Output from sample data

2014-04-07 23:40:33.345 aegsgssdh
wqtqttqtqtq
post
agggsdgg
2014-04-08 01:00:54.777 ggsdgwettwetewt
cvdgwetegdkiytitityi
error

Answer 2

Change the beginning of the script to either

$ awk \
  '$0 ~ /^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-2][0-9]:[0-5][0-9]/ \
      {
           ︙

or

$ awk \
  '$0 ~ /^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-2][0-9]:[0-5][0-9]/    {
           ︙

Currently, your script has three statements:

1. If the line matches /^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-2][0-9]:[0-5][0-9]/, print it (default action).
2. For all lines (default condition), set p to 1 if $1 $2 is within the range and 0 if it is not.
3. If p is non-zero, print the line.

So every line that looks like a date-time stamp (based on your regexp) gets printed based on statement 1.  And every line that contains a date-time stamp within the selection range gets printed based on the interaction between statements 2 and 3.

Obviously, you want to bind condition 1 and statement 2 together.

Answer 3

I created a simple script for your your purpose. Check is this helpful for you

[upkar@server2 one]# cat logxtract.sh

L1=$(grep -n "2014-04-07 23:[0-9][0-9]" log | awk -F":" '{print $1}')

L2=$(grep -n "2014-04-08 02:[0-9][0-9]" log | awk -F":" '{print $1}')

sed -n $L1,"$L2"p log

Output of the script

[upkar@server2 one]# sh logxtract.sh

2014-04-07 23:40:33.345 aegsgssdh

wqtqttqtqtq

post

agggsdgg

2014-04-08 01:00:54.777 ggsdgwettwetewt

cvdgwetegdkiytitityi

error

2014-04-08 02:02:22.009 qwwqtwtwebbcbewdhshsdh