1

I have a file with contents like (could be more than 3 entries)

A Version: x_02.28.03.03 000000 aaa 2019/05/21 03:33:04
B Version: x_02.28.03.03 000000 aaa 2019/05/21 03:33:04
C Version: 0.01.011 #3 PREEMPT Tue Apr 4 09:14:17 UTC 2023

Now I would like extract date with time in unix timestamp from all the entries. that is, I am interested in 2019/05/21 03:33:04, 2019/05/21 03:33:04 and Tue Apr 4 09:14:17 UTC 2023. And these entries should be in same formiat so that I can compare them afterwards. Also, position is not fixed (but it will be the last two fields of a line).

Below is part of bash script:

#!/bin/bash

ver_file="/home/test/tmp.txt"

ver_c=$(grep -E "C Version:" $ver_file | cut -d" " -f3-)

echo "$ver_c"

Can aynone please let me know how can I extract dates from a file?

P.S: I am developing on Ubuntu with WSL2 but the target will be using busybox date. Please let me know if any info is missing.

4
  • The third one is clearly UTC. But what timezone are the datetime values in the first two lines? Apr 11, 2023 at 7:05
  • I am not sure about that...
    – Preeti
    Apr 11, 2023 at 7:06
  • 2
    It's impossible to compare them correctly without knowing this. Or else you need to make a reasonable assumption Apr 11, 2023 at 7:07
  • 2
    (a) The last line uses six fields for the datetime. (Possible that some whitespace is tabs and other is spaces, but not likely in log records). (b) The TZ query means: 2019/05/21 03:33:04 Pacific Daylight Time or Japan Standard Time -- there is an invisible sixteen-hour difference. Apr 11, 2023 at 9:49

3 Answers 3

2

The description is not really a question, and somewhat confusing. But, dates are always fun to work with so I hope this helps.

Extracting unstructured dates depends on the source. Dates in files are notorious. In the examples given, the only obvious consistency I see is the date strings are at the end of the lines and all start at column 6. That's what I'd look for, first.

If the positions are not 'fixed', assuming they won't all start at column 6, then the date in the third line is also not the last two columns. A confusing example. Regardless, it can still be done. More logic would be needed to evaluate the different types of date strings and what to do with each. Again, it really depends on the input data quality (GIGO).

This could be done different ways with GNU bash & core utilities. The main tool to highlight is the GNU date command, which can evaluate dates for validity and normalize them. In this case, "UTC 2023" is technically a valid date, so GNU date will not error on it (and that must be caught with bash). Still, problems like this can be solved very simply with a high degree of accuracy.

Something like this, assuming all date strings start at column 6 or a valid date is in the last two columns ...

while read line; do
    echo $line

    DATE_SIX="$(echo $line | cut -f6- -d' ')"
    if date --utc --date "${DATE_SIXE}" &> /dev/null; then
        DATE_SIX_NORMAL="$(date --utc --date "${DATE_SIX}")"
        DATE_SIX_EPOCH="$(date --utc --date "${DATE_SIX}" +%s)"
    else
        DATE_SIX_NORMAL="BAD DATE"
        DATE_SIX_EPOCH=0
    fi
    echo "DATE_SIX='${DATE_SIX}', DATE_SIX_NORMAL='${DATE_SIX_NORMAL}', DATE_SIX_EPOCH=${DATE_SIX_EPOCH}"

    DATE_LAST_TWO="$(echo $line | awk '{print $(NF-1)" "$(NF)}')"
    if [[ "${DATE_LAST_TWO}" != *":"* ]] || [[ "${DATE_LAST_TWO}" != *"/"* ]]; then
        # GNU date evaluates "UTC 2023" as a valid date, but it's not what's wanted ...
        DATE_LAST_TWO_NORMAL="BAD DATE"
        DATE_LAST_TWO_EPOCH=0
    else
        if date --utc --date "${DATE_LAST_TWO}" &> /dev/null; then
            DATE_LAST_TWO_NORMAL="$(date --utc --date "${DATE_LAST_TWO}")"
            DATE_LAST_TWO_EPOCH="$(date --utc --date "${DATE_LAST_TWO}" +%s)"
        else
            DATE_LAST_TWO_NORMAL="BAD DATE"
            DATE_LAST_TWO_EPOCH=0
        fi
    fi
    echo "DATE_LAST_TWO='${DATE_LAST_TWO}', DATE_LAST_TWO_NORMAL='${DATE_LAST_TWO_NORMAL}', DATE_LAST_TWO_EPOCH=${DATE_LAST_TWO_EPOCH}"

    echo
done < in.tmp

The output from that looks like this. Of course, DATE_EPOCH could be used as an integer for comparisons.

A Version: x_02.28.03.03 000000 aaa 2019/05/21 03:33:04
DATE_SIX='2019/05/21 03:33:04', DATE_SIX_NORMAL='Tue May 21 03:33:04 AM UTC 2019', DATE_SIX_EPOCH=1558409584
DATE_LAST_TWO='2019/05/21 03:33:04', DATE_LAST_TWO_NORMAL='Tue May 21 03:33:04 AM UTC 2019', DATE_LAST_TWO_EPOCH=1558409584

B Version: x_02.28.03.03 000000 aaa 2019/05/21 03:33:04
DATE_SIX='2019/05/21 03:33:04', DATE_SIX_NORMAL='Tue May 21 03:33:04 AM UTC 2019', DATE_SIX_EPOCH=1558409584
DATE_LAST_TWO='2019/05/21 03:33:04', DATE_LAST_TWO_NORMAL='Tue May 21 03:33:04 AM UTC 2019', DATE_LAST_TWO_EPOCH=1558409584

C Version: 0.01.011 #3 PREEMPT Tue Apr 4 09:14:17 UTC 2023
DATE_SIX='Tue Apr 4 09:14:17 UTC 2023', DATE_SIX_NORMAL='Tue Apr  4 09:14:17 AM UTC 2023', DATE_SIX_EPOCH=1680599657
DATE_LAST_TWO='UTC 2023', DATE_LAST_TWO_NORMAL='BAD DATE', DATE_LAST_TWO_EPOCH=0

... and there are other ways than cut, with awk, bash string manipulation, etc.

The GNU date command can convert and normalize timestamps.

I'm also assuming the original dates without a timezone are UTC.

However, it's also possible to specify a custom timezone, too (i.e. using TZ before date).

For example,

$ date --utc --date="2019/05/21 03:33:04"
Tue May 21 03:33:04 AM UTC 2019
$ date --utc --date="Tue Apr 4 09:14:17 UTC 2023"
Tue Apr  4 09:14:17 AM UTC 2023

Or, to convert the original date string to epoch time ...

$ date --utc --date="2019/05/21 03:33:04" +%s
1558409584
 date --utc --date="Tue Apr 4 09:14:17 UTC 2023" +%s
1680599657

... or, use any combination of options or FORMAT controls from man date(1), e.g.

$ date --utc --date="2019/05/21 03:33:04" --rfc-email
Tue, 21 May 2019 03:33:04 +0000
$ date --utc --date="2019/05/21 03:33:04" +%Y%m%d%H%M%S
20190521033304
 date --utc --date="Tue Apr 4 09:14:17 UTC 2023" +%s
1680599657
$ TZ=America/New_York date --date="Tue Apr 4 09:14:17 UTC 2023"
Tue Apr  4 05:14:17 AM EDT 2023

For comparisons, I prefer unix epoch timestamps.

2

You haven't defined the timezone for the first values, so I have assumed it's "local time" for whatever part of the world you're in.

I've used GNU grep and date here:

grep -oE '..../../.. ..:..:..$|... [[:digit:]]+ ..:..:.. [[:alnum:]]+ ....$' datafile |
    while IFS= read date
    do
        esec=$(date --date "$date" +%s)
        printf "%s --> %d\n" "$date" "$esec"
    done

For your sample data here is the output when run in my (UK - GMT/BST) timezone. Your seconds values will be different unless you're also in GMT/BST or the equivalent WET timezone.

2019/05/21 03:33:04 --> 1558405984
2019/05/21 03:33:04 --> 1558405984
Apr 4 09:14:17 UTC 2023 --> 1680599657
4
  • Hello @roaima, thanks for the answer. I am on busybox and above solution doesn't work for me. It gives date: invalid date error. How can I make it work with busybox date? My busybox version is BusyBox v1.24.1
    – Preeti
    Apr 17, 2023 at 14:40
  • Your question says Ubuntu on WSL. Isn't that with GNU date? Apr 17, 2023 at 16:37
  • That's correct. But my script will be executed on a system with busybox.. sorry this was not mentioned in OP
    – Preeti
    Apr 17, 2023 at 19:41
  • It was a pretty important piece of information to omit, especially as you said you were using Ubuntu on WSL Apr 17, 2023 at 20:55
2

These dates are easily parsed by perl's Date::Parse:

To prepend the Unix epoch time (which can be compared numerically):

perl -MDate::Parse -pe '
  $_ = str2time(m{( 20\d\d/\d\d.*|\S+ \S+ \d+ \S+ UTC 20\d\d$)}) . " $_"
  ' < your-file

Which gives:

1558409584 A Version: x_02.28.03.03 000000 aaa 2019/05/21 03:33:04
1558409584 B Version: x_02.28.03.03 000000 aaa 2019/05/21 03:33:04
1680599657 C Version: 0.01.011 #3 PREEMPT Tue Apr 4 09:14:17 UTC 2023

Or for ISO8601-style time format (which can be compared lexically):

perl -MDate::Parse -MPOSIX -pe '
  $_ = strftime("%FT%T", strptime m{(20\d\d/\d\d.*|\S+ \S+ \d+ \S+ UTC 20\d\d$)}) . " $_"
  ' < your-file

Which gives:

2019-05-21T02:33:04 A Version: x_02.28.03.03 000000 aaa 2019/05/21 03:33:04
2019-05-21T02:33:04 B Version: x_02.28.03.03 000000 aaa 2019/05/21 03:33:04
2023-04-04T09:14:17 C Version: 0.01.011 #3 PREEMPT Tue Apr 4 09:14:17 UTC 2023

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .