I am trying to break a large log file into smaller files based on date.

The file is of the following form, where some lines may not have a date. Those lines should be included with the previous dated line.

2014-04-07T23:59:58 CheckForCallAction [ERROR] Exception caught
Undated line 1
Undated line 2
2014-04-08T00:00:03 MobileAppRequestFilter [DEBUG] Action
undated line 3
2015-04-08T00:00:03 MobileAppRequestFilter [DEBUG] ActionB

I found How to extract logs between two time stamps which is close to what I want, except my log file does not include a "[" at the start of the date, or "]" and the end of the date.

The command from that link is:

awk -F'[[]|[]]' \
  '$0 ~ /^\[/ && $2 >= "2014-04-07 23:00" { p=1 }
   $0 ~ /^\[/ && $2 >= "2014-04-08 02:00" { p=0 }
   p { print $0 }' > test1.log  logwith[.log

I have been trying for several days to modify this, but I just can't seem to get it.

A desired enhancement would be to not have to specify a start and end date, but rather automatically name the output files by either year, or year-month.

  • Please see formatting tools for help on formatting your posts.
    – terdon
    Apr 7 '16 at 14:47
  • There is an example extraction for data "without [] braces" in the answer you referenced: unix.stackexchange.com/a/123983/4252
    – KM.
    Apr 7 '16 at 14:52
  • The length of the date and time strings are looking very uniform to me. You can extract both with a simple cut command or $1 in awk replaces the actual date and $2 the actual time. In short, you don't need square brackets. The post you linked is a common case but not necessarily the norm.
    – MelBurslan
    Apr 7 '16 at 14:52
  • @KM, the only compact example at unix.stackexchange.com/a/123983/4252 does not include any line that do not include a date. I need those lines.
    – Mike
    Apr 7 '16 at 16:33
  • @MelBursian Thanks for the response. I am not knowledgeable enough to be able to convert your guidance into a workable command. Everything I tried had $1 either a date, or for lines without a date, a string. That made me lost as to how to proceed.
    – Mike
    Apr 7 '16 at 16:55

Use T as the field delimiter and check for date-like strings explicitly. For example, to split by year:

awk -FT '($1~/^[0-9]+-[0-9]+-[0-9]+$/){d=substr($1,1,4)}{print > d".log"}' logfile 

And by year+month:

awk -FT '($1~/^[0-9]+-[0-9]+-[0-9]+$/){split($1,d,"-")}{print > d[1]d[2]".log"}' logfile 

Here, we check that the first field (defined by T, so the whole date on lines starting with dates, that's what -FT means) is a set of 3 numbers separated by -. If it is, to get the year, we extract the first 4 characters (d=substr($1,1,4)) and, to get the month, we split the 1st field on -, saving the resulting strings in the array d (split($1,d,"-")), and use the 1st two elements of the array (d[1]d[2]) for the file name.

  • Thank you. This is so close to what I am looking for. I need to be able to break up the original file either by year, or by year-month. All variations I tired do not include the non-dated lines for some reason. Can you tweak this?
    – Mike
    Apr 7 '16 at 16:52
  • @Mike please edit your question and explain exactly what you need. Let me know when you've done so and I'll give be it a try.
    – terdon
    Apr 7 '16 at 16:54
  • I thought I had the requirements in the question "A desired enhancement would be to not have to specify a start and end date, but rather automatically name the output files by either year, or year-month." If that is not clear, let me know and I will try a different way. What you provided is exactly what I wanted, only that I wanted either a yearly or monthly file, not a daily one.
    – Mike
    Apr 7 '16 at 17:23
  • @Mike indeed you did, sorry, I missed that. I'm afraid I don't have the time to do this right now, but I'll answer tomorrow. The basic nidea will probably be just taking - as the field delimiter.
    – terdon
    Apr 7 '16 at 17:27
  • I will appreciate that. The variations I tried do not include the lines without dates, which is needed. Looking forward to your resolution.
    – Mike
    Apr 7 '16 at 18:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.