I am trying to break a large log file into smaller files based on date.

The file is of the following form, where some lines may not have a date. Those lines should be included with the previous dated line.

2014-04-07T23:59:58 CheckForCallAction [ERROR] Exception caught
Undated line 1
Undated line 2
2014-04-08T00:00:03 MobileAppRequestFilter [DEBUG] Action
undated line 3
2015-04-08T00:00:03 MobileAppRequestFilter [DEBUG] ActionB

I found How to extract logs between two time stamps which is close to what I want, except my log file does not include a "[" at the start of the date, or "]" and the end of the date.

The command from that link is:

awk -F'[[]|[]]' \
  '$0 ~ /^\[/ && $2 >= "2014-04-07 23:00" { p=1 }
   $0 ~ /^\[/ && $2 >= "2014-04-08 02:00" { p=0 }
   p { print $0 }' > test1.log  logwith[.log

I have been trying for several days to modify this, but I just can't seem to get it.

A desired enhancement would be to not have to specify a start and end date, but rather automatically name the output files by either year, or year-month.

  • Please see formatting tools for help on formatting your posts.
    – terdon
    Apr 7, 2016 at 14:47
  • There is an example extraction for data "without [] braces" in the answer you referenced: unix.stackexchange.com/a/123983/4252
    – KM.
    Apr 7, 2016 at 14:52
  • The length of the date and time strings are looking very uniform to me. You can extract both with a simple cut command or $1 in awk replaces the actual date and $2 the actual time. In short, you don't need square brackets. The post you linked is a common case but not necessarily the norm.
    – MelBurslan
    Apr 7, 2016 at 14:52
  • @KM, the only compact example at unix.stackexchange.com/a/123983/4252 does not include any line that do not include a date. I need those lines.
    – Mike
    Apr 7, 2016 at 16:33
  • @MelBursian Thanks for the response. I am not knowledgeable enough to be able to convert your guidance into a workable command. Everything I tried had $1 either a date, or for lines without a date, a string. That made me lost as to how to proceed.
    – Mike
    Apr 7, 2016 at 16:55

1 Answer 1


Use T as the field delimiter and check for date-like strings explicitly. For example, to split by year:

awk -FT '($1~/^[0-9]+-[0-9]+-[0-9]+$/){d=substr($1,1,4)}{print > d".log"}' logfile 

And by year+month:

awk -FT '($1~/^[0-9]+-[0-9]+-[0-9]+$/){split($1,d,"-")}{print > d[1]d[2]".log"}' logfile 

Here, we check that the first field (defined by T, so the whole date on lines starting with dates, that's what -FT means) is a set of 3 numbers separated by -. If it is, to get the year, we extract the first 4 characters (d=substr($1,1,4)) and, to get the month, we split the 1st field on -, saving the resulting strings in the array d (split($1,d,"-")), and use the 1st two elements of the array (d[1]d[2]) for the file name.

  • Thank you. This is so close to what I am looking for. I need to be able to break up the original file either by year, or by year-month. All variations I tired do not include the non-dated lines for some reason. Can you tweak this?
    – Mike
    Apr 7, 2016 at 16:52
  • @Mike please edit your question and explain exactly what you need. Let me know when you've done so and I'll give be it a try.
    – terdon
    Apr 7, 2016 at 16:54
  • I thought I had the requirements in the question "A desired enhancement would be to not have to specify a start and end date, but rather automatically name the output files by either year, or year-month." If that is not clear, let me know and I will try a different way. What you provided is exactly what I wanted, only that I wanted either a yearly or monthly file, not a daily one.
    – Mike
    Apr 7, 2016 at 17:23
  • @Mike indeed you did, sorry, I missed that. I'm afraid I don't have the time to do this right now, but I'll answer tomorrow. The basic nidea will probably be just taking - as the field delimiter.
    – terdon
    Apr 7, 2016 at 17:27
  • I will appreciate that. The variations I tried do not include the lines without dates, which is needed. Looking forward to your resolution.
    – Mike
    Apr 7, 2016 at 18:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .