I'm working with huge log files that accumulate over days that I can't truncate/rotate but need to parse new entries hourly.
I've been using
grep to grab entries with a specific string then counting how many I get and tossing the first
N is the number of entries
I've already ingested on all prior loops, but of course this means inefficiently grepping the whole file every loop. I'm relatively unix naive, but I feel like there's a more efficient way to do this? I don't think
tail would work because I won't know how many new lines have been written since the last parsing. This post talks of skipping, but using a search string to determine how many lines to skip whereas I'd be looking to supply the skip number as an argument. This one speaks to skipping a specified number of characters on each line, but I'd be looking to skip a specified number of lines.