I'm running Nginx on Ubuntu with the default 'main' log format, which produces output like this: - - [11/Feb/2018:11:43:10 +0000] "GET /blog/ HTTP/1.1" 200 4438 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" "-"

I have one master log file that never rotates, which I use with GoAccess (log parsing/reporting software). I wish to delete lines in that file with log entries older than 30 days or so. Can this be done, preferably with a bash one-liner?

I plan to add this to an existing daily cronjob to produce a rolling 30 day report. I'm hoping to use something like this, but I can't quite get it to parse the logs correctly: sed -i '/<magical-invocation-goes-here> --date="-30 days"/d' example.log

  • why not just rotate your log file? part of the reason for log rotation is so you don't have to do crazy and error-prone things like deleting lines from the beginning of a log that is continually being appended to (which risks losing recently appended lines). – cas Feb 12 '18 at 2:16
  • "why not just rotate your log file?" I do, the web server operates in the default way, with logrotate doing its thing. On top of that system, I have a cronjob running hourly export of a GoAccess HTML report, and that uses the (extra) log file being processed here. Do you have a suggestion for how to retain the last 30 days of server logs which uses a different approach? Here is an overview of what I'm doing, but I'm always happy to simplify. – Tom Brossman Feb 12 '18 at 8:14
  • Wait...so instead of updating goaccess incrementally (as discussed at github.com/allinurl/goaccess/issues/334), you're concatenating the rotated log file onto a master log file, then you want to delete all but the last 30 days from the master log, and then feed the master log file into goaccess and have it process the entire 30 days from scratch every time you run it? that's...ummm...an "interesting" way to do it. – cas Feb 12 '18 at 10:14
  • That discussion involves an optional feature (B+Tree storage) that I'm not using as the server has plenty of memory to handle the entire log. I'll need to take another look at how I am doing this and apply things I've learned since, but I think I still need to re-process the whole log every time since it's a rolling total, which incorporates some variables (referrer spam block, geoip lookups). Like I said, happy to receive suggestions here or in chat - just let me know. – Tom Brossman Feb 12 '18 at 10:24
  • Yeah, i know. i read that in the docs & the discussion. The btree storage option also allows for incremental updates. which not only uses less RAM & CPU, it runs a lot faster then re-processing the same log entries repeatedly. also eliminates the need to create this odd rolling 30-day master log file. – cas Feb 12 '18 at 10:37

GNU awk solution:

Sample test.log: - - [11/Feb/2018:11:43:10 +0000] "GET /blog/ HTTP/1.1" 200 4438 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" "-" - - [11/Aug/2017:11:43:10 +0000] "GET /blog/ HTTP/1.1" 200 4438 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" "-" - - [01/Jan/2018:11:43:10 +0000] "GET /blog/ HTTP/1.1" 200 4438 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" "-" - - [11/Feb/2018:11:43:10 +0000] "GET /blog/ HTTP/1.1" 200 4438 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" "-"

awk -v m1_ago=$(date -d"-1 month" +%s) \
     split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec", month);
     for (i in month) m_nums[month[i]] = i
 { split(substr($4,2), a, "[/:]") }
 mktime(sprintf("%d %d %d %d %d %d", a[3], m_nums[a[2]], a[1], a[4], a[5], a[6])) > m1_ago
' test.log > tmp_log && mv tmp_log test.log

Final test.log contents: - - [11/Feb/2018:11:43:10 +0000] "GET /blog/ HTTP/1.1" 200 4438 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" "-" - - [11/Feb/2018:11:43:10 +0000] "GET /blog/ HTTP/1.1" 200 4438 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" "-"
  • Thanks very much, tested and working nicely. I was able to use awk -i inplace -v... and trim the > tmp_log && mv tmp_log test.log from the end, but this answer did exactly what I wanted. – Tom Brossman Feb 12 '18 at 8:17
  • 1
    @TomBrossman, yes, since gawk 4.1.0 you may apply -i inplace. The approach tmp_log && mv tmp_log test.log is less version dependent – RomanPerekhrest Feb 12 '18 at 9:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.