2

I have a bunch of applications on a CentOS 6.3 box, which generate daily logs (text files with a very similar content) in this vein:

/data/logs/app/my-2014-06-29.log
/data/yet_another_app/logs/catalina.20140629
/data/in_fact/there_are/around_500_such_paths/2014-06-29-1.log

I need to keep 5 years of these logs. I currently treat them in a traditional Unix way, just compressing them in place (as .log.bz2), but I'm unhappy they still occupy a lot of space. I feel I could save a lot of space but compacting them and then compressing (think one giant .zip that holds entire year of logs).

Requirements:

RHEL kosher

No "compile from source" please. If it isn't mature enough to be distributed in an .rpm then it's not for me.

grep

Archive should allow a recursive grep (like grep -rli):

grep_my_zip  -rli  pattern   /this/and/that/wildcards/2014-06-*

The archive shouldn't de-compress everything when grepping, just the files that match a wildcard.

append

Each week I'm putting my files into the same archive and deleting them from filesystem:

append_and_remove   $( find /this/and/that  -name '*.log' -type f -mtime +31 )     # weekly log cleaning job

This is not the weekly archive, I want to append to a single big yearly archive each week.

MS Windows access

Archive, while mainly operated on CentOS Linux, should also allow a single file retrieval on Windows for emergency purposes (so a squashfs-style filesystem is not a viable option).

Optional stuff (a plus, but not neccessary)

  • append should be space effective - consider that previous log files are very similar to the current log files
  • logrotate integration for the weekly job
  • grep_my_zip that works on Windows
  • archive could also be updateable (i.e. writable), not only appendable

Originally posted on this pitiful sibling

1

My research on various compressors show that space saving is not worth going into the non-traditional path.

  • 8800M unpacked bi-weekly input (5842 files)
  • 270M just bzip2 (traditional logrotate style - separate .log.bz2 files)
  • 257M tar bzip2
  • 400M zip
  • 132M p7zip (Windows .7z compatible)
  • 140M tar xz
  • 129M tar lrzip

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.