I have a bunch of applications on a CentOS 6.3 box, which generate daily logs (text files with a very similar content) in this vein:
/data/logs/app/my-2014-06-29.log
/data/yet_another_app/logs/catalina.20140629
/data/in_fact/there_are/around_500_such_paths/2014-06-29-1.log
I need to keep 5 years of these logs.
I currently treat them in a traditional Unix way, just compressing them in place (as .log.bz2), but I'm unhappy they still occupy a lot of space. I feel I could save a lot of space but compacting them and then compressing (think one giant .zip that holds entire year of logs).
Requirements:
RHEL kosher
No "compile from source" please. If it isn't mature enough to be distributed in an .rpm then it's not for me.
grep
Archive should allow a recursive grep (like grep -rli):
grep_my_zip -rli pattern /this/and/that/wildcards/2014-06-*
The archive shouldn't de-compress everything when grepping, just the files that match a wildcard.
append
Each week I'm putting my files into the same archive and deleting them from filesystem:
append_and_remove $( find /this/and/that -name '*.log' -type f -mtime +31 ) # weekly log cleaning job
This is not the weekly archive, I want to append to a single big yearly archive each week.
MS Windows access
Archive, while mainly operated on CentOS Linux, should also allow a single file retrieval on Windows for emergency purposes (so a squashfs-style filesystem is not a viable option).
Optional stuff (a plus, but not neccessary)
- append should be space effective - consider that previous log files are very similar to the current log files
- logrotate integration for the weekly job
- grep_my_zip that works on Windows
- archive could also be updateable (i.e. writable), not only appendable
Originally posted on this pitiful sibling
