I have a bunch of applications on a CentOS 6.3 box, which generate daily logs (text files with a very similar content) in this vein:
/data/logs/app/my-2014-06-29.log /data/yet_another_app/logs/catalina.20140629 /data/in_fact/there_are/around_500_such_paths/2014-06-29-1.log
I need to keep 5 years of these logs.
I currently treat them in a traditional Unix way, just compressing them in place (as
.log.bz2), but I'm unhappy they still occupy a lot of space. I feel I could save a lot of space but compacting them and then compressing (think one giant .zip that holds entire year of logs).
No "compile from source" please. If it isn't mature enough to be distributed in an .rpm then it's not for me.
Archive should allow a recursive grep (like
grep_my_zip -rli pattern /this/and/that/wildcards/2014-06-*
The archive shouldn't de-compress everything when grepping, just the files that match a wildcard.
Each week I'm putting my files into the same archive and deleting them from filesystem:
append_and_remove $( find /this/and/that -name '*.log' -type f -mtime +31 ) # weekly log cleaning job
This is not the weekly archive, I want to append to a single big yearly archive each week.
MS Windows access
Archive, while mainly operated on CentOS Linux, should also allow a single file retrieval on Windows for emergency purposes (so a squashfs-style filesystem is not a viable option).
Optional stuff (a plus, but not neccessary)
- append should be space effective - consider that previous log files are very similar to the current log files
- logrotate integration for the weekly job
- grep_my_zip that works on Windows
- archive could also be updateable (i.e. writable), not only appendable
Originally posted on this pitiful sibling