I apologize for length of this question. It's difficult to explain, even though I was fluent in English, which I'm not ;)
Not really sure where post this question, because it has java, log4j library and linux.
I have several log files (created by log4j using
RollingFileAppender). Named file.log and file.log.1 to file.log.10
All files are overwritten more than once a day. I mean, if I receive an incidence, I almost sure I have no logs to see what happened, because it would be overwritten.
My purpose is to make a backup of these files periodically, with some conditions
- First, do not miss data (solved running a job often enough)
- Do not repeat information. (it would be solved if information was sorted)
- Data must be sorted. (Problem!!)
- Each separate file is sorted, but not all together, because there are two server instances that write to them.
I mean that is possible one instance write to *.log.1 and the other *.log.2 at once. So, I cannot merge them all and expect to keep them sorted.
- I can't increase space available on file system.
Log register layout:
Each line is like this:
2014-11-28 14:33:10,015 main ca.cpy.net.txc.batch.SendEER INFO - information
DailyRollingFileAppenderas type of appender. Unfortunately, Apache documentation says
"DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss".
So, I can't use it.
- Use apache log4j extras libraries, but I'm not allowed to do that. It does not depend on me.
Do all stuff by myself. It consists in:
- merge all files
- sort them
- discard all saved data in previous backups.
The problem is the sort step. This is what I tried:
for ((i=10; i >= 1; i--)); do cat file.log.$i >> $FILE_OUT; ## put all files in one (as much sorted as possible) done; cat file.log >> $FILE_OUT; ## append last sort -s -t ' ' -k 1.1,1.4n -k 1.6,1.7n -k 1.9,1.10n -k 2.1,2.2n -k 2.4,2.5n -k 2.7,2.8n -k 2.10,2.12n -3k $FILE_OUT -o $FILE_SORTED # Sort by date/time
Well, this would work if each register appended to log had one only line (ie: no end-of-line character \n). For example, such sort command above would break a register like this:
2014-11-28 14:33:10,015 main ca.cpy.net.txc.batch.SendEER INFO - ***** RESULTATS ENVIAMENT EXPEDIENT ***** Total documents a tractar en DB: 86 *****************************************
It would sort just first line, and the other three would be put at the beginning of the output file.
Is there a way to sort merged files without breaking each register that contains more than one line? Any other idea will be very welcome, too.