Sort and backup log4j files

Question

I apologize for length of this question. It's difficult to explain, even though I was fluent in English, which I'm not ;)
Not really sure where post this question, because it has java, log4j library and linux.

Scenario:

I have several log files (created by log4j using RollingFileAppender). Named file.log and file.log.1 to file.log.10
All files are overwritten more than once a day. I mean, if I receive an incidence, I almost sure I have no logs to see what happened, because it would be overwritten.

Purpose:

My purpose is to make a backup of these files periodically, with some conditions

• First, do not miss data (solved running a job often enough)
• Do not repeat information. (it would be solved if information was sorted)
• Data must be sorted. (Problem!!)

Considerations:

• Each separate file is sorted, but not all together, because there are two server instances that write to them.
  I mean that is possible one instance write to *.log.1 and the other *.log.2 at once. So, I cannot merge them all and expect to keep them sorted.
• I can't increase space available on file system.

Log register layout:

Each line is like this:

2014-11-28 14:33:10,015 main ca.cpy.net.txc.batch.SendEER INFO - information

Attempts:

1. Move from RollingFileAppender to DailyRollingFileAppender as type of appender. Unfortunately, Apache documentation says

   "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss".

   So, I can't use it.

2. Use apache log4j extras libraries, but I'm not allowed to do that. It does not depend on me.

3. Do all stuff by myself. It consists in:

   • merge all files
   • sort them
   • discard all saved data in previous backups.
   • compress

   
   

   The problem is the sort step. This is what I tried:

   for ((i=10; i >= 1; i--)); do 
       cat file.log.$i >> $FILE_OUT;  ## put all files in one (as much sorted as possible)
   done;  
   cat file.log >> $FILE_OUT;  ## append last 
   sort -s -t ' ' -k 1.1,1.4n -k 1.6,1.7n -k 1.9,1.10n -k 2.1,2.2n -k 2.4,2.5n -k 2.7,2.8n -k 2.10,2.12n -3k $FILE_OUT -o $FILE_SORTED # Sort by date/time
   

   
   Well, this would work if each register appended to log had one only line (ie: no end-of-line character \n). For example, such sort command above would break a register like this:

   2014-11-28 14:33:10,015 main ca.cpy.net.txc.batch.SendEER INFO  - 
    ***** RESULTATS ENVIAMENT EXPEDIENT ***** 
       Total documents a tractar en DB: 86
    ***************************************** 
   

   It would sort just first line, and the other three would be put at the beginning of the output file.

Is there a way to sort merged files without breaking each register that contains more than one line? Any other idea will be very welcome, too.

Answer 1

Here's one way:

Make the log files NUL-delimited. That is, make every record end with a NUL (\0) character. Then you can avail of the support for NUL-delimited text found in a number of tools (sed, sort, xargs, find, etc.). One way could be to do:

perl -pe 's/^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})/\0$1/' file.log.2 > file.log.2.NULL

• Prepend every timestamp with a \0 (This is the other way around - making records begin with NUL, but in effect...)

Then you can do:

sort -szt ' ' -k1,2 file.log{.{10..1},}.NULL -o $FILE_SORTED

• -s is for a stable sort (so that tied entries are sorted in order of appearance)
• -z turns on NUL-delimited text support
• I have changed the keys, since as I noted in the comments, timestamps of the form YYYY-MM-DD HH:MM:SS,UUU are lexicographically sortable. You don't need numeric sort for them.

Or you can avoid all these temporary files altogether:

perl -pe 's/^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})/\0$1/' file.log{.{10..1},} | \    
 sort -szt ' ' -k1,2 -o $FILE_SORTED