1

I apologize for length of this question. It's difficult to explain, even though I was fluent in English, which I'm not ;)
Not really sure where post this question, because it has java, log4j library and linux.

Scenario:

I have several log files (created by log4j using RollingFileAppender). Named file.log and file.log.1 to file.log.10
All files are overwritten more than once a day. I mean, if I receive an incidence, I almost sure I have no logs to see what happened, because it would be overwritten.

Purpose:

My purpose is to make a backup of these files periodically, with some conditions

  • First, do not miss data (solved running a job often enough)
  • Do not repeat information. (it would be solved if information was sorted)
  • Data must be sorted. (Problem!!)

Considerations:

  • Each separate file is sorted, but not all together, because there are two server instances that write to them.
    I mean that is possible one instance write to *.log.1 and the other *.log.2 at once. So, I cannot merge them all and expect to keep them sorted.
  • I can't increase space available on file system.

Log register layout:

Each line is like this:

2014-11-28 14:33:10,015 main ca.cpy.net.txc.batch.SendEER INFO - information

Attempts:

  1. Move from RollingFileAppender to DailyRollingFileAppender as type of appender. Unfortunately, Apache documentation says

    "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss".

    So, I can't use it.

  2. Use apache log4j extras libraries, but I'm not allowed to do that. It does not depend on me.
  3. Do all stuff by myself. It consists in:

    • merge all files
    • sort them
    • discard all saved data in previous backups.
    • compress


    The problem is the sort step. This is what I tried:

    for ((i=10; i >= 1; i--)); do 
        cat file.log.$i >> $FILE_OUT;  ## put all files in one (as much sorted as possible)
    done;  
    cat file.log >> $FILE_OUT;  ## append last 
    sort -s -t ' ' -k 1.1,1.4n -k 1.6,1.7n -k 1.9,1.10n -k 2.1,2.2n -k 2.4,2.5n -k 2.7,2.8n -k 2.10,2.12n -3k $FILE_OUT -o $FILE_SORTED # Sort by date/time
    


    Well, this would work if each register appended to log had one only line (ie: no end-of-line character \n). For example, such sort command above would break a register like this:

    2014-11-28 14:33:10,015 main ca.cpy.net.txc.batch.SendEER INFO  - 
     ***** RESULTATS ENVIAMENT EXPEDIENT ***** 
        Total documents a tractar en DB: 86
     ***************************************** 
    

    It would sort just first line, and the other three would be put at the beginning of the output file.

Is there a way to sort merged files without breaking each register that contains more than one line? Any other idea will be very welcome, too.

  • 1
    I'd like to point out that dates of the form YYYYdMMdDD can be sorted lexicographically (d being some delimiter), so you can sort directly on the first key. The same holds for HHdMMdSS, if you use 24-hour format. (so -k1 -k2.1,2.8). Can you post a few files (at some pastebin, or on Github) with scrubbed example data, so I can do some trials? Also, if you're on bash: sort .... file.log.{10..1} file.log -o $FILE_SORTED – muru Dec 3 '14 at 14:03
  • Is number of lines in each block of text fixed? – jimmij Dec 3 '14 at 14:08
  • @muru, I will, but I have to hide some information (company policy). It'll take me some time after lunch ;) – Albert Dec 3 '14 at 14:16
  • @jimmij: No, it is not fixed. It can contain a summary of some process, an XML, a response from a REST service, etc. There is not pattern. – Albert Dec 3 '14 at 14:17
  • It probably mixes the logs together and may not work correctly, but you can try to run parallel tail -f for each logfile and append their output to the same output file. – jofel Dec 3 '14 at 16:19
1

Here's one way:

Make the log files NUL-delimited. That is, make every record end with a NUL (\0) character. Then you can avail of the support for NUL-delimited text found in a number of tools (sed, sort, xargs, find, etc.). One way could be to do:

perl -pe 's/^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})/\0$1/' file.log.2 > file.log.2.NULL
  • Prepend every timestamp with a \0 (This is the other way around - making records begin with NUL, but in effect...)

Then you can do:

sort -szt ' ' -k1,2 file.log{.{10..1},}.NULL -o $FILE_SORTED
  • -s is for a stable sort (so that tied entries are sorted in order of appearance)
  • -z turns on NUL-delimited text support
  • I have changed the keys, since as I noted in the comments, timestamps of the form YYYY-MM-DD HH:MM:SS,UUU are lexicographically sortable. You don't need numeric sort for them.

Or you can avoid all these temporary files altogether:

perl -pe 's/^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})/\0$1/' file.log{.{10..1},} | \    
 sort -szt ' ' -k1,2 -o $FILE_SORTED
  • At first sight, it looks wonderful. I still have to do some tests and finish the script. I'll let you know. – Albert Dec 3 '14 at 17:57
  • One question: The best way to restore, could be cat $FILE_SORTED | tr -d '\0'? I'm quite sure there is not other '\0' character elsewhere, but just in case: Is there a way to force that only replace '\0' at the beginning of each line? I'm not able to do it with sed. – Albert Dec 3 '14 at 18:15
  • Put yet another perl at the end of the pipeline: perl ... | sort ... | perl -pe 's/^\0//' > $FILE_SORTED, and skip the -o $FILE.. for sort. – muru Dec 3 '14 at 18:15
  • Of course, what was I thinking! Don't know why I was fighting with $1, $2 args to print the rest of each line... I must be tired xD. Thanks. – Albert Dec 3 '14 at 18:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.