How to continuously take backup of log files?

Question

I have some applications in Lunux which creates Log files of around 41 MB each. The log files switches from log.1 to log.2 and starts overwriting log.2 once this 41MB is completed in log.1 file.

I want to take the backup of these files in the way that whenever the new file log.2 reaches the size more than 1MB it take backup (gzip and store at another location) of other file (log.1).

I have been using a script which takes the backup of file whenever it reaches 41MB but the trouble here is sometimes this 41MB file backup is take by script, still the data is getting written on the same file at a very huge rate because of which it loses some seconds of log data which is the reason i am trying the alternate way of making backup of other file once the new file reaches 1MB or something so that i am sure that the other file is finished writing.

Any idea how do i do that?

Answer 1

This can be done through logrotate , see below example for apache logs

Step 1# First make sure logrotate is installed in your system, by default is available on almost all distro of Linux

Step 2# Create /etc/logrotate_apache.conf as below

#-------------------------------------------------------------------------------------
/var/log/httpd/*log {
    missingok
    minsize 1M
    create
    compress
    rotate 1
    lastaction
        /sbin/service httpd reload > /dev/null 2>/dev/null || true
        Log_dir="/var/log/httpd/Apache_$(date +%F)/"
        # move to other location
        [[ ! -d "${Log_dir}" ]] && /bin/mkdir "${Log_dir}" &&  /bin/mv /var/log/httpd/*.gz "${Log_dir}"
    endscript
}

# system-specific logs may be also be configured here.
#-------------------------------------------------------------------------------------

Step 3# Set Cron task as below

00 * * * * /usr/sbin/logrotate -f /etc/logrotate_apache.conf

Answer 2

You can try logrotate, Some Examples are mentioned here

The size option will let you rotate the log once it reaches a specified size, say 41MB.
Use the postrotate option, to execute a custom script which you can use to transfer the rotated log file to another location.