3

I've a Tomcat server where logs are stored as catalina.out. It's a pretty huge file and causes troubles a lot of times.

Because of it taking too much space, other servers have to stop involunatarily. I want at least a duct tape fix to this issue, so that I don't have to bother with tickets that are just deleting catalina.out and restarting the server.

I am not sure, but its 1 day logs are over 5 GB. So, say I want to make a cron job that deletes files older than a day or the last 5 GB from that file. What command would work here?

Is it recommended to solve this problem this way? Or are there better approaches?

8
  • 8
    The first point to address is to investigate why your Catalina log file is growing so rapidly. Are you running with debug enabled in a production environment, for example? Aug 17 at 6:07
  • 1
    Or you can modify log parameters to automatically rotate log files after, for example, a day, or after exceeded some size, say, 1GB. But it is depend on which logging framework are you using.
    – Jan Marek
    Aug 17 at 7:02
  • 1
    Don't run in debug mode on a production server. If you're simply throwing away the log file data then you don't need it, so don't generate it Aug 17 at 7:43
  • 1
    "It's very funny and unorganized" - this could be your opportunity to recommend professional best practice Aug 17 at 7:57
  • 1
    @roaima I'm seeking for best practices. Currently I can see logrotate. 30% of our tickets will reduce if we follow best practices.
    – achhainsan
    Aug 17 at 7:59

3 Answers 3

5

catalina.out should be mostly empty, meaning the application intercepts all errors and handles them itself, possibly logging them elsewhere as configured. But more often than not, nothing such is done and catalina.out is considered as the application logs. This causes problems because by default Tomcat doesn't rotate this file.

The better approach to truncating the file (and beside fixing the application) is to have rotation in place for catalina.out. This might already be provided by the specific packaged installation (eg: CentOS7's tomcat 7 is/was shipped with proper catalina.out log rotation in place), or with a recent enough version of tomcat as described in Bug 64430 whose fix has been integrated in newer Tomcat and backported to Tomcat >= 7.0.105, >= 8.5.56 and >= 9.0.36:

Example (all one line)

CATALINA_OUT_CMD="/usr/bin/rotatelogs -f $CATALINA_BASE/logs/catalina.out.%Y-%m-%d.log 86400"

Rotated files should be further handled with logrotate or an equivalent specialized tool.

Now to strictly answer the question, or at least the part about handling the single catalina.out file. OP writes both:

How to remove everything except last n bytes from the file in Linux?

=> keep the end

deletes [...] the last 5GB from that file

=> keep the start

Keep the start

Thus removing the end. Even if removing the end of a log rather than the start of a log is a bad idea:

POSIX truncate(2):

NAME

truncate - truncate a file to a specified length

[...]

DESCRIPTION

The truncate() function shall cause the regular file named by path to have a size which shall be equal to length bytes.

Which has its truncate(1) command implementation. For GNU truncate (along GNU stat(1)) that would be used like this in shell (one has to check the file is greater than 5GiB first or its size would be increased instead). Keeping 5GiB:

if [ $(stat -c %s catalina.out) -gt $((5*1024*1024*1024)) ]; then
    truncate -s $((5*1024*1024*1024)) catalina.out
fi

Keep the end

Thus deleting the start of the file. It's possible on Linux and adequate filesystem (eg: Ext4, XFS ...) to do this without involving any copy of data, using fallocate(1) (with additional Linux-only features):

-p, --punch-hole

Deallocates space (i.e., creates a hole) in the byte range starting at offset and continuing for length bytes.

This doesn't involve any copy of data, but requires this to be block-aligned (so usually 4096 bytes-aligned). One can always keep the end of the file, a bit like a ring buffer, without any copy cost. This could be used like this (extra computation is needed to align to a 4096 multiple):


oldsize=$(stat -c %s catalina.out) || exit 1

if [ $oldsize -gt $((5*1024*1024*1024)) ]; then
    holesize=$(( (oldsize-5*1024*1024*1024)/4096*4096 ))
    fallocate --punch-hole --length "$holesize" catalina.out
fi

--punch-hole (freeing the disk space by making the file sparse and keeping its apparent size) is used rather than --collapse-range ("moving" remaining data to the start reducing the size) to avoid disruption of the ongoing output appended to catalina.out by Tomcat, else Tomcat should be stopped before and restarted after.

2
  • will take me few weeks to understand that bash script. lots of new commands i've not studied yet. looks like dreaded c programming to me :)
    – achhainsan
    Aug 17 at 7:35
  • 1
    I was worried that the numbers could approach bash arithmetic limits, but it's fine : superuser.com/a/1030129/174998 Aug 17 at 8:19
2
tail catalina.out > catalina.new
rm catalina.out
mv catalina.new catalina.out

Then you have to restart or reload tomcat to append logs to new file (it still has opened file descriptor of old catalina.out file, which are now removed).

tail (from coreutils) has these parameters:

-c, --bytes=[+]NUM output the last NUM bytes; or use -c +NUM to output starting with byte NUM of each file

-n, --lines=[+]NUM output the last NUM lines, instead of the last 10; or use -n +NUM to output starting with line NUM

For other parameters, please, consult man tail.

2
  • 2
    Considering that logrotate is very common in today's Linux, when not use that to limit the file space usage? Also when considering that the process will continue to write to the file being opened, why not mv catalina.out catalina.old && tail catalina.old > catalina.out (The process will continue to write to catalina.old until restarted)?
    – U. Windl
    Aug 17 at 7:05
  • I use > catalina.out To empty the logs, was looking for something better. Thank you
    – achhainsan
    Aug 17 at 7:15
2

Instead of treating the symptom, treat the disease.

As others have mentioned, "proper logging" will save you from this. But proper logging involves the development team, a develop-test-release cycle, etc. and you probably want to fix your problem now.

The reason catalina.out is filling-up is because the application is writing to stdout, probably with System.out.println or similar. That's not great and should be fixed over time.

Luckily, Tomcat has a band-aid for you. All you have to do is edit the file META-INF/context.xml in the application (or, wherever your <Context> is defined, possibly in conf/server.xml, or conf/[engine]/[host]/[app].xml) and add this to your <Context>:

swallowOutput="true"

Search https://tomcat.apache.org/tomcat-9.0-doc/config/context.html for "swallowOutput" to see the description.

That will take everything from your application and put it into the application-specific log file, which is rotatable.

Then you can just delete whole-logs after a few days, etc. instead of playing games with trying to "shorten" files.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .