We have a java based web-app running on CentOS 6.0 64bit and it writes out many log files. Some of the files are rotated on daily basis using logrotate using the daily, copytruncate and other few options.

One such particular log file stops growing at exactly "102400000" bytes. The process continues working fine. But nothing gets written in the log then.

I did a stat on the file using watch to see if after reaching the "102400000" byte size the modified time changes, but it doesn't. Nothing happens to the file. It just stays there as it is. Other log files in the same directory continue to be written into.

The things mentioned in another StackExchange question here do not apply. I checked and rechecked if any where the mentioned ulimit option was set, but there is none any.

Doing strace -f -p PID showed the process to be trying to write but encountering the "File too large" and "File size limit exceeded" messages:

[pid  5679] write(1, "\n\n\n\n\n23-01-2013 12:00:46:921  Av"..., 128) = -1 EFBIG (File too large)
[pid  5679] --- SIGXFSZ (File size limit exceeded) @ 0 (0) ---

How do I find as to what is imposing this limit on the file size. Other log files in the same directory are quite large (greater than 1Gb in size) and have no issues as such.

  • Is this web-app home-brewn or something publicly available? (Maybe with a mailing list, documentation) – sr_ Jan 22 '13 at 9:28
  • @sr_ Its home brewn, but this problem is occurring with only two particular instances of deployment. And with same file. At other installations there is no problem. – Gautam Somani Jan 22 '13 at 9:34
  • 2
    If it was a ulimit thing, the application would get killed (unless it ignores the SIGXFSZ signal). More probably it's the application itself or java that sets the limit. You could strace -f it to see if it is attempting to write the log file. – Stéphane Chazelas Jan 22 '13 at 10:11
  • Presumably the application log doesn't show anything of interest? – Reed Kraft-Murphy Jan 22 '13 at 10:24
  • @ReedKraft-Murphy The log is important. It is monitored from time to time. Once it reaches the 102400000 size, I'm moving it myself, though I can use logrotate also, but I want to know the reason first. I believe rarely does one faces such issues, and I can learn a lot from it. – Gautam Somani Jan 22 '13 at 11:11

So (from your comment), there is a ulimit after all (causing the SIGXFSZ and EFBIG) which explains why the size of the log can't get any larger.

You can confirm with:

grep 'Max file size' "/proc/$pid/limits"

(where $pid is the id of the java process meant to write to that file).

Possibly it is set by the process itself (check for setrlimit or ulimit in the strace output), It could also be set by a wrapper script that starts java or the java application (look for ulimit -f 100000).

  • There is a ulimit 100000 option set in the "limits" file under "/proc". But the script that is starting is using the command ulimt 100000 only. There is no -f option being used. Also in the strace output, taken for 2 minutes, there is no mention of ulimit or setrlimit. Do I take more strace output? Would it come eventually? – Gautam Somani Jan 23 '13 at 7:53
  • -f is the default option, so ulimit 100000 is the same as ulimit -f 100000 – Stéphane Chazelas Jan 23 '13 at 7:56
  • Oh yeah u r right! They wanted to increase the number of open files, and they missed the -n option. Thanks a lot for pointing this out. :) – Gautam Somani Jan 23 '13 at 8:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.