We have a system we're trying to fault test. The scenario I'm running involves simulating a full disk (our attached logger disk to be specific). I've filled the disk using
df -h to see how much space is left (so assume 4G for this example) and then
sudo fallocate -l 10G /opt/var/big.file to generate a file up to the limit.
When I do
df I do see that
Use% is 100% and
Available is 0. However, if I go to my log directory I see things like:
-rw-r--r-- 1 0 0 28M Nov 7 17:43 service.log -rw-r--r-- 1 0 0 4.0K Nov 7 17:43 service.log-2017-11-07-17-13.gz -rw-r--r-- 1 0 0 1.2K Nov 7 17:43 service.log-2017-11-07-17-14.gz -rw-r--r-- 1 0 0 27M Nov 7 17:44 service.log-2017-11-07-17-15 -rw-r--r-- 1 0 0 0 Nov 7 17:44 service.log-2017-11-07-17-15.gz -rw-r--r-- 1 0 0 1.3K Nov 7 17:40 service.log-2017-11-07-17-1.gz -rw-r--r-- 1 0 0 28M Nov 7 17:41 service.log-2017-11-07-17-2 -rw-r--r-- 1 0 0 4.0K Nov 7 17:41 service.log-2017-11-07-17-3
tail -f service.log I can see that data is being appended. I can also see that my
log4j2 settings are rotating files appropriately (although several files show as unzipped until sometime later). Once a file finishes zipping up, logs are appended to the end of the active log file. It's like we're writing to the log so fast (DEBUG logging is turned on, which write about 40M of log per second, and the rotate max size is set to 200M) that it can't fully zip the files before getting requested to zip another file... or something like that.
I'm not sure what to make of this - could someone help shed some light?