1

We have a system we're trying to fault test. The scenario I'm running involves simulating a full disk (our attached logger disk to be specific). I've filled the disk using df -h to see how much space is left (so assume 4G for this example) and then sudo fallocate -l 10G /opt/var/big.file to generate a file up to the limit.

When I do df I do see that Use% is 100% and Available is 0. However, if I go to my log directory I see things like:

-rw-r--r-- 1 0 0  28M Nov  7 17:43 service.log
-rw-r--r-- 1 0 0 4.0K Nov  7 17:43 service.log-2017-11-07-17-13.gz
-rw-r--r-- 1 0 0 1.2K Nov  7 17:43 service.log-2017-11-07-17-14.gz
-rw-r--r-- 1 0 0  27M Nov  7 17:44 service.log-2017-11-07-17-15
-rw-r--r-- 1 0 0    0 Nov  7 17:44 service.log-2017-11-07-17-15.gz
-rw-r--r-- 1 0 0 1.3K Nov  7 17:40 service.log-2017-11-07-17-1.gz
-rw-r--r-- 1 0 0  28M Nov  7 17:41 service.log-2017-11-07-17-2
-rw-r--r-- 1 0 0 4.0K Nov  7 17:41 service.log-2017-11-07-17-3

If I tail -f service.log I can see that data is being appended. I can also see that my log4j2 settings are rotating files appropriately (although several files show as unzipped until sometime later). Once a file finishes zipping up, logs are appended to the end of the active log file. It's like we're writing to the log so fast (DEBUG logging is turned on, which write about 40M of log per second, and the rotate max size is set to 200M) that it can't fully zip the files before getting requested to zip another file... or something like that.

I'm not sure what to make of this - could someone help shed some light?

5

By default 5% of the volume space is reserved for root on format. There is no more user space on the volume, but the logs are being written by root.

  • Can I force this 5% to be simulated as full somehow? – MrDuk Nov 7 '17 at 17:58
  • Fill it as root. – Ignacio Vazquez-Abrams Nov 7 '17 at 17:58
  • I did - I'll update my command above, but it was ran as sudo – MrDuk Nov 7 '17 at 17:59
  • You didn't fill it enough. – Ignacio Vazquez-Abrams Nov 7 '17 at 18:00
  • I'm not super convinced yet. It's a 100G disk, it had 4G free (reported), I attempted to allocate a 10G file (which should have allocated the 4G + 5% (5G) from room right?). What am I missing? – MrDuk Nov 7 '17 at 18:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.