Restrict log file size

Question

I'm trying to limit the number of lines in my log file using:

php $PWD/private_html/run_worker.php | head -n 20000 >> $PWD/private_html/data/logfile.txt 2>&1 &

If I run the command without the "| head -n 20000", my output is being redirected. However, when I include the limiting code, nothing gets written to the log.

Any help would be appreciated.

Answer 1

Suffering from Buffering

First off, there may be buffering; by default output to not-terminal-not-standard-error is block buffered (per setvbuf(3)) so one should not expect to see any logs until after a buffer is filled (or not at all when something crashes or eats a fatal signal). This is best tested with two terminals, one to run the commands, and the other to tail -f the output file:

# terminal watcher (how do the log lines appear in the logfile?)
rm logf; touch logf; tail -f logf

# terminal test commands (feed logfile)
php -r 'while(1){echo "logline\n";sleep(1);}' >> logf
...
^C
perl -E 'while(1){say "logline";sleep 1}' >> logf
...
^C

From this it appears (at least on my system) that PHP does not block buffer standard out, a departure from the default; however, head does buffer, and stdbuf does not help, as the log lines all appear in a clump:

php -r 'while(1){echo "logline\n";sleep(1);}' | head -7 >> logf
php -r 'while(1){echo "logline\n";sleep(1);}' | stdbuf -o 0 -- head -7 >> logf

So head is probably not the right tool for this task. (As a side-note, various applications will buffer internally, so whatever horrible things stdbuf is doing may not help:

stdbuf -o 0 perl -E 'while(1){say "logline";sleep 1}' >> logf
perl -E 'STDOUT->autoflush;while(1){say "logline";sleep 1}' >> logf

If at all possible, set the appropriate buffering in the application itself.)

As to the Logs

A hard limit on log lines would require something that consumes N lines, but then stays open (or does the application then get killed, or...?):

php -r 'while(1){echo "logline\n";sleep(1);}' 2>&1 \
| perl -ne 'BEGIN{STDOUT->autoflush; $N=7 } if($N>0){print;$N--}' >> logf

Or this could be done internally with PHP, assuming all the logs can be routed through a function to handle this task, or a logging module or framework. Otherwise, there are a bunch of tools that log rotate standard out, with varying levels of options and learning curves involved, though these appear to lack "N lines" and instead set caps based on size.