a script to log […] every 5 seconds […] I need timestamps […] a loop […] the file gets to a certain size
You want a tool that takes as its input the log output of your main program, and that writes it to a size-capped log file, and adds timestamps. Tools exist that do this and more. The "more" that they do is automatic rotation of the log file that is also triggerable on demand, maintaining a size-capped log directory of current and old log files.
You have a choice of tools:
- Dan Bernstein's
multilog from daemontools
- Bruce Guenter's
multilog from daemontools-encore
- Laurent Bercot's
s6-log from s6
- Gerrit Pape's
svlogd from runit
- Wayne Marshall's
tinylog from perp
cyclog from nosh
Assume a long-running utility that prints your desired log output every few seconds, something like:
done | grep --line-buffered -- '^Core'
Usage is as simple as:
monitor-sensors | cyclog ./temps
monitor-sensors | multilog t ./temps
and can be easily tweaked:
monitor-sensors | cyclog --max-file-size 32768 --max-total-size 1048576 ./temps
monitor-sensors | s6-log t s32768 S1048576 ./temps
monitor-sensors | multilog t s32768 n5 ./temps
From here, using toolsets like daemontools-encore/runit/perp/s6/nosh, it is but a small step to moving the left and right hand sides of this pipe into
run programs and running this as a linked pair of actual dæmons.
But they can all, also, handle being spun up for one-off output to existing log directories, if you just want to (say) perform one-off invocations from a command line.
Some of these tools can do other forms of timestamps, but they can all do TAI64N timestamps. TAI64N timestamps are designed to be capable of nanosecond precision, albeit that some of the aforementioned do not quite implement this; are designed to quite happily cope with things like leap seconds, as long as your TZ database knows about them; and are trivial to
sort, or even
Convert from TAI64N timestamps to your current timezone's local time (or, given that it's just the
TZ environment variable, an arbitrary timezone of your choosing) using tools such as:
Watch such logs as they are written with:
tail -F, albeit that
tail has known problems when there are very speedy rotations. (This is one of several known problems with
tail. Other known problems including handling in-place truncation of log files that are rapidly followed by more log data. This problem can be triggered by inferior systems such as
logrotate. Fortunately, the aforementioned tools do not truncate files once they are fully written and do not risk such additional
follow-log-directories from nosh, which "knows" this kind of log directory and uses a "cursor" (persistently maintained on disc) to reliably track the position in the log directory to continue from even if several rotations happen when the log follower isn't looking.
Other sorts of processing can be done with tools such as: