We have a long running process1 whose output is captured into a log file based on when the process is started:

export LOG_FILE=${LOGS_DIR}/$LOGNAME/${LOGNAME}-`date +%Y%m%d-%H%M%S`.txt

Our problem is that even though most of the application logging is done through our logging framework, some things still sneak out to stdout & stderr, and when something goes wrong23, the console output log can get quite large.

All logs (both console logs, and logs from our logging framework) get written to a high performance filesystem with limited space, so we run a jenkins job to archive files which haven't been touched in a week (compressing them if necessary).

This works well for our main logging framework, which rolls over log files at midnight, before compressing the log of the previous day. Runaway logging will only have at most a days uncompressed output, and compression can be embarrassingly efficient when compressing runaway logs, so the auto archiving minimises the risk of the filesystem filling up.

The console logs work less well however, since these files may be kept around for months at a time, and only a restart of the process will cause a new file to be created, allowing the jenkins jobs to compress and archive the file a week later.

So, how can I get our console logs to split on a daily basis?

What I would like is to replace the >> ${LOG_FILE} with something like | logsplitter --append ${FILE_SPEC} without having to write (and exhaustively test) this logsplitter program/script. I'm hoping that something like this might already exist, but my google foo has so far failed me.

1 - It could run for months.
2 - Some libraries we use generate error messages directly to the console.
3 - Failures in the logging framework can result in falling back to logging to stderr.

  • Didn't the Related questions at the right of this page help you? – FedonKadifeli Jun 20 at 18:35
  • Non of the Related questions I'm shown answer my question @FedonKadifeli. Perhaps you could suggest which you think does? Alternatively, feel free to suggest how I might improve my question to clear up any confusion over what I'm asking. – Mark Booth Jun 21 at 9:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.