I need to code a daemon that track the content of log-files on multiple hosts. The "log drives" (hosts are Windows Servers) are all mounted in a single Linux host.

The tricky part is that the log-files will come (when uploaded by client of servers) and go (when archived), and don't have fixed names. They all have the naming pattern "app.DATE.log.N", and the directories are fixed (list of servers is fixed).

So I need a "tail -f" that takes a "pattern" instead of an actual name, and check regularly (every few seconds; as a near real-time monitoring) if new files have been created with this pattern, and can handle files being delete too (they are moved and compressed after 24 hours).

Ideally, I would just use existing unix tools for the "tracking", so I can code a simple daemon that reads from stdin. And the solution should be robust enough to deal with IO errors, since we are talking about drives mounted over the network.

I've found that there are tools like "lnav" and "multitail", but both seem designed for interactive use, which is not what I need.

  • 1
    You might be able to bodge something together using incron (which uses the kernel's inotify API to watch for selectable filesystem events, like files being created or deleted), spawning your own watcher process from there. – n.st Nov 29 '18 at 14:01

I'd use syslog (e.g. rsyslog or syslog-ng) and logrotate to achieve what you want. Here's a tutorial to give you the full picture.

  • 1
    Hello and welcome to the U&L stack exchange site! Please review the Help Center to get information on how to best post to this site. To get to your answer, please edit your post to include additional context While those links may provide a solution it is best to show the relevant steps here on this site as the links may change in the future. Thank you! – kemotep Nov 29 '18 at 14:33
  • rsyslog seems to be able to do almost anything, log-related (never heard of it before, but I'm a dev, not a sysadmin). I think this will do nicely. – Sebastien Diot Nov 29 '18 at 14:40
  • You may want to explore logstash, fluentd, filebeat... There's plenty of logging utilities available that may already do what you want to achieve. – Stefaan Ghysels Nov 29 '18 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.