Filtering log statements before they're written

Question

I have an application I need to run in verbose mode to get the information I'm interested in. This produces an enormous log file that is <1% interesting for my requirements.I want some way of logging only those statements I'm interested in.

The command accepts a logfile name to which it writes its log.It's running as a systemd service.

How can I intercept and select from the log write statements before they're put to disk?

I tried giving a filename for a fifo and reading from that through a python script but that got too hairy.

Answer 1

I tried giving a filename for a fifo and reading from that through a python script but that got too hairy.

That's the least hairy option. Every reasonable approach to this involves piping the output through a stream filter tool, whether it's done manually with a Python script, or with grep, or with syslog-ng. All the filter needs to do is read a line from its input, then either write it to output or not.

It is possible to LD_PRELOAD a library that performs filtering on library functions such as fwrite() or syslog(), but it will only work with programs that use libc-provided functions; many language runtimes do not use libc fwrite() (e.g. Golang has its own stdio) and LD_PRELOAD has no effect on statically-linked executables.

And while it is possible to avoid this by using e.g. ptrace() or eBPF or even FUSE to intercept the write() syscalls at OS level, that's only more troublesome because file writes are typically buffered by the program – you don't have one write() per line, you have one write() call per 8k buffer, which will often span multiple lines, start in the middle of a line, etc. For example, a single write() call could contain seven and a half lines, so if you intercepted it you couldn't just accept/discard the entire call, you'd need to split and reassemble the buffer to retain only the lines you need, keep state across calls to handle split lines, all such things Python did for you.

(Which means, even though systemd does use seccomp and eBPF for some things, you will not find a systemd parameter to magically filter out some data from file – not stdout/stderr, but file that the program itself opened – writes that systemd has absolutely no involvement in.)