9

systemd appears to split very long log lines in multiple log messages:

$ journalctl -u myunit
Nov 12 08:00:18 ovh7 uwsgi[32441]: SHORT LINE
Nov 12 08:00:18 ovh7 uwsgi[32441]: START of VERY VERY LONG LINE ON STDOUT
Nov 12 08:00:18 ovh7 uwsgi[32441]: CONTINUE VERY VERY LONG LINE
Nov 12 08:00:18 ovh7 uwsgi[32441]: SHORT LINE

Now, I do not really mind but I need to join the separate log messages to get back the original stdout from my process. I thought I could merely comb through the logs for the pid and join the MESSAGE strings using the journalctl json output:

def main():
    import optparse, json, sys
    parser = optparse.OptionParser()
    parser.add_option('--pid')
    parser.add_option('-f', '--file')
    options, args = parser.parse_args()

    with open(options.file, 'r') as f:
        for line in f:
            d = json.loads(line)
            if d['_PID'] == options.pid:
                sys.stdout.write(d['MESSAGE'].encode("utf-8"))

Sadly, the above does not work because systemd also appears to trim the trailing \n from stdout. It generates this:

SHORT LINE START of VERY VERY LONG LINE ON STDOUT CONTINUE VERY VERY LONG LINE SHORT LINE

Now, I can try to add an extra \n for each MESSAGE but this generates this:

SHORT LINE
START of VERY VERY LONG LINE ON STDOUT
CONTINUE VERY VERY LONG LINE
SHORT LINE

None of the two above results are very helpful. I need this:

SHORT LINE
START of VERY VERY LONG LINE ON STDOUT CONTINUE VERY VERY LONG LINE
SHORT LINE

And I see nothing in the output of journalctl that would allow me to infer that two consecutive messages are coming from the same line in the original stdout output. Any idea that would allow me to reconstruct this data correctly without having to generate a separate log file for my program's stdout ?

2
  • 1
    One heuristic is to concatenate when a message length is 2048 bytes (or a bit less as whitespace is removed).
    – meuh
    Nov 12, 2016 at 15:33
  • good point meuh
    – mathieu
    Nov 12, 2016 at 17:47

1 Answer 1

3

Since systemd v235, systemd has had a LineMax= directive for journald.conf, which is set by default to 48k. So your first option is to set this to a larger value if you want to support really long lines in your logs.

Second, if you use an output format like json (journalctl -o json), you can see if a line break was insert. It will appear in the JSON like this:

  _LINE_BREAK: "line-max"

If you run into that, you know the next record is a continuation of this line. FOr more about this feature you can see this git commit:

https://github.com/systemd/systemd/commit/ec20fe5ffb8a00469bab209fff6c069bb93c6db2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .