1

I'm having a systemd service defined as follows, that works fine:

[Unit]
Description=my service
After=network.target

[Service]
User=myuser
Group=mygroup
WorkingDirectory=/home/myuser/myapp
Environment="PATH=/home/myuser/myapp/.venv/bin"
ExecStart=/home/myuser/myapp/.venv/bin/python3 /home/myuser/myapp/run.py
Restart=on-failure

[Install]
WantedBy=multi-user.target

This is a Python web application based on Flask framework. Normally in the stdout of the application I can see incoming requests "live", I mean when I run the app like python run.py.

Now after starting the service I'd like to follow logs of the app and I do:

sudo journalctl -f -u my_app.service

and incoming logs are awfully slow - sometimes it takes minutes or more for them to appear in logs. Afterwards they all have proper timestamp though, so it's not like they're disappearing, they do, but after a long time.

What I've tried:

  • to redirect systemd service output to files:

    StandardOutput=file:/var/log/my_app/output.log

    StandardError=file:/var/log/my_app/error.log

    with no luck - they save fine but with the same slow speed

  • to try to dump journalctl logs to offline faster setting SyncIntervalSec from default 5m to 5s - didn't help either

Is there any way to pass those logs faster from my application to journald? I don't have troubles with other services like system authentication ones - I see records immediately. My journald.conf file has default parameters (except one above), my systemd is version 237 and I'm running Ubuntu 18.04.

  • 1
    Can you check if .../python3 -u /home/myuser/myapp/run.py will fix this? It looks like Python stdout buffering is what's preventing your log lines from being flushed right away... Buffering is typically turned off when writing to the terminal (so when you run it manually), but needs an option when writing to file or pipe. – filbranden Mar 8 at 16:11
  • 1
  • Thank you very much, sorry for repeated question - I spend lots of time searching but seems that I used wrong keywords. @filbranden if you post your comment as an answer, I'll be happy to accept it. – adamczi Mar 8 at 17:28
  • @adamczi Totally understandable. It's hard to think that the system producing the logs is the one at fault, especially when it works fine when you run it on a terminal... Once you've been using Unix/Linux for 20+ years, that becomes "obvious" to you, but I definitely agree that's counter intuitive for a beginner. I'll go ahead and post my comment as an answer. Thanks for confirming the -u setting worked for you! – filbranden Mar 8 at 17:39
2

The problem is actually with buffering from the Flask application and not with how systemd or journald are ingesting those logs.

This can be counter-intuitive, since as you mentioned, running python3 run.py directly on the command-line works and shows logs properly and also timestamps look correct on the logs.

The former happens because Unix/Linux will typically set up stdout to be unbuffered when connected to a terminal (since it's expecting interaction with an user), and unbuffered when connected to a file (in case of StandardOutput=file:...) or to a pipe (in case you're logging to the journal, which is the default.)

The latter is because the Python/Flask logger is adding timestamps, so even though it's buffering that output, when it finally issues it into the logs, all the timestamps are there.

Some applications will know this is typically an issue and will set up buffering on stdout appropriately when using it for logs, but this doesn't seem to be the case with this particular Python/Flask setup you are using.

On Python, it's fairly easy to globally change stdout to unbuffered mode, which you can do by:

  1. Passing a -u flag to python3 in your command.
  2. Setting PYTHONUNBUFFERED=1 in your environment (which you can do in the systemd service unit with an additional Environment=PYTHONUNBUFFERED=1 line.)

You confirmed this worked for your specific case, so that's great!

For non-Python applications suffering from similar issues, there are command-line tools such as unbuffer and stdbuf which can often solve this same problem.

Solutions are usually specific to the kind of application, which is somewhat unfortunate, but often googling or looking for other answers in Stack Exchange (once you know buffering is the issue) will usually lead you to an useful suggestion.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.