0

I have a problem concerning the logging of a process:

  • I have 3 scripts, a.py, b.py and c.sh
  • a.py calls b.py and b.py calls c.sh

The goal is to continuously log std out/err (write to file and console)

I am currently try to start the initial script like this:

python -u a.py 2>&1 | tee logfile.txt

Now the problem is that the output from c.sh only gets printed/logged once b.py completely finished its execution, but I rather want it live (all the lines from c.sh don't appear after each other, they all appear together once the whole process is finished).

Is there a better way of logging or do I achieve this with tee somehow?

3

I personally believe that one the very first things that you have to do in a command-line script or application is setup a logging system. Just with a couple of re-usable lines, you can have logging on all your python applications and even use different debug levels :).

I wouldn't base logging in writing stdout to files with tee, just log to both stdout and your logging channel. I'm assuming that you can modify your python scripts, everything below applies to that situation. (scroll down for the "using tee answer" :).

In Python, you can use the Logging module (check the Logging HOWTO in the official Python documentation : https://docs.python.org/2/howto/logging.html) but unfortunately different processes cannot write to the same file (different theads of the same process can).

So, in this case you're describing, I think the base case will be log to syslog, either with the Logging library, or with the Syslog Library (https://docs.python.org/2/library/syslog.html).

You could log to syslog's local0.* facility on all the applications and let syslog (rsyslog in my case) to write to the file:

local0.*                /var/log/myapp.log

The best about this system is that you can even log to remote syslog servers on different machines, not only in the one executing the apps.

PS: Don't forget to set a /etc/logrotate.d/myapp policy to rotate your app log files!

The tee option

If you still need to use tee, did you try to use the -a flag for the 2nd and 3rd process? As stated in http://docstore.mik.ua/orelly/unix/upt/ch13_10.htm:

The tee ( 13.9 ) command writes its standard input to a file and writes the same text to its standard output. You might want to collect several commands' output and tee them all to the same file, one after another. The obvious way to do that is with the -a option

$ some-command | tee teefile

$ another-command | tee -a teefile

$ a-third-command | tee -a teefile

  • I call a.py, which calls python -u b.py | tee -a log, but still, I get the results from b.py live in file, but I don't get them live on terminal, there the results get printed once b.py finished running (but I want them line by line on terminal, not all together at the end) – RandomDisplayName May 1 '15 at 7:07
  • Nevermind, its working, I executed the commands using local, where I had Capture=True, this has to be set to false, so your answer is all good, thank you. – RandomDisplayName May 1 '15 at 7:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.