I'm trying to get python to tail a logfile e.g. syslog.

I've tried opening the file in read mode, but it never realises if another process writes to the file. Using a popen, and actual 'tail -f' together with registering the popened process to select, python is able to read any writes. And using the python systemd module, i can get systemd to respond with the contents, but I'd like to know if there are other ways instead of these methods and, why the opening a file for reading and constantly trying to read 1024 bytes, doesn't work.

  • i do not think that tail keeps a file open for reading ... your program should not either – jsotola Oct 8 '19 at 4:40
  • if you're trying to implement tail in python, why not look at how tail works? It's actually pretty complicated. github.com/coreutils/coreutils/blob/master/src/tail.c . If you just want the functionality of tail, the popen to tail -f is a perfectly acceptible solution. tail is orders of magnitude more efficient than your python anyway, and having glanced at that code, way more sophisticated than you're gonna be. – Daniel Farrell Oct 8 '19 at 5:02
  • Right, so i took a look at how tail works using the source package from coreutils, and it seems to be keeping the file open for reads by reopening if it dies. As in how else would tail be tailing something if it cannot read the file for tailing? There's options for using inotify as well, but hmm, the point isn't to use an external program for reading but to use python for reading the file. – placid chat Oct 8 '19 at 5:09
  • I'm asking because i was using tail from two different accounts, and for some reason logger messages wasn't in one of them, so having a purely written version ala python would've been better – placid chat Oct 8 '19 at 7:53

It has to work without reopening the file for such logfiles like syslog. Tail also does not reopen files when not needed (and not asked for). It's just that You need polling on regular files, waiting for data is only possible in pipes and sockets and such.

It would be much better to post the non working code examples You tried to tell You what You is wrong and how to fix. Also it's not very understandable what You want to achieve. If You want something like select or blocking read till data is available that's simply not possible with regular files (by design, they are files not pipes or such stuff). Only possibility would be to rely on inotify for that or do it the plain old way with polling. (That's same as tail does.)

  • Yes you are right, select.poll was used in a pipe to a 'tail -f' command, and not on a file descriptor. On the other hand the while loop and the recheck function of tail is how tail checks if any of the writers have gone away and to reopen stale file descriptors. I've got 1 journalctl -f open, 2 tails from seperate accounts, and a python tail program. Using logger, sometimes the tails don't show any outputs. – placid chat Oct 8 '19 at 20:11
  • Some things to note, syslog doesn't output repeated messages, and a logrotate can affect the output. The python program reads in 1024 bytes which is 2*BUFSIZ but didn't output anything when logger was used whereas journalctl did. For now, though, it seems everything is working fine. Constantly reading in the file seems to work. – placid chat Oct 8 '19 at 20:12
  • Still not really what was the problem and what You mean with separate accounts. – EOhm Oct 8 '19 at 20:29
  • tailing the file using 2 different accounts. – placid chat Oct 8 '19 at 20:33
  • For tail to be following file names so close a file if it is renamed or deleted you need to tell tail to do that with --folow=name. To keep it retrying to open a file you need --retry (or the well know form tail -F, note the capital f). That avoids any problems with log rotation and ensures the tail keeps running in the current file (thought the writers must also close the file when it get renamed or unlinked if they keep it open). – EOhm Oct 8 '19 at 20:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.