1

How do you open a file descriptor and echo it to a terminal while it's being written to from a process?

I have a backup program, Duplicity, that writes its logs to a file descriptor specified by the --log-fd=16 parameter.

Sure enough, if I run lsof -p <duplicity PID> I see:

python2 9224 myuser    0r      CHR                1,3      0t0         6 /dev/null
python2 9224 myuser    1w      CHR                1,3      0t0         6 /dev/null
python2 9224 myuser    2w      CHR                1,3      0t0         6 /dev/null
python2 9224 myuser    3u  a_inode               0,11        0      7005 [eventfd]
python2 9224 myuser    4u     unix 0x0000000000000000      0t0    158199 type=STREAM
python2 9224 myuser    5u  a_inode               0,11        0      7005 [eventfd]
python2 9224 myuser    6u  a_inode               0,11        0      7005 [eventfd]
python2 9224 myuser    7r      DIR                8,3     4096  22414346 <some random file being accessed during the backup>
python2 9224 myuser    8r      CHR                1,9      0t0        11 /dev/urandom
python2 9224 myuser   15r     FIFO               0,10      0t0    157054 pipe
python2 9224 myuser   16w     FIFO               0,10      0t0    157054 pipe

However, if I try to open the file descriptor in Python, I get an error:

>>> import os
>>> os.fdopen(16)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 9] Bad file descriptor

Why is this? How do I read the the file descriptor?

5
  • 2
    File descriptors are local to a process. Every process has their own fd 0, 1, and 2, and so on. It's just an index into a kernel structure in the process' execution context.
    – Kusalananda
    May 24, 2017 at 19:55
  • You could possibly use some kernel tracing tool e.g. SystemTap to snoop on the file descriptor calls of that process (but why?)
    – thrig
    May 24, 2017 at 20:32
  • @thrig, I'm using Deja-Dup for backups, which calls duplicity behind the scenes. Unfortunately, Deja-Dup outputs zero progress information, even though duplicity supports it, so I'm trying to view duplicity's logs to track progress.
    – Cerin
    May 25, 2017 at 0:01
  • So, this is basically an issue relating to micro-managing a backup? Don't you trust duplicity to do its job?
    – Kusalananda
    May 25, 2017 at 18:21
  • @Kusalananda, No, I want to see progress, so I know if there's 1 hour or 8 hours remaining...
    – Cerin
    May 25, 2017 at 22:04

3 Answers 3

2

Use strace (trace system calls and signals).

Usage:

sudo strace -p <PID of writing process> -s 9999 -e write=<corresponding FD>

From manual page:

       -p pid      Attach to the process with the process ID pid and begin tracing.  The trace may be terminated
                   at any time by a keyboard interrupt signal (CTRL-C).  strace will respond by detaching itself
                   from  the  traced process(es) leaving it (them) to continue running.  Multiple -p options can
                   be used to attach to many processes in addition to command (which is optional if at least one
                   -p option is given).  -p "`pidof PROG`" syntax is supported.
    
       -s strsize  Specify the maximum string size to print (the default is 32).  Note that  filenames  are  not
                   considered strings and are always printed in full.
    
       -e read=set
              Perform a full hexadecimal and ASCII dump of all the data read from file descriptors listed in the
              specified set.  For example,  to  see  all  input  activity  on  file  descriptors  3  and  5  use
              -e read=3,5.   Note  that  this  is independent from the normal tracing of the read(2) system call
              which is controlled by the option -e trace=read.

       -e write=set
              Perform a full hexadecimal and ASCII dump of all the data written to file  descriptors  listed  in
              the  specified  set.   For  example,  to  see  all output activity on file descriptors 3 and 5 use
              -e write=3,5.  Note that this is independent from the normal tracing of the write(2)  system  call
              which is controlled by the option -e trace=write.

Ref: https://man7.org/linux/man-pages/man1/strace.1.html

5
  • The question is asking how to read from a (known) file descriptor open in another process. strace can’t help with that... Nov 6, 2020 at 21:50
  • What's blocking someone from using strace to read data from a FD if PID# and FD# are already known?
    – Toran Sahu
    Nov 7, 2020 at 4:04
  • You can’t ask strace to read on its own; all it can do is show you the data that’s read by the process being traced, it won’t issue extra reads by itself. In the context of this question, the process under consideration isn’t reading, it’s writing. -e write might do what the OP is after, but -e read won’t. Nov 7, 2020 at 8:02
  • Got it. The answer could have been more precise and descriptive. I thought the provided description of read and write would be intuitive and enough. I'll update the answer. Thanks for the correction/suggestion.
    – Toran Sahu
    Nov 8, 2020 at 10:09
  • Note: if you take the time to understand how strace implements this, that's one of the ways to build your own tool that does what you want without the whole feature set and specific output format of strace. (P.S. +1 to this answer.)
    – mtraceur
    Apr 6 at 23:19
1

I believe duplicity's --log=fd option is intended for complex pipelines where you want to separate stderr and stdout from your log.

This answer to this question gives an example. Here's a simple example:

#!/bin/sh
# Generate output on three different fds
echo hello >&3
echo world >&2
echo today >&1

And when executed like this,

./foo 2> 2.log 3> 3.log 1> 1.log

Results in

$ cat 1.log 2.log 3.log
today
world
hello
6
  • So you're saying --log-fd=16 means there are 16 different pipes?
    – Cerin
    May 25, 2017 at 0:00
  • No it means you can redirect its output using "16> logfile" on the command line.
    – rrauenza
    May 25, 2017 at 6:14
  • How do I determine the full path for the logfile using "16"?
    – Cerin
    May 25, 2017 at 16:28
  • You don't. You need to understand Unix file redirection - that's your missing piece. Only the calling process can capture that output.
    – rrauenza
    May 25, 2017 at 18:06
  • Can you add --log-file filename to the duplicity command line?
    – rrauenza
    May 25, 2017 at 18:09
0

Linux recently gained system calls for exactly this kind of thing:

  1. Use pidfd_open to get a "PID FD" from a PID.

  2. Use pidfd_getfd to get a file descriptor from another process through its PID FD.

As of Python 3.9, pidfd_open is available as os.pidfd_open.

pidfd_getfd is not yet exposed through the Python standard library, but luckily ctypes lets us call syscall, Linux system call numbers never change, and Linux system call API and ABI only changes in backwards-compatible ways.

So!

from ctypes import CDLL, c_int, c_long, c_uint, get_errno
from functools import partial
from os import strerror


_syscall = CDLL(None, use_errno=True).syscall

# Non-variadic system call number argument:
_syscall.argtypes = [c_long]


def pidfd_getfd(pidfd, targetfd):
    fd = _syscall(
             438,  # system call number of pidfd_getfd
             c_int(pidfd),
             c_int(targetfd),
             c_uint(0),  # unused "flags" argument
         )
    if fd == -1:
        errno = get_errno()
        raise OSError(errno, strerror(errno))
    return fd

So in your example, where the PID of interest is 9224, instead of calling os.fdopen(16), you'd do os.fdopen(pidfd_getfd(os.pidfd_open(9224), 16)).

Note that this only works if you have the necessary permissions to reach into the target process, so you might need to run this code with elevated privileges (e.g. sudo) depending on how the process was started and how your system is configured.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .