I have a modern Linux desktop with lots of processes running concurrently. One of those processes, and I don't know which one, calls a function
some_func from a popular dynamic library
libx11, so a lot of processes use it), and I want to know what process does that (and ideally, have a stack trace of each invocation).
How do I determine which process makes a call to
Options that I've considered thus far:
latrace: Having an
ltrace-style detailed list of what process called the function I'm interested in with what arguments would be perfect, but
ltraceonly works with an individual process or process groups. I cannot just type
ltrace -e some_func@some_lib -fp 1and see all uses system-wide.
- Find what processes use my library with
lsof, then proceed with step 1: That would be very cumbersome, since there are too many processes using the same library, but not calling said function.
grep -r some_func /usr, then see if there are only a couple of binaries capable of calling the function, and work my way from there. Although that could work in some limited amounts of cases, this is by no means a general solution, and wouldn't work if e.g.
some_funcis ubiquitous in various binaries but is seldom called.
- Use the kernel audit system. If I was tracing a system call, I could type
auditctl -S some_syscall ...and that would do the trick in logging system-wide invocations. However,
auditctldoes not seem to be capable of doing the same level of granularity with library functions.
- Finally, I could rebuild the library, adding a new line to the function I'm interested in that would log all its invocations. While this would be guaranteed to solve my issue, that solution would be cumbersome and require modifying/recompiling the library and at least 2 reboots to roll out the instrumented library and to roll it back after finding the culprit.
Is there an easier way?
(I want to point out that this is meant to be a general question, and am mostly interested in general solutions that would Just Work.)
I found a nice comparison article mentioning a few more tracing facilities I wasn't aware of, which may be worth exploring.