What is this `gmain` process with these unknown PIDs in my trace file?

Question

Reading up on an answer to find out which process registered inotify watchers, I executed the following commands

echo 1 | sudo tee /sys/kernel/debug/tracing/events/syscalls/sys_exit_inotify_add_watch/enable
sudo cat /sys/kernel/debug/tracing/trace 

The resulting output from the trace file had a header that indicated that the first column should have the name of the process (task?), along with the PID:

#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
           gmain-1715  [004] .... 23200.386116: sys_inotify_add_watch -> 0xfffffffffffffffe

Surprisingly, none of the PIDs listed in the many lines of output exist when I grep for them using ps -ef | grep $THE_PID (where $THE_PID would be 1715). Also, the task name is unknown to me and does not exist in the ps output either.

Further, the entire list is all tuples like gmain-<some number>. Not the expected postgres or node. So what is this gmain process? And what are these PIDs that do not exist in /proc/? Are they historic by the time I try accessing them?

I see gmain is a thing in these kernel docs by Theodore Ts'o on tracing, so it's something.

Answer 1

These are not processes. These are tasks. Linux works in terms of tasks. You do not see their IDs in a process list because these tasks happen to be threads within a process. They are the worker threads of some multi-threaded GIO process somewhere. You'll find them in the task/ subdirectory of the /proc/<process> subdirectory (example /proc/860/task/926).

Answer 2

As suspected, this gmain process had something to do with GTK or Gnome, but the thing to note is that it wasn't a process at all, but a thread (the gtk main loop)! That's also why it didn't show when grepping ps.

I understood this when using the -q option to ps, which lets you list the pids you are interested in. The pid that came up wasn't the one I passed as an option at all, but still showed when I did pstree -p, which made me think this was perhaps some kind of thread.

Using my newfound knowledge, I found that I could list all the threads, that also have PIDs (is this the right name?) of their own by supplying ps with the -L option.

Example: sudo ps -efL -q 906.

This (and pstree) led me to find out that the thread belonged to NetworkManager).