3

I'm trying to debug a failing legacy process monitor that searches processes that are running. I determined it is being caused by some mysterious linux behavior. The process being ran is /opt/my/path/directoryname/daemonname but for some odd reason, it shows up with spaces, not /, around directoryname. Many daemons are running from the same directory and show up with the correct path fine. It's just this one particular daemon on every server.

ps -ef shows the following for this daemon:

/opt/my/path directoryname daemonname --arg1 VALUE1 --arg-two VALUE.TWO --arg-three ARG.UMENT.THREE.ERR --worker daemonname --pid-role daemonname

This daemon is being launched by a perl script. Comparing it to others, it looks like the directory path is being provided using the exact same env variable as all the other scripts, but those other scripts result in correct path like /opt/my/path/directoryname/daemonname. The relevant perl lines look something like exec => catfile( $settings->config("/directories/executables"), $daemon_name ), and I've confirmed the value of config("/directories/executables") is the correct value /opt/my/path/directoryname

Searching for this question has been a nightmare due to all of the questions about directory names having spaces. This is not that. None of the path should contain a space. ps shows my mkdir command with a space where I had a slash is the closest I could find, which was unanswered.

4
  • "Debugging a failing legacy process monitor"… Did you manage to find precisely how the string referring to the path is built ? and how it is printed ?
    – MC68020
    Commented Apr 26 at 16:27
  • @MC68020 If you're asking where I got the value with the spaces, that's what is showing when I run ps -ef Commented Apr 26 at 16:31
  • Thnks for that info. OK then do ps -eo exe and ps -eo command both show identically (with regards to the path) ?
    – MC68020
    Commented Apr 26 at 16:52
  • @MC68020 $ procps-ng version 3.3.10 must not support the exe option but yes, ps -eo command gives the same command Commented Apr 26 at 17:28

1 Answer 1

10

Among the fields returned by ps -f is the argument list passed to the command execution that the process (or one of its ancestor) last made. You can get just that with ps -o args.

When a process executes a file, that's with the execve() system call:

execve("/path/to/executable", [arg0, arg1..., 0], [env1, env2... 0])

arg0 by convention is the name of the executable. When a shell interprets a:

cmd arg

command line, it calls execve("/path/to/cmd", ["cmd", "arg", 0], ...). When that's:

/path/to/cmd arg

It calls execve("/path/to/cmd", ["/path/to/cmd", "arg", 0], ...).

Those strings (cmd, arg) end up at the bottom of the stack of the process, NUL-delimited. Inside the process, that's what argv[0], argv[1]... point to.

On Linux, that area is exposed in /proc/<pid>/cmdline, and that's where ps gets it from for the args field.

Technically, ps takes that list of strings and prints it joined with spaces so that in the end the args fields looks like the shell command line that would be used to execute the command.

Here, if ps shows:

/opt/my/path directoryname daemonname --arg1...

Those spaces may indicate either a real space in those strings, or separate strings or the fact that spaces have been replaced by NULs in those strings.

The latter is plausible if the process happened to use the standard strtok() function or a similar approach on its argv[0] to split it on /s possibly to find out the base name of the executable it was run as or the different directory components of its path.

For instance, if I create:

#include <string.h>
#include <unistd.h>
int main(int argc, char *argv[])
{
  strtok(argv[0], "/");
  while (strtok(NULL, "/"));
  pause();
}

And compile and run it:

$ cc a.c
$ "$PWD/a.out" &
[1] 67758
$ ps -fp "$!"
UID          PID    PPID  C STIME TTY          TIME CMD
chazelas   67758   60043  0 19:19 pts/4    00:00:00 /home chazelas a.out

You'll see the /s have been replaced with spaces.

Actually, they have been replaced with NULs, as can be verified with:

$ sed -n l "/proc/$!/cmdline"
/home\000chazelas\000a.out\000$

But ps saw those NULs and interpreted them as the delimiters for 3 different arguments which it joined with spaces.

As you later noted yourself in comment, dirname() is another standard function that may replace /s with 0 in its argument.

$ cat b.c
#include <libgen.h>
#include <unistd.h>
int main(int argc, char *argv[]) {dirname(dirname(argv[0])); pause();}
$ cc b.c
$ "$PWD/a.out" &
[1] 42128
$ ps -fp "$!"
UID          PID    PPID  C STIME TTY          TIME CMD
chazelas   42128   39744  0 20:18 pts/6    00:00:00 /home chazelas a.out

Then assuming that's the correct explanation, it's not a weird Linux behaviour it's just the application itself modifying its argv[0] after it has started (probably by replacing /s with NULs within, probably to extract some path components).

On Linux, a more reliable way to find which executable a process is currently running is to do a readlink() on /proc/<pid>/exe, for instance with the realpath or readlink utility, or the stat builtin of zsh:

$ zmodload zsh/stat
$ stat +link /proc/$!/exe
/home/chazelas/a.out
$ readlink "/proc/$!/exe"
/home/chazelas/a.out
$ realpath "/proc/$!/exe"
/home/chazelas/a.out

And the procps implementation of ps on Linux returns it with ps -o exe.

There is no foolproof way to know for sure what arguments were passed to the command as what's shown in /proc/<pid>/cmdline can be modified by the process.

Audit logs may have the information though.

4
  • 2
    I can see it's quite plausible that the program uses strrchr(argv[0], '/') and then overwrites that with NUL to give strings for the program name and directory (perhaps it uses that directory to find helper programs and/or data files). Good bit of analysis there Stéphane, and - as always - very clearly explained. Commented Apr 27 at 5:38
  • I need to brush up on my C it looks like. There's no direct strrchr but I see some similar potential culprits like dirname that may affect the pointer value. Appreciate the detailed response. Commented Apr 29 at 19:11
  • 1
    @k.schroeder31, good point about dirname(3). That looks even more likely than strtok(). Commented Apr 29 at 19:14
  • @k.schroeder31 see edit Commented Apr 29 at 19:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .