1

How would you go about detecting and alerting about newly orphaned pids? I would like to cron a script that checks once a day for any orphaned pids (PPID = 1) for a specific user and only alert if a new orphaned process has been detected.

e.g ps -efa output grepping for orphaned pids for "user"

day 1

   user  1111111        1   0   Dec 18      -  0:00 command <--- send email about this pid

day 2

   user  1111111        1   0   Dec 18      -  0:00 command
   user  2222222        1   0   Dec 14      -  0:00 command <--- send email about this pid
2
  • I may have misunderstood your question. A comment to my answer below indicates that you actually were asking about how to create alerts from an analysis of a log of the daily cron output. If you can confirm that's actually what you want, I'll add that in an edit tomorrow.
    – Seamus
    Dec 20, 2021 at 8:57
  • Re alerting about newly orphaned pids & the subsequent reference <--- send email...: Could you please clarify what sort of alert might work here? If email is what you need, do you have an SMTP server set up? Is the mail to be strictly a local message, or (for example) a gmail account?
    – Seamus
    Dec 20, 2021 at 18:44

1 Answer 1

1

I'll treat this as in two-steps: 1. detecting the orphan(s), 2. set an alert when a new orphan(s) is found.

1. Detecting the orphan(s)

You may start with something like this:

ps -eo pid,ppid,ruser,stat,command  

This will give you a table listing all processes with a header for the 5 outputs listed here (pid,ppid,ruser,stat,command). Please refer to the section STANDARD FORMAT SPECIFIERS of man ps for a description of these 5 output fields.

Successive pipes of the output to grep may be used to filter the list, or awk may be more efficient to capture the desired processes; e.g:

ps -eo pid,ppid,ruser,stat,command | awk '{ if ($2 == 1 && $3 == "pi") print $0 "\n";}'
  926     1 pi       Ss   /lib/systemd/systemd --user

Here, awk prints the entire line/record ($0) of any line of the ps output IF the second field ($2) has a ppid of 1, AND the third field ($3) is a string for a user (pi in this case). Running this on my system yielded the result shown above.

But now it gets tricky... AFAIK, there is no value of the stat field that by and of itself identifies an orphan process; i.e. a process whose parent process has died, but has been adopted by the init process - whose pid = 1. You may inspect the PROCESS STATE CODES section of man ps to review the possible values of the stat field. A value of "Z" indicates Zombie process - which is similar to an orphan, but different.

In summary then, ps cannot tell us definitively whether or not a process is an orphan, and so we must regard the output of the command above as a list of suspects to be further investigated.

Depending upon your users and your system, you may be able to eliminate some processes based on the value of the stat code, or the command field. For example, the command above found a process (systemd) with stat=Ss; this process is not an orphan. AFAICT, the 5 parameters being output from the ps command above provide a reasonable basis for meeting your objectives.

2. Set an alert when a new orphan(s) is found

Following is the proposed approach to setting an alert using the orphan detection command in step 1. above.

Two (2) files will be used: OrphansOfRecord, and OrphansOfTheDay. OrphansOfTheDay will be generated as follows:

ps -eo pid,ppid,ruser,stat,command | awk '{ if ($2 == 1 && $3 == "pi") print $0 "\n";}' > OrphansOfTheDay

Once generated, each line of OrphansOfTheDay is compared with each line of OrphansOfRecord; i.e. for each line in OrphansOfTheDay:

  • If that line is NOT found in OrphansOfRecord, it is a NEW Orphan, and the Alarm is set.
  • If there are no NEW Orphans, the alarm is not set; i.e. no alarm for cleared Orphans

A NEW Orphan process is defined as follows:

  • The pid is new OR the command is new
awk 'NR == FNR{a[$5]b[$1];} !($5 in a)||!($1 in b){print "ALERT" > "alertfile"; close ("alertfile")}' OrphansOfRecord OrphansOfTheDay

Parsing this awk command:

  • NR == FNR is a condition in awk - a condition that evaluates TRUE while the first file listed in the arguments is being read - OrphansOfRecord in this case. The condition evaluates FALSE after reading the first file.

  • While, NR == FNR is true, the action {a[$5]b[$1];} is executed. This action stores the values of field $5 (command) and $1 (pid) from each line in OrphansOfRecord in the arrays a and b, respectively. For example, if there are 3 lines (or records) in the OrphansOfRecord file, then a & b will each contain 3 elements when this command is completed.

  • After reading all the lines in OrphansOfRecord, NR == FNR becomes false, and awk begins reading the second file OrphansOfTheDay. After the first line of OrphansOfTheDay is read, the 2nd condition is evaluated: !($5 in a)||!($1 in b). This condition compares the $5 value from that line against the values in array a, and the $1 value from the same line against the values in the array b. Note that the condition tests for no match of $5 in a or a match of $1 in b.

  • This continues as awk tests each line of OrphansOfTheDay. When the 2nd condition is TRUE, the 2nd action is executed: {print "ALERT" > "alertfile"; close ("alertfile")}. This action creates alertfile if it does not already exist by redirecting the print "ALERT" to alertfile & then closing it to ensure the output buffer is flushed.

The "ALERT" output may be used to signal that a new, suspected Orphan has been found. The presence of alertfile - or its contents "ALERT" - may be used to determine if an email needs to be sent.

At this point, OrphansOfTheDay has been processed, and an "ALERT" has been created if a suspected Orphan was found. Two things remain to be done:

  1. Write OrphansOfTheDay to OrphansOfRecord:
mv OrphansOfTheDay OrphansOfRecord
  1. If an "ALERT" was set, an email is sent & alertfile is cleared:
if [ -e alertfile ] 
then 
   mail -s "ALERT: NEW ORPHAN FOUND" pi < OrphansOfRecord 
   rm alertfile
fi

Xmas is calling, and I'm out of time for a few days. I'll put the script together asap - or you may proceed on your own.

15
  • The problem is that if this is run through cron, it will need to store PID results from previous runs and compare them. The OP already knows how to identify the orphans (PPID will be 1, so the trick is finding new ones), but cron will run the script and exit so there is no way to compare with previous runs unless you store them in a text file which is read by the script on startup.
    – terdon
    Dec 20, 2021 at 8:38
  • @terdon: I did not get that the OP knew how to id the orphans... you can't rely on PPID=1 exclusively. But yeah - I can add something to redirect the cron output to a file.
    – Seamus
    Dec 20, 2021 at 8:41
  • No, of course, but if you get those with PPID==1 on the first run, that should include the things actually started by the init, so any new ones with PPID==1 are the ones to look out for. Also, I think it will be a relatively safe bet that any processes belonging to the target user and not root with PPID==1 will be the orphans.
    – terdon
    Dec 20, 2021 at 8:43
  • @terdon: Yeah, maybe :) It's late here, so I'll edit this answer tomorrow & leave a Comment for the OP.
    – Seamus
    Dec 20, 2021 at 8:55
  • 1
    On a second though etime or lstart won't help, in case the process is running for a long time and only became orphan in the last 24 hours.
    – aviro
    Dec 20, 2021 at 19:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .