0

How to get the real listener PID in AIX 7.1? Shouldn't rmsock work as widely described on the internet?

https://www-01.ibm.com/support/docview.wss?uid=isg3T1019570

The case:

There is a program that implements both server and client protocols which is chosen by arguments at runtime. Like as:

$ myprogram -switcher-mode:port
$ myprogram -provider-mode:port

As soon as the listener is launched we can get its PID using rmsock. Let's assume an arbitrary port number of 40000:

$ netstat -Aan | grep '\.40000.*LISTEN$' | awk '{ print $1 }' | xargs -n1 -i rmsock {} tcpcb
The socket 0xf1000e000334b808 is being held by proccess 10683226 (serviceprg)

$ ps -fp 10683226
     UID      PID     PPID   C    STIME    TTY  TIME CMD
    test 10683226        1   0 08:17:43      -     2 serviceprg -switcher-mode:40000

We can see that it's the correct process, the switcher that listen to the port: serviceprg -switcher-mode:40000.

When the problem begins:

As soon as we start new processes (in background) in provider mode, the rmsock might return their PID as if they were the listener (it's randomly). However those processes may be launched from two ways, manually and automatically (both ways cause the issue) by the switcher when it "feels" that it needs more providers to perform the burst of requests, so it launches more providers using the libc.a function system() like this:

system("serviceprg -provider-mode:40000 1>/dev/null 2>/dev/null &");

So, when we check the lister PID using rmsock it might returns the wrong PID, like as:

$ netstat -Aan | grep '\.40000.*LISTEN$' | awk '{ print $1 }' | xargs -n1 -i rmsock {} tcpcb
The socket 0xf1000e000334b808 is being held by proccess 10690461 (serviceprg)

$ ps -fp 10690461
     UID      PID     PPID   C    STIME    TTY  TIME CMD
    test 10690461        1   0 08:20:04      -  1:10 serviceprg -provider-mode:40000

Check that the program being run for that PID is a process launched after the real listener, and it doesn't listen to the port but connects to it as a client program: serviceprg -provider-mode:40000

When we kill that process, let's say the above PID 10690461, rmsock shifts and might answer another process as the listener wrongly (or the real one, this behaviour seems to be random), when you kill every client connected to that port, then rmsock will always return the correct PID of the real listener.


Just in case you want to know a bit more about its architecture: Other programs connect to that port asking for services, the listener is a switcher that delivers each client request to a instance of the program that provides the service, but it's in that case the same program and it connects to the same port, look at the following diagram: enter image description here

  • I'm confused by your netstat pipeline output, then. Does netstat not report the (already-running) switcher-mode process socket? I'd expect 2 or more sockets & "rmsock" reports, but you're only getting one. Perhaps you could split out the pipeline a bit to see what you're getting back at each step? – Jeff Schaller Feb 1 at 19:23
  • It's filtering the netstat output for: port + the "LISTEN" word at the end of the line ("LISTEN$"). That's why we get only one, but the rmsock points the listener socket to the wrong process (sometimes). The full netstat output grepping only the port number is huge with all sort of statuses (ESTABLISHED, TIME_WAIT and the desired one: LISTEN) – Luciano Feb 1 at 19:34
0

Since netstat -Aan doesn't give the full process listing, to discriminate between a provider and a switcher process, you'll have to gather the PCB IDs, then the PIDs, then check the arguments:

for pid in $(netstat -Aan | grep '\.40000.*LISTEN$' | awk '{ print $1 }' | xargs -n1 -i rmsock {} tcpcb | awk '{print $9}')
do
  ps -o args= -p "$pid" | grep -q "provider-mode:" && echo this is a provider
  ps -o args= -p "$pid" | grep -q "switcher-mode:" && echo this is a switcher
done
  • I simplified the description omitting that the serviceprg launches more subprocesses that for unknown reason and certain circumstances might be blocked and although the main process (the real listener) isn't there anymore a subprocess might hold the socket as if it were a listener when it is not (even a java vm launched by it). And no one can open this port while netstat shows anyone listing to it. Now we have to get the listener from netstat, kill the process, and repeat it until there is no listeners anymore. – Luciano Feb 1 at 18:39
  • Imagine the provider process (that one that doesn't listen to the port). It launches a java vm and other process (all them using a "clean fork": system()). When we ask netstat+rmsock who is listening to the port it sometimes says that it's the java vm, which hasn't any kind of argument giving us tips that it's related to the problem. – Luciano Feb 1 at 18:43
  • It seems that AIX messes its "internal table" related to the socket fds and PIDs when those process are launched (a clean fork using "system" function, it's not a raw fork/exec) by any other process that has opened the related port (as a client or as a server). – Luciano Feb 1 at 18:47
  • I admit that I haven't loaded the entire scenario in my head; so you're saying that checking for the process arguments isn't enough? Is it possible that these processes are leaving behind sockets in FIN_WAIT states that are confusing the results? – Jeff Schaller Feb 1 at 19:04
  • Unfortunately no. Unless nestat is listing wrong information, as you can see at the grep clause filtering "LISTEN$". – Luciano Feb 1 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.