5

I have a problem that I am trying to hack around where I have lots of local port forwards getting sockets stuck in CLOSE_WAIT and lots of remote port forwards getting sockets stuck in FIN_WAIT2.

Currently I have no idea why this happens. Seems like a bug in ssh (running OpenSSH_6.6.1p1, OpenSSL 0.9.8j-fips 07 Jan 2009 on SuSE11 R4). However, I would like to simulate this so that I can write a socket smasher program.

I've tried writing python scripts that open sockets and die, I've tried opening them in threads and killing the calling app. I've tried establishing MySQL connections and letting them dangle (because this is what is causing the CLOSE_WAITs, but I can't replicate it) I could try 1000 other things, but none of them might cause this to happen. Both of the applications that are initiating the connections that get hung are closed source :( (one is Science Logic database connections, the other is some proprietary connection from a Cisco CSPC box)

So what do I need to do to get a socket stuck in CLOSE_WAIT and what do I need to do to get a socket stuck in FIN_WAIT2?

Improve this question

1 Answer 1

Reset to default
0

CLOSE-WAIT happens when the peer system (process) closed its side of the TCP connection, this has been detected by the local OS and transmitted to the local process, but the local process hasn't yet acknowledged this by also closing its side of the TCP connection. This usually becomes visible when an application is busy, has a bug making it "forget" to close some of its sockets or when it hangs (thus being unable to do further closes). Meanwhile the remote side will have a corresponding FIN-WAIT-2 but this one will eventually expire.

It can be reproduced with a local listening and forking socat process that will send a STOP signal to each of forked socat subprocess to "hang" it, and a remote connection that gives up after 1 second (also done with a socat command):

  • local that will immediately STOP itself guarantying a CLOSE-WAIT state since it won't be able to close its side of the TCP connection:

    socat tcp4-listen:5555,reuseaddr,fork system:'kill -STOP $SOCAT_PID'

    With each received connection a socat subprocess is fork-ed and will itself fork a shell which will immediately STOP this socat subprocess (using the inherited variable $SOCAT_PID), preventing it to (detect remote side has closed and) close its side of the TCP connection.

  • remote (or also local in this example to keep it simple) that will give up after 1 second of inactivity, getting an associated FIN-WAIT-2 state along the peer's CLOSE-WAIT:

    for i in $(seq 1 5); do  socat -T 1 -u tcp4:127.0.0.1:5555 -; echo $i; done

example result once the loop above completed:

$ ss -tn sport == 5555 or dport == 5555
State        Recv-Q   Send-Q     Local Address:Port        Peer Address:Port    
FIN-WAIT-2   0        0              127.0.0.1:33836          127.0.0.1:5555    
FIN-WAIT-2   0        0              127.0.0.1:33846          127.0.0.1:5555    
FIN-WAIT-2   0        0              127.0.0.1:33842          127.0.0.1:5555    
CLOSE-WAIT   1        0              127.0.0.1:5555           127.0.0.1:33840   
CLOSE-WAIT   1        0              127.0.0.1:5555           127.0.0.1:33836   
CLOSE-WAIT   1        0              127.0.0.1:5555           127.0.0.1:33842   
FIN-WAIT-2   0        0              127.0.0.1:33840          127.0.0.1:5555    
CLOSE-WAIT   1        0              127.0.0.1:5555           127.0.0.1:33846   
CLOSE-WAIT   1        0              127.0.0.1:5555           127.0.0.1:33834   
FIN-WAIT-2   0        0              127.0.0.1:33834          127.0.0.1:5555

FIN-WAIT-2 will eventually expire, but CLOSE-WAIT will stay as long as the (stopped) processes exist. Interrupting the main listening socat will kill its children and clean everything.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .