CLOSE-WAIT happens when the peer system (process) closed its side of the TCP connection, this has been detected by the local OS and transmitted to the local process, but the local process hasn't yet acknowledged this by also closing its side of the TCP connection. This usually becomes visible when an application is busy, has a bug making it "forget" to close some of its sockets or when it hangs (thus being unable to do further closes). Meanwhile the remote side will have a corresponding FIN-WAIT-2 but this one will eventually expire.
It can be reproduced with a local listening and forking
socat process that will send a STOP signal to each of forked
socat subprocess to "hang" it, and a remote connection that gives up after 1 second (also done with a
local that will immediately STOP itself guarantying a CLOSE-WAIT state since it won't be able to close its side of the TCP connection:
socat tcp4-listen:5555,reuseaddr,fork system:'kill -STOP $SOCAT_PID'
With each received connection a
socat subprocess is fork-ed and will itself fork a shell which will immediately STOP this
socat subprocess (using the inherited variable
$SOCAT_PID), preventing it to (detect remote side has closed and) close its side of the TCP connection.
remote (or also local in this example to keep it simple) that will give up after 1 second of inactivity, getting an associated FIN-WAIT-2 state along the peer's CLOSE-WAIT:
for i in $(seq 1 5); do socat -T 1 -u tcp4:127.0.0.1:5555 -; echo $i; done
example result once the loop above completed:
$ ss -tn sport == 5555 or dport == 5555
State Recv-Q Send-Q Local Address:Port Peer Address:Port
FIN-WAIT-2 0 0 127.0.0.1:33836 127.0.0.1:5555
FIN-WAIT-2 0 0 127.0.0.1:33846 127.0.0.1:5555
FIN-WAIT-2 0 0 127.0.0.1:33842 127.0.0.1:5555
CLOSE-WAIT 1 0 127.0.0.1:5555 127.0.0.1:33840
CLOSE-WAIT 1 0 127.0.0.1:5555 127.0.0.1:33836
CLOSE-WAIT 1 0 127.0.0.1:5555 127.0.0.1:33842
FIN-WAIT-2 0 0 127.0.0.1:33840 127.0.0.1:5555
CLOSE-WAIT 1 0 127.0.0.1:5555 127.0.0.1:33846
CLOSE-WAIT 1 0 127.0.0.1:5555 127.0.0.1:33834
FIN-WAIT-2 0 0 127.0.0.1:33834 127.0.0.1:5555
FIN-WAIT-2 will eventually expire, but CLOSE-WAIT will stay as long as the (stopped) processes exist. Interrupting the main listening
socat will kill its children and clean everything.