1

I've a VPS and about 100 headless device connected to VPS througth ssh inverse connection. In the last days, the VPS reject ssh connections few minutes after reboot it with this message kex_exchange_identification: read: Connection reset by peer.

If I'm connected to VPS with debug ssh flag sometimes I saw debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1.

I readed about it and seems the error is caused by reaching the maximum number of sockets allowed.

With ss -xa | wc -l:

494

But 430 are like follows and I don't know what service open this sockets:

Netid  State      Recv-Q Send-Q   Local Address:Port       Peer Address:Port
u_dgr ESTAB      0      0               * 2147483647                  * 0

With cat /proc/user_beancounters:

Version: 2.5
       uid  resource                     held              maxheld              barrier                limit              failcnt
   103848:  kmemsize                 66312098             68554752  9223372036854775807  9223372036854775807                    0
            lockedpages                     0                  503               262144               262144                    0
            privvmpages                369146               386161  9223372036854775807  9223372036854775807                    0
            shmpages                    74623                74943  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numproc                       357                  393  9223372036854775807  9223372036854775807                    0
            physpages                  146470               154269                    0               262144                    0
            vmguarpages                     0                    0               294912  9223372036854775807                    0
            oomguarpages                99708               106874               262144  9223372036854775807                    0
            numtcpsock                    338                  339  9223372036854775807  9223372036854775807                    0
            numflock                       59                   64  9223372036854775807  9223372036854775807                    0
            numpty                          2                    4  9223372036854775807  9223372036854775807                    0
            numsiginfo                      0                   24  9223372036854775807  9223372036854775807                    0
            tcpsndbuf                 7093000              7110440  9223372036854775807  9223372036854775807                    0
            tcprcvbuf                 5766768              5783152  9223372036854775807  9223372036854775807                    0
            othersockbuf               523272               871456  9223372036854775807  9223372036854775807                    0
            dgramrcvbuf                     0                30520  9223372036854775807  9223372036854775807                    0
            numothersock                  500                  500                  500                  500                 8621
            dcachesize               26527977             26991757  9223372036854775807  9223372036854775807                    0
            numfile                      6064                 6290  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numiptent                      30                   30  9223372036854775807  9223372036854775807                    0

How can I know what service is opening so many sockets?

Improve this question

1 Answer 1

Reset to default
1

The inclusion of /proc/user_beancounters tells me this is a virtualized instance running on an OpenVZ host. The limits of which would be at the discretion of your provider.

The counter that is failing is numothersock which is defined as "Maximum number of non-TCP sockets (local sockets, UDP and other types of sockets)."

Examine the output of ss -xa (all unix sockets) and ss -ua (all UDP sockets). I have found the "Local Address:Port" column to be useful to find which processes are holding those sockets.

You would then look at the output from lsof and dig out process names and numbers using the port found above:

lsof | grep $PORT

That should give you some indication of what process is holding those sockets.

Improve this answer
2
  • The ss -xa command output show me at 85% lines like this: u_str ESTAB 0 0 * 2147483647 * 0. The other 15% lines returned are like: u_str LISTEN 0 0 /path_to_different_service.2438 2147483647 * 0. How can I know to which process they belong? @GracefulRestart
    – crossmax
    Jan 29, 2020 at 9:39
  • ESTAB would mean that is an active connection, LISTEN would be a socket ready for connections. You would now look through lsof to match the local port number to get the service. This is working on my machine: for port in $(ss -xa | awk '{print $6}'); do lsof|grep $port; done. The first column in the output should be the service name that owns the socket.
    – GracefulRestart
    Jan 29, 2020 at 18:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .