1

I am running a Debian GNU/Linux 9.5 (stretch) with kernel: 4.9.0-7-amd64.

I have found the culprit of a memory consumption problem I am facing to be an in-app mechanism for sending logs to FluentD daemon, so I am trying to figure out the TCP memory usage.

According to the following outputs

/proc/net/sockstat:

sockets: used 779
TCP: inuse 23 orphan 0 tw 145 alloc 177 mem 4451
UDP: inuse 5 mem 2
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

mem metric is the number of pages (4K) used by TCP memory. So TCP Memory usage equals 4451 * 4 = 17804k

ss -atmp:

State      Recv-Q Send-Q Local Address:Port                 Peer Address:Port                
LISTEN     0      128        *:ssh                      *:*                     users:(("sshd",pid=559,fd=3))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d712)
LISTEN     0      4096   127.0.0.1:8125                     *:*                     users:(("netdata",pid=21419,fd=33))
     skmem:(r0,rb33554432,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      4096       *:19999                    *:*                     users:(("netdata",pid=21419,fd=4))
     skmem:(r0,rb33554432,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768      *:3999                     *:*                     users:(("protokube",pid=3504,fd=9))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768  127.0.0.1:19365                    *:*                     users:(("kubelet",pid=2607,fd=10))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768  127.0.0.1:10248                    *:*                     users:(("kubelet",pid=2607,fd=29))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768  127.0.0.1:10249                    *:*                     users:(("kube-proxy",pid=3250,fd=10))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      128        *:sunrpc                   *:*                     users:(("rpcbind",pid=232,fd=8))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
ESTAB      0      0      172.18.25.47:ssh                  46.198.221.224:35084                 users:(("sshd",pid=20049,fd=3),("sshd",pid=20042,fd=3))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
TIME-WAIT  0      0      100.96.18.1:48226                100.96.18.110:3006                 
    
ESTAB      0      0      172.18.25.47:62641                172.18.18.165:3999                  users:(("protokube",pid=3504,fd=11))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d15390)
ESTAB      0      0      172.18.25.47:3999                 172.18.63.198:46453                 users:(("protokube",pid=3504,fd=17))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
SYN-SENT   0      1      172.18.25.47:28870                172.18.23.194:4000                  users:(("protokube",pid=3504,fd=3))
     skmem:(r0,rb12582912,t1280,tb12582912,f2816,w1280,o0,bl0,d0)
TIME-WAIT  0      0      100.96.18.1:34744                100.96.18.108:3008                 
    
ESTAB      0      0      172.18.25.47:3999                 172.18.18.165:23733                 users:(("protokube",pid=3504,fd=8))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
TIME-WAIT  0      0      100.96.18.1:12992                100.96.18.105:3007                 
    
TIME-WAIT  0      0      100.96.18.1:48198                100.96.18.110:3006                 
    
TIME-WAIT  0      0      100.96.18.1:63502                100.96.18.102:8001                 
    
ESTAB      0      0      127.0.0.1:10249                127.0.0.1:53868                 users:(("kube-proxy",pid=3250,fd=5))
     skmem:(r0,rb12582912,t0,tb12582912,f4096,w0,o0,bl0,d0)
TIME-WAIT  0      0      100.96.18.1:58032                100.96.18.101:3000                 
    
TIME-WAIT  0      0      100.96.18.1:17158                100.96.18.104:8000                 
    
ESTAB      0      0      172.18.25.47:38474                172.18.18.165:https                 users:(("kubelet",pid=2607,fd=38))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d112)
TIME-WAIT  0      0      100.96.18.1:17308                100.96.18.104:8000                 
    
ESTAB      0      0      127.0.0.1:32888                127.0.0.1:10255                 users:(("go.d.plugin",pid=21570,fd=8))
     skmem:(r0,rb12582912,t0,tb12582912,f20480,w0,o0,bl0,d3)
TIME-WAIT  0      0      100.96.18.1:57738                100.96.18.101:3000                 
    
TIME-WAIT  0      0      100.96.18.1:23650                100.96.18.97:3004                 
    
TIME-WAIT  0      0      100.96.18.1:34518                100.96.18.103:3001                 
    
ESTAB      0      0      127.0.0.1:53868                127.0.0.1:10249                 users:(("go.d.plugin",pid=21570,fd=6))
     skmem:(r0,rb12582912,t0,tb12582912,f8192,w0,o0,bl0,d1)
TIME-WAIT  0      0      100.96.18.1:23000                100.96.18.98:3002                 
    
ESTAB      0      0      172.18.25.47:38498                172.18.18.165:https                 users:(("kube-proxy",pid=3250,fd=7))
     skmem:(r0,rb12582912,t0,tb12582912,f8192,w0,o0,bl0,d0)
TIME-WAIT  0      0      100.96.18.1:26430                100.96.18.100:3005                 
    
TIME-WAIT  0      0      100.96.18.1:34882                100.96.18.103:3001                 
    
ESTAB      0      0      172.18.25.47:3999                 172.18.44.34:57033                 users:(("protokube",pid=3504,fd=14))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
ESTAB      0      0      172.18.25.47:3999                 172.18.25.148:60423                 users:(("protokube",pid=3504,fd=18))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
ESTAB      0      0      172.18.25.47:61568                35.196.244.138:https                 users:(("netdata",pid=21419,fd=70))
     skmem:(r0,rb12582912,t0,tb262176,f0,w0,o0,bl0,d0)
TIME-WAIT  0      0      100.96.18.1:13154                100.96.18.105:3007                 
    
ESTAB      0      0      172.18.25.47:54289                172.18.30.39:3999                  users:(("protokube",pid=3504,fd=12))
     skmem:(r0,rb12582912,t0,tb12582912,f4096,w0,o0,bl0,d15392)
TIME-WAIT  0      0      100.96.18.1:34718                100.96.18.108:3008                 
    
TIME-WAIT  0      0      100.96.18.1:24078                100.96.18.97:3004                 
    
LISTEN     0      128       :::ssh                     :::*                     users:(("sshd",pid=559,fd=4))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      4096      :::19999                   :::*                     users:(("netdata",pid=21419,fd=5))
     skmem:(r0,rb33554432,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768     :::4000                    :::*                     users:(("protokube",pid=3504,fd=5))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768     :::32003                   :::*                     users:(("kube-proxy",pid=3250,fd=13))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768     :::31719                   :::*                     users:(("kube-proxy",pid=3250,fd=12))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768     :::10250                   :::*                     users:(("kubelet",pid=2607,fd=24))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d23)
LISTEN     0      32768     :::9100                    :::*                     users:(("node_exporter",pid=11027,fd=3))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768     :::31532                   :::*                     users:(("kube-proxy",pid=3250,fd=11))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768     :::30892                   :::*                     users:(("kube-proxy",pid=3250,fd=9))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768     :::10255                   :::*                     users:(("kubelet",pid=2607,fd=26))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      128       :::sunrpc                  :::*                     users:(("rpcbind",pid=232,fd=11))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768     :::10256                   :::*                     users:(("kube-proxy",pid=3250,fd=8))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13492                
    
ESTAB      0      0        ::ffff:172.18.25.47:10250                 ::ffff:172.18.25.148:55670                 users:(("kubelet",pid=2607,fd=40))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d15400)
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13096                
    
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13384                
    
ESTAB      0      0        ::ffff:172.18.25.47:10250                  ::ffff:172.18.44.34:49454                 users:(("kubelet",pid=2607,fd=59))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d7698)
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13200                
    
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13502                
    
TIME-WAIT  0      0        ::ffff:172.18.25.47:4000                  ::ffff:172.18.63.198:25438                
    
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13586                
    
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13298                
    
ESTAB      0      0        ::ffff:172.18.25.47:9100                  ::ffff:172.18.25.148:45776                 users:(("node_exporter",pid=11027,fd=7))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d15419)
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13292                
    
ESTAB      0      0       ::ffff:127.0.0.1:10255                 ::ffff:127.0.0.1:32888                 users:(("kubelet",pid=2607,fd=5))
     skmem:(r0,rb12582912,t0,tb12582912,f4096,w0,o0,bl0,d0)
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13206                
    
ESTAB      0      0        ::ffff:172.18.25.47:10250                 ::ffff:172.18.18.165:33482                 users:(("kubelet",pid=2607,fd=32))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d7707)
TIME-WAIT  0      0        ::ffff:172.18.25.47:4000                   ::ffff:172.18.30.39:45200                
    
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13594                
    
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13390                
    
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13090                
    
ESTAB      0      0        ::ffff:172.18.25.47:10250                 ::ffff:172.18.25.148:55590                 users:(("kubelet",pid=2607,fd=41))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d15418)
ESTAB      0      0        ::ffff:172.18.25.47:10250                 ::ffff:172.18.25.148:55536                 users:(("kubelet",pid=2607,fd=11))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d15401)
ESTAB      0      0        ::ffff:172.18.25.47:10250                 ::ffff:172.18.25.148:55762                 users:(("kubelet",pid=2607,fd=43))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d15407)

According to ss manual:

              skmem:(r<rmem_alloc>,rb<rcv_buf>,t<wmem_alloc>,tb<snd_buf>,
                            f<fwd_alloc>,w<wmem_queued>,o<opt_mem>,
                            bl<back_log>,d<sock_drop>)

              <rmem_alloc>
                     the memory allocated for receiving packet

              <rcv_buf>
                     the total memory can be allocated for receiving packet

              <wmem_alloc>
                     the memory used for sending packet (which has  been  sent
                     to layer 3)

              <snd_buf>
                     the total memory can be allocated for sending packet

              <fwd_alloc>
                     the memory allocated by the socket as cache, but not used
                     for receiving/sending  packet  yet.  If  need  memory  to
                     send/receive  packet,  the  memory  in this cache will be
                     used before allocate additional memory.

              <wmem_queued>
                     The memory allocated for sending packet  (which  has  not
                     been sent to layer 3)

              <ropt_mem>
                     The  memory used for storing socket option, e.g., the key
                     for TCP MD5 signature

              <back_log>
                     The memory used for the sk backlog queue.  On  a  process
                     context,  if  the  process is receiving packet, and a new
                     packet is received, it will be put into  the  sk  backlog
                     queue, so it can be received by the process immediately

              <sock_drop>
                     the  number  of packets dropped before they are de-multi‐
                     plexed into the socket


Adding all the skmem values except for the rb and tb (because they are the maximum amount that can be allocated) and d for dropped packets, I should get a value pretty close to /proc/net/sockstat value. However the value I get is 53k which is nowhere near the 17804k.

Is my logic correct? So what am I missing here?

3
  • You've got a lot of stuff going to to attempt tracking memory usage. – rfmodulator Dec 18 '20 at 18:07
  • @rfmodulator I agree, however the procedure is the same no matter how many processes you are running. Anyway, I have finally figured it out, check my answer below... – mitsos1os Dec 18 '20 at 18:11
  • ...unless there are processes that aren't being accounted for in your procedure. – rfmodulator Dec 18 '20 at 18:29
0

After a lot of searching around, I finally came to a conclusion.

My understanding of how to calculate TCP memory usage is correct.

For every socket add socket_memory = rmem_alloc + wmem_alloc + fwd_alloc + wmem_queued + opt_mem + back_log (the r,t, f, w, bl, o fields in skmem)

The reason that my total captured sockets memory above does not add up, is that a lot of the connections are running inside docker containers which are not displayed in the main system ss output, however they are displayed in kernel output of /proc/net/sockstat.

More info in this helpful stackoverflow question: https://stackoverflow.com/questions/37171909/when-using-docker-established-connections-dont-appear-in-netstat

That explains the difference. For a host-only-running processes situation the memory summed would be a match.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.