6

Our memcache daemon reports non-zero 'curr_connections'...

$ telnet memcache-server 11211
Escape character is '^]'.
stats
...
STAT curr_connections 12
...

...and yet, lsof shows no socket connections:

$ ssh memcache-server
# lsof -P -i -n | grep memcache
memcached 1759 memcached   26u  IPv4  11638      0t0  TCP *:11211 (LISTEN)
memcached 1759 memcached   27u  IPv6  11639      0t0  TCP *:11211 (LISTEN)
memcached 1759 memcached   28u  IPv4  11642      0t0  UDP *:11211 
memcached 1759 memcached   29u  IPv6  11643      0t0  UDP *:11211 

I am guessing 'curr_connections' does not mean what I think it does...

0

1 Answer 1

7

I think you're correct in your logic that stat curr_connections is the number of connections that are current.

curr_connections - Number of open connections to this Memcached server, should be the same value on all servers during normal operation. This is something like the count of mySQL's "SHOW PROCESSLIST" result rows.

Source: Memcached statistics (stats command)

When I setup memcached I noticed that it always maintained 10 as the least amount of curr_connections too.

$ echo stats | nc localhost 11211 | grep curr_conn
STAT curr_connections 10

But why 10?

If you run memcached in verbose mode you'll notice the following output:

$ memcached -vv
...
<26 server listening (auto-negotiate)
<27 server listening (auto-negotiate)
<28 send buffer was 212992, now 268435456
<28 server listening (udp)
<28 server listening (udp)
<28 server listening (udp)
<29 send buffer was 212992, now 268435456
<28 server listening (udp)
<29 server listening (udp)
<29 server listening (udp)
<29 server listening (udp)
<29 server listening (udp)

If you tally up the listening servers (8) + 2 servers (auto-negotiated) you'll discover why there are 10 base servers to start, at least that's what I thought at first. But you need to dig deeper to better understand what's going on.

It would appear that memcached is multi-threaded and so the output that you were taking notice of isn't really how one would tally up the connections.

Notice the threads

The output of ps -eLf shows the threads:

$ ps -eLf | grep memc
saml     20036  4393 20036  0    6 20:07 pts/7    00:00:00 memcached -vv
saml     20036  4393 20037  0    6 20:07 pts/7    00:00:00 memcached -vv
saml     20036  4393 20038  0    6 20:07 pts/7    00:00:00 memcached -vv
saml     20036  4393 20039  0    6 20:07 pts/7    00:00:00 memcached -vv
saml     20036  4393 20040  0    6 20:07 pts/7    00:00:00 memcached -vv
saml     20036  4393 20041  0    6 20:07 pts/7    00:00:00 memcached -vv

There is one master node and 5 worker threads.

Here's what the output of lsof looks like when I've made 3 connections to memcached using the same method as you, telnet localhost 11211.

   ss #1

So it would appear that each thread is maintaining a connection (or a reference to each connection) as they're made and kept open. As soon as the telnet connections are closed they go away from this list.

So how can we count the connections?

So if you wanted to tally up the connections you could either subtract 10 from the curr_connections result, or you could run lsof and count the number of connections that are associated with the main PID. Notice this output:

$ lsof |grep memcac | grep IPv
memcached 20036          saml   26u     IPv4            7833065       0t0       TCP *:memcache (LISTEN)
memcached 20036          saml   27u     IPv6            7833066       0t0       TCP *:memcache (LISTEN)
memcached 20036          saml   28u     IPv4            7833069       0t0       UDP *:memcache 
memcached 20036          saml   29u     IPv6            7833070       0t0       UDP *:memcache 
memcached 20036          saml   30u     IPv6            7962078       0t0       TCP localhost:memcache->localhost:38728 (ESTABLISHED)

That last line is an active connection. So we could count them like this:

$ lsof -p $(pgrep memcached) | grep "memcache->" | wc -l
1

But your output shows IPv4 & IPv6, what's up with that?

To simplify the example even more let's start memcached up and force it to only listen on a single IPv4 address, 192.168.1.20. In the above examples we were starting memcached up on all interfaces and connecting to it using localhost, so let's take another look.

$ memcached -vv -l 192.168.1.20
...
<26 server listening (auto-negotiate)
<27 send buffer was 212992, now 268435456
<27 server listening (udp)
<27 server listening (udp)
<27 server listening (udp)
<27 server listening (udp)

Notice that we're only getting 1/2 as many? Previously we had 2 auto-negotiated servers and 8 UDP, this time we have 1 auto and 4 UDP. Why? Well we've told memcached to only listen on the IPv4 interface, previously it was listening on everything, the IPv4 and localhost. We can convince ourselves of this by attempting to connect to the server on localhost:

$ telnet localhost 11211
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

See we can't connect. But we can using the IPv4 address:

$ telnet 192.168.1.20 11211
Trying 192.168.1.20...
Connected to 192.168.1.20.
Escape character is '^]'.

What does stats say now?

STAT curr_connections 5

See? The curr_connections is showing the number 5 (1 auto + 4 UDP).

4 UDPs?

Why are we running 4 UDPs? This would appear to be a default setting in memcached. You can see the settings using the stats settings command when youtelnet` to the server:

stats settings
STAT maxbytes 67108864
STAT maxconns 1024
STAT tcpport 11211
STAT udpport 11211
STAT inter 192.168.1.20
STAT verbosity 2
...
STAT num_threads 4
STAT num_threads_per_udp 4
...

Can we change that value? Sure it's the -t # switch to memcached.

$ memcached -vv -l 192.168.1.20 -t 1
...
<11 server listening (auto-negotiate)
<12 send buffer was 212992, now 268435456
<12 server listening (udp)

So now we only have the main listener (auto) and 1 UDP thread. If we check stats now:

STAT curr_connections 2

Incidentally, we can't set the number of threads to 0.

$ memcached -vv -l 192.168.1.20 -t 0
Number of threads must be greater than 0

So the lowest we can get curr_connections to go is 2. If we open up 6 telnets (5 from ourself - greeneggs and 1 from another remote server named skinner) we'll see the following in our lsof output:

$ lsof -p $(pgrep memcached) | grep ":memcache"
memcached 24949 saml   11u     IPv4             847365      0t0     TCP greeneggs.bubba.net:memcache (LISTEN)
memcached 24949 saml   12u     IPv4             847366      0t0     UDP greeneggs.bubba.net:memcache 
memcached 24949 saml   13u     IPv4             855914      0t0     TCP greeneggs.bubba.net:memcache->greeneggs.bubba.net:48273 (ESTABLISHED)
memcached 24949 saml   14u     IPv4             872538      0t0     TCP greeneggs.bubba.net:memcache->skinner.bubba.net:41352 (ESTABLISHED)
memcached 24949 saml   15u     IPv4             855975      0t0     TCP greeneggs.bubba.net:memcache->greeneggs.bubba.net:48298 (ESTABLISHED)
memcached 24949 saml   16u     IPv4             855992      0t0     TCP greeneggs.bubba.net:memcache->greeneggs.bubba.net:48305 (ESTABLISHED)
memcached 24949 saml   17u     IPv4             859065      0t0     TCP greeneggs.bubba.net:memcache->greeneggs.bubba.net:48316 (ESTABLISHED)
memcached 24949 saml   18u     IPv4             859077      0t0     TCP greeneggs.bubba.net:memcache->greeneggs.bubba.net:48326 (ESTABLISHED)

So we still have the auto + the 1 UDP and 6 other connections. Our stats command shows this:

STAT curr_connections 8

So no surprise there.

4
  • Thanks for investigating this. It's still not clear what exactly is happening... for example, in my case subtracting 10 would still leave 2 - which are not there in my lsof output. IMHO this looks like a bug in memcached - why would stats report connections... from the memcached threads themselves? And what kind of connects are these? Also, in your screenshot you appear to be connected over IPv6 - which is not the case for me. All in all, I find memcached stats to be... weird, at the very least.
    – ttsiodras
    Mar 2, 2014 at 6:34
  • @ttsiodras - No I'm definitely not using IPv6, I only have IPv4, the memcached daemon out of the box sets up listeners on IPv6, but I have no IP address there so ignore that.
    – slm
    Mar 2, 2014 at 6:38
  • @ttsiodras - can you get the full output from lsof -p $(pgrep memcached) | grep "memcache->"? This should show you all the active connections. I do not believe it's a bug, as I said it spins up threads.
    – slm
    Mar 2, 2014 at 6:40
  • @ttsiodras - I've added another example, does that help?
    – slm
    Mar 2, 2014 at 7:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.