3

I am migrating a kafka/zookeeper cluster on Windows to Debian wheezy.

  • Java version: 1.7.0_80
  • Debian version: 7.9
  • Zookeeper version: 3.3.5+dfsg1-2 0
  • Kafka version: 2.10-0.8.2.1

If I configure zookeeper on the Debian servers with IP addresses for the other Debian servers, everything works fine. If I use DNS names instead, the leader election fails on the Debian servers.

On the Debian servers, I can lookup the IP of any of the other Debian servers using 'host' command, so DNS resolution is working.

Everything is automated: server creation, Debian installation, zookeeper installation, zookeeper configuration; so the window for manual config errors are at a bare minimum and easy to reproduce or change.

Using clientPortAddress=DNSNAME does not make any difference; it still fails. There is nothing configured in iptables. There is no firewall in between these servers.

In the following, servers 1-3 are Windows 2012R2 servers and servers 4-6 are Debian servers.

This config works:

 server.1=testkafka400:2888:3888
 server.2=testkafka401:2888:3888
 server.3=testkafka402:2888:3888
 server.4=10.1.132.152:2888:3888
 server.5=10.1.132.153:2888:3888
 server.6=10.1.132.154:2888:3888

This config does not work:

 server.1=testkafka400:2888:3888
 server.2=testkafka401:2888:3888
 server.3=testkafka402:2888:3888
 server.4=testkafka403:2888:3888
 server.5=testkafka404:2888:3888
 server.6=testkafka405:2888:3888

When I use the DNS names, I get the following output -- where the exceptions just repeat themselves. Please note that the following log is from a cluster setup containing only Debian servers, using DNS names, for the sake of testing. If I shift to IP, the cluster works and can hold an election.

[2015-11-03 13:55:52,309] INFO Reading configuration from: /etc/zookeeper/config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2015-11-03 13:55:52,322] INFO Defaulting to majority quorums (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2015-11-03 13:55:52,344] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2015-11-03 13:55:52,344] INFO autopurge.purgeInterval set to 24 (org.apache.zookeeper.server.DatadirCleanupManager)
[2015-11-03 13:55:52,345] INFO Purge task started. (org.apache.zookeeper.server.DatadirCleanupManager)
[2015-11-03 13:55:52,454] INFO Purge task completed. (org.apache.zookeeper.server.DatadirCleanupManager)
[2015-11-03 13:55:52,472] INFO Starting quorum peer (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2015-11-03 13:55:52,581] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-11-03 13:55:52,601] INFO tickTime set to 3000 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2015-11-03 13:55:52,601] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2015-11-03 13:55:52,601] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2015-11-03 13:55:52,601] INFO initLimit set to 20 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2015-11-03 13:55:52,626] INFO Reading snapshot /etc/zookeeper/data/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileSnap)
[2015-11-03 13:55:52,675] INFO My election bind port: testkafka403.prod.local/127.0.1.1:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2015-11-03 13:55:52,713] INFO LOOKING (org.apache.zookeeper.server.quorum.QuorumPeer)
[2015-11-03 13:55:52,715] INFO New election. My id =  4, proposed zxid=0x100000014 (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2015-11-03 13:55:52,717] INFO Notification: 1 (message format version), 4 (n.leader), 0x100000014 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x1 (n.peerEpoch) LOOKING (my state) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2015-11-03 13:55:52,732] WARN Cannot open channel to 5 at election address testkafka404.prod.local/10.1.132.153:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.net.SocketTimeoutException
at java.net.SocksSocketImpl.remainingMillis(SocksSocketImpl.java:111)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:341)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:449)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:430)
at java.lang.Thread.run(Thread.java:745)
[2015-11-03 13:55:52,737] WARN Cannot open channel to 6 at election address testkafka405.prod.local/10.1.132.154:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:341)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:449)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:430)
at java.lang.Thread.run(Thread.java:745)
[2015-11-03 13:55:52,919] WARN Cannot open channel to 6 at election address testkafka405.prod.local/10.1.132.154:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)

We really would like to be able to use DNS names, but have no clue as to where we should begin looking for a solution any more. Maybe we missed installing or activating an important Debian or Java feature?

  • Doesn't seem to be a Debian issue. I see the same thing with zookeeper 3.4.6 on centos 7 running openjdk 8 (1.8.0u65, to be exact). Are your Debian hosts running OpenJDK or Oracle Java? – sworisbreathing Nov 26 '15 at 7:21
  • Oracle java. But the hosts file solved the issue. – Christian Mikkelsen Apr 5 '16 at 16:36
6

Okay, so I have an idea of what's going on here. I saw the same issue when trying to set up a 3-node Spring-XD cluster in Vagrant, on Linux VMs.

This configuration worked:

server.1=172.28.128.3:2888:3888
server.2=172.28.128.4:2888:3888
server.3=172.28.128.7:2888:3888

But this one didn't:

server.1=spring-xd-1:2888:3888
server.2=spring-xd-2:2888:3888
server.3=spring-xd-3:2888:3888

The "smoking gun" was this line in my zookeeper log:

2015-11-26 20:48:31,439 [myid:1] - INFO [Thread-2:QuorumCnxManager$Listener@504] - My election bind port: spring-xd-1/127.0.0.1:3888

So, why was Zookeeper binding the election port on the loopback interface? Well...

My /etc/hosts on one of the VMs looked like this:

127.0.0.1   spring-xd-1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

## vagrant-hostmanager-start
172.28.128.3    spring-xd-1
172.28.128.4    spring-xd-2
172.28.128.7    spring-xd-3
## vagrant-hostmanager-end

I removed the hostname from the 127.0.0.1 line in /etc/hosts and bounced the zookeeper service on all 3 nodes, and BAM! everything came up roses. So, now the host file on each machine looks like this:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

## vagrant-hostmanager-start
172.28.128.3    spring-xd-1
172.28.128.4    spring-xd-2
172.28.128.7    spring-xd-3
## vagrant-hostmanager-end

I'm guessing you didn't see the issue on Windows because the hosts file (C:\Windows\System32\drivers\etc\hosts) has no entries by default. You should be able to reproduce the problem on Windows, by adding a similar 127.0.0.1 line to it.

I'm calling this a Zookeeper bug. Editing the hosts file was good enough to prove out the issue and remediate it in Vagrant, but I wouldn't recommend it for any "real" environment.

EDIT: According to http://ccl.cse.nd.edu/operations/condor/hostname.shtml, this seems to be a fairly common problem with clustered apps on Linux, and recommends editing the hosts file as I've described above. However, the Zookeeper documentation on cluster setup doesn't mention it.

  • Thank you! Makes perfect sense, I will test it and get back :) – Christian Mikkelsen Nov 27 '15 at 8:12
  • It took a while to get it tested but it worked :) – Christian Mikkelsen Apr 5 '16 at 16:35
  • Actually in my case the hosts file had 127.0.1.1 entry which appears to be a debian thing. Stripping that line away solved the issue. See serverfault.com/questions/363095/… for deeper explanation of the 127.0.1.1 issue. – Christian Mikkelsen Apr 5 '16 at 17:53
  • Spent a day on this.. Thanks a lot this works... – amateur Jul 16 '16 at 19:00
0

Probably this issue caused by setting node hostname to 127.0.0.1 in /etc/hosts. In this case ZK will bind leader|election ports to 127.0.0.1 address.

Config param quorumListenOnAllIPs=true should remedy this issue and bind election|leader ports to 0.0.0.0.

More options and their impact can be found in ZK administration guide

It is always good to check source code.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.