I have a six-node glusterfs cluster running. This morning, I noticed that one of the machines was behaving strangely, so I rebooted it, just to be safe - a manual STONITH, if you will.

After rebooting, three of the other nodes recognize the rebooted machine as "Connected" in gluster pool list and gluster peer status, but the other two show a status of "Disconnected". Strangely, even the nodes which show "Disconnected" in gluster pool list still show it as "Connected" in gluster volume heal [volname] info.

I've tried gluster peer probe from both sides with no effect. I have verified that I can connect to both port 24007 and port 49154 (the brick port shown in gluster volume status on the peers which recognize it as Connected) on the rebooted node from the machines which consider it to be "Disconnected".

/var/log/glusterfs/glustershd.log on the nodes which see the rebooted server as Disconnected contains:

[2018-01-09 11:36:39.258109] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-palantir-client-4: disconnected from palantir-client-4. Client process will keep trying to connect to glusterd until brick's port is available
[2018-01-09 11:36:50.074074] E [socket.c:2309:socket_connect_finish] 0-palantir-client-4: connection to xxx.xxx.xxx.205:24007 failed (No route to host)

However, an hour and a half later, it has not reconnected, despite the first log entry's claim that it would keep trying.

Given all this, what do I need to do to get the two errant peers to reconnect to the rebooted node?


After much web (and soul) searching, I took the chance of stopping and restarting the glusterfs server service (systemctl restart glusterfs-server) on the two nodes which saw their rebooted peer as Disconnected and this got things back in sync.

Most importantly, performing these restarts resulted in no lost data, even though one of the restarted nodes was the replica of the peer it saw as Disconnected. Presumably, replication was still taking place by way of the nodes which still saw the rebooted peer as connected.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.