I have a six-node glusterfs cluster running. This morning, I noticed that one of the machines was behaving strangely, so I rebooted it, just to be safe - a manual STONITH, if you will.
After rebooting, three of the other nodes recognize the rebooted machine as "Connected" in
gluster pool list and
gluster peer status, but the other two show a status of "Disconnected". Strangely, even the nodes which show "Disconnected" in
gluster pool list still show it as "Connected" in
gluster volume heal [volname] info.
gluster peer probe from both sides with no effect. I have verified that I can connect to both port 24007 and port 49154 (the brick port shown in
gluster volume status on the peers which recognize it as Connected) on the rebooted node from the machines which consider it to be "Disconnected".
/var/log/glusterfs/glustershd.log on the nodes which see the rebooted server as Disconnected contains:
[2018-01-09 11:36:39.258109] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-palantir-client-4: disconnected from palantir-client-4. Client process will keep trying to connect to glusterd until brick's port is available [2018-01-09 11:36:50.074074] E [socket.c:2309:socket_connect_finish] 0-palantir-client-4: connection to xxx.xxx.xxx.205:24007 failed (No route to host)
However, an hour and a half later, it has not reconnected, despite the first log entry's claim that it would keep trying.
Given all this, what do I need to do to get the two errant peers to reconnect to the rebooted node?