0

I implemented DRBD resource level fencing with crm-fence-peer.9.sh and crm-unfence-peer.9.sh scripts on both nodes.

enter image description here

Now, I have the following situation on my lab nodes:

  • both nodes otrs1 and otrs2 are online

  • resource are running on otrs1

  • as per drbdadm status otrs1 holds the primary role and otrs2 holds the secondary role

enter image description here

Now when I reboot otrs1 on otrs2 I get the following message:

enter image description here

Can see that the resources are moved to otrs2:

enter image description here

I can see the location constraint create: enter image description here

If the replication link becomes connected again and DRBD completes its synchronization process, then the constraint is removed. The cluster manager is now free to promote the resource. In fact the constraint is now removed:

enter image description here

But as soon I disable the NIC on otrs2 (the currently active node) I can see the split-brain occurred:

enter image description here

Obviously this is a split-brain scenario. Why's that so ? is it because

In case of the crm-fence-peer script it is necessary that Pacemakers communication stays available when DRBD’s network link breaks.

Source https://docs.linbit.com/docs/users-guide-9.0/#s-automatic-split-brain-recovery-configuration

1

Correct. It is most likely because of:

In case of the crm-fence-peer script it is necessary that Pacemakers communication stays available when DRBD’s network link breaks.

I am assuming you only have the one NIC/network link. So, when you take down the NIC the pacemaker cluster splits. As the cluster nodes can no longer communicate at all, the current primary node cannot place the constraint in the peer's CIB as it cannot communicate with the peer.

To avoid split-brains in this situation you will either need true node level fencing (STONITH), or at the very least, multiple communication paths for Corosync.

| improve this answer | |
  • Thank you very much! Just want to put into consideration that in production both nodes would run as guests virtualized with VMWare. Is there any node level fencing agent for that ? Could you possibly point my to any documentation ? – blabla_trace Feb 21 '19 at 18:23
  • PS. yes you are correct there's only one NIC in each node but what I don't get is why does that same not happen when the node gets rebooted ? I mean the connection is interrupted as well, when the other node reboots it can't have its CIB update either, right ? – blabla_trace Feb 21 '19 at 18:29
  • 1
    How is the node rebooted? Is it a graceful reboot, or an abrupt power-cycle of the node? The behavior would be very different depending. I don't use vmware myself, but I know others have used the fence_vmware_soap STONITH agent with success. – Dok Feb 21 '19 at 21:10
  • I was using the reset functionality within VirtualBox. – blabla_trace Feb 21 '19 at 21:20
  • PS. why would it make a difference? Could you please explain ? Thank you! – blabla_trace Feb 21 '19 at 21:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.