0

I'm creating a cluster system using two ESXi hosts, with a CentOS 7 server on each.

Going through I created the filesystem, and it mounts on node1.

When I perform a standby or reboot from node01 to node02 the failover works as it should. However, if I perform it from node02 back to node01 it returns a resource error about failing to mount the filesystem under /mbdata

I am receiving this message:

Failed Resource Actions:
* mb-drbdFS_start_0 on node01 'unknown error' (1): call=75, status=complete, exitreason='Couldn't mount device [/dev/drbd0] as /mbdata',
    last-rc-change='Thu May  7 16:09:25 2020', queued=1ms, exec=129ms

When I clean the resources, and node02 is online it starts running again. I have googled to see why I am getting this error, but the only thing I can see is that the server is not notifying the new master that is in fact the master (not slave). But I haven't found anything to help me to activate this.

I have tried umount on both systems - but usually get on node02 that it is not mounted. I have tried mounting the system on both (but then one is read-only and defeats the purpose of the cluster controlling it). I was following a tutorial in the beginning but they didn't list having the error - they just said it kicks over to the new node so I'm lost!

The only difference I have done is not use /mnt as the destination, but my own directory - but I didn't think that would be the problem.

What I'm trying to have is:

  • have a fence on each ESXi host (physical server, to reboot it's own VM)
  • have a DRBD storage so I can have shared storage
  • have a virtual IP for client access
  • have Apache to run the web server
  • have MariaDb for the SQL database
  • run them on the same servers (colocation) and have the other as full standby

enter image description here

When it does run I have:

[root@node01 ~]# pcs status
Cluster name: mb_cluster
Stack: corosync
Current DC: node01 (version 1.1.21-4.el7-f14e36fd43) - partition with quorum
Last updated: Fri May  8 09:46:59 2020
Last change: Fri May  8 09:22:59 2020 by hacluster via crmd on node01

2 nodes configured
8 resources configured

Online: [ node01, node02 ]

Full list of resources:

mb-fence-01 (stonith:fence_vmware_soap):        Started node01
mb-fence-02 (stonith:fence_vmware_soap):        Started node02

Master/Slave Set: mb-clone [mb-data]
    Masters: [ node01 ]
    Slaves: [ node02 ]

Resource Group: mb-group
    mb-drbdFS   (ocf::heartbeat:Filesystem):    Started node01
    mb-vip      (ocf::heartbeat:IPaddr2):   Started node01
    mb-web      (ocf::heartbeat:apache):    Started node01
    mb-sql      (ocf::heartbeat:mysql):     Started node01

      Daemon Status:
          corosync: active/enabled
          pacemaker: active/enabled
          pcsd: active/enabled

And the constraints:

[root@node01 ~]# pcs constraint list --full
Location Constraints:
    Resource: mb-fence-01
    Enabled on: node01 (score:INFINITY) (id:location-mb-fence-01-node01-INFINITY)

    Resource: mb-fence-02
    Enabled on: node02 (score:INFINITY) (id:location-mb-fence-02-node02-INFINITY)

Ordering Constraints:
    start mb-drbdFS then start mb-vip (kind:Mandatory) (id:order-mb-drbdFS-mb-vip-mandatory)
    start mb-vip then start mb-web (kind:Mandatory) (id:order-mb-vip-mb-web-mandatory)
    start mb-vip then start mb-sql (kind:Mandatory) (id:order-mb-vip-mb-sql-mandatory)
    promote mb-clone then start mb-drbdFS (kind:Mandatory) (id:order-mb-clone-mb-drbdFS-mandatory)

Colocation Constraints:
    mb-drbdFS with mb-clone (score:INFINITY) (with-rsc-role:Master) (id:colocation-mb-drbdFS-mb-clone-INFINITY)
    mb-vip with mb-drbdFS (score:INFINITY) (id:colocation-mb-vip-mb-drbdFS-INFINITY)
    mb-web with mb-vip (score:INFINITY) (id:colocation-mb-web-mb-vip-INFINITY)
    mb-sql with mb-vip (score:INFINITY) (id:colocation-mb-sql-mb-vip-INFINITY)

Ticket Constraints:
  • @A.B the filesystem is XFS, and the STONITH is set so each node is for each physical ESXi server (no vCentre) – markb May 7 at 7:14
  • @A.B but if I reboot / shutdown a node shouldn't the mount go? and then be available? which is not happening. i just tested again and I can go with node1 as master, shutdown, it fails over to node2 fine. then launch node1 online, and wait 5 minutes. shutdown node1 and get the error above – markb May 7 at 7:24
  • Check the pacemaker logs around the timestamp of the error. It should give you some more output that will hint to the cause of the failure to mount. Also, verify that the DRBD resources are connected and healthy. – Dok May 7 at 15:01
1

You have no ordering constraint telling the cluster to start the Filesystem only after the DRBD device is promoted to master. Add the following order constraint:

# pcs constraint order promote data then start drbd-FS
| improve this answer | |
  • thanks @Matt, I updated my question as I think I missed a part when copy and pasting from notepad. I already have the order but from the clone to FS – markb May 8 at 0:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.