MySQL Server monitor_20000 on node1 'not running' - HA Cluster - Pacemaker - Corosync - DRBD

Question

I get this Error after finishing configuring cluster. And after trying to fail-BACK from node2

mysql_service01_monitor_20000 on node1 'not running' (7): call=20, status=complete, exitreason='none'

shutdown cluster & restart mariadb

pcs cluster stop --all

service mariadb restart

service mariadb stop

pcs cluster start --all

Everything comes online.

pcs cluster standby node1

Fails over to node2. But I get this error again...

mysql_service01_monitor_20000 on node1 'not running' (7): call=77, status=complete, exitreason='none'

Try to Failback to node1

pcs cluster unstandby node1
pcs cluster standby node2

Doesn't fail back over and brings up the following:

Failed Actions:
* mysql_service01_monitor_20000 on node2 'not running' (7): call=141, status=complete, exitreason='none',
last-rc-change='Mon Jun 13 20:33:36 2016', queued=0ms, exec=43ms
* mysql_service01_monitor_20000 on node1 'not running' (7): call=77, status=complete, exitreason='none',
last-rc-change='Mon Jun 13 20:31:23 2016', queued=0ms, exec=42ms
* mysql_fs01_start_0 on node1 'unknown error' (1): call=113, status=complete, exitreason='Couldn't mount filesystem /dev/drbd0 on /var/lib/mysql',
last-rc-change='Mon Jun 13 20:33:47 2016', queued=0ms, exec=53ms

After shutting down cluster and restarting MariaDB again on both nodes I get this up starting up again.

mysql_service01    (ocf::heartbeat:mysql): FAILED node1

After a full pcs cluster stop --all (successfull) and reboot and pcs cluster start --all. Everything works!

It's sort of haphazard, but it really is HA, and I have e-mail notification of fail-over setup, so hopefully we could finish the day on backup, shutdown and restart services back on node1. But I would love to know whats going on, and how to stop this, it will sure make the demo to my boss's look bad.

My Configs :

Disable Firewall/SELinux

sed -i 's/\(^SELINUX=\).*/\SELINUX=disabled/' /etc/selinux/config
systemctl disable firewalld.service
systemctl stop firewalld.service
iptables --flush
reboot

Install PaceMaker + Corosync (CentOS 7)

hostnamectl set-hostname $(uname -n | sed s/\\..*//)
yum install -y pcs policycoreutils-python psmisc
echo "passwd" | passwd hacluster --stdin
systemctl start pcsd.service
systemctl enable pcsd.service

Authorize on Node1

pcs cluster auth node1 node2 -u hacluster -p passwd
pcs cluster setup --force --name mysql_cluster node1 node2
pcs cluster start --all
pcs status | grep UNCLEAN

Install DRBD/MariaDB :

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
yum install -y kmod-drbd84 drbd84-utils mariadb-server mariadb
systemctl disable mariadb.service

cat << EOL > /etc/my.cnf
[mysqld]
symbolic-links=0
bind_address            = 0.0.0.0
datadir                 = /var/lib/mysql
pid_file                = /var/run/mariadb/mysqld.pid
socket                  = /var/run/mariadb/mysqld.sock

[mysqld_safe]
bind_address            = 0.0.0.0
datadir                 = /var/lib/mysql
pid_file                = /var/run/mariadb/mysqld.pid
socket                  = /var/run/mariadb/mysqld.sock

!includedir /etc/my.cnf.d
EOL

Drbd Res :

cat << EOL >/etc/drbd.d/mysql01.res
resource mysql01 {
 protocol C;
 meta-disk internal;
 device /dev/drbd0;
 disk   /dev/sdb1;
 handlers {
  split-brain "/usr/lib/drbd/notify-split-brain.sh root";
 }
 net {
  allow-two-primaries no;
  after-sb-0pri discard-zero-changes;
  after-sb-1pri discard-secondary;
  after-sb-2pri disconnect;
  rr-conflict disconnect;
 }
 disk {
  on-io-error detach;
 }
 syncer {
  verify-alg sha1;
 }
 on node1 {
  address  192.168.1.216:7788;
 }
 on node2 {
  address  192.168.1.220:7788;
 }
}
EOL

fdisk /dev/sdb

drbdadm create-md mysql01
modprobe drbd
drbdadm up mysql01

drbdadm -- --overwrite-data-of-peer primary mysql01
drbdadm primary --force mysql01
watch cat /proc/drbd
mkfs.ext4 /dev/drbd0
mount /dev/drbd0 /mnt
df -h | grep drbd
umount /mnt
mount /dev/drbd0 /mnt # I Always get IO Errors so I just
drbdadm up mysql01 # Both nodes
watch cat /proc/drbd
mount /dev/drbd0 /mnt
df -h | grep drbd
systemctl start mariadb
mysql_install_db --datadir=/mnt --user=mysql
umount /mnt
systemctl stop mariadb

PaceMaker Corosync Config :

pcs -f clust_cfg resource create mysql_data01 ocf:linbit:drbd \
  drbd_resource=mysql01 \
  op monitor interval=30s
    pcs -f clust_cfg resource master MySQLClone01 mysql_data01 \
  master-max=1 master-node-max=1 \
  clone-max=2 clone-node-max=1 \
  notify=true
    pcs -f clust_cfg resource create mysql_fs01 Filesystem \
  device="/dev/drbd0" \
  directory="/var/lib/mysql" \
  fstype="ext4"
    pcs -f clust_cfg resource create mysql_service01 ocf:heartbeat:mysql \
  binary="/usr/bin/mysqld_safe" \
  config="/etc/my.cnf" \
  datadir="/var/lib/mysql" \
  pid="/var/lib/mysql/mysql.pid" \
  socket="/var/lib/mysql/mysql.sock" \
  additional_parameters="--bind-address=0.0.0.0" \
  op start timeout=60s \
  op stop timeout=60s \
  op monitor interval=20s timeout=30s
    pcs -f clust_cfg resource create mysql_VIP01 ocf:heartbeat:IPaddr2 \
 ip=192.168.1.215 cidr_netmask=32 nic=eth0 \
 op monitor interval=30s
    pcs -f clust_cfg constraint colocation add mysql_service01 with mysql_fs01 INFINITY
    pcs -f clust_cfg constraint colocation add mysql_VIP01 with mysql_service01 INFINITY
    pcs -f clust_cfg constraint colocation add mysql_fs01 with MySQLClone01 INFINITY with-rsc-role=Master
    pcs -f clust_cfg constraint order mysql_service01 then mysql_VIP01
    pcs -f clust_cfg constraint location mysql_fs01 prefers node1=50
    pcs -f clust_cfg property set stonith-enabled=false
    pcs -f clust_cfg property set no-quorum-policy=ignore
    pcs -f clust_cfg resource defaults resource-stickiness=200
    pcs -f clust_cfg resource group add SQL-Group  mysql_service01  mysql_fs01 mysql_VIP01
    pcs cluster cib-push clust_cfg
    pcs status

Update to Comment :

Would this suffice, I assume I want the Clone before the FS and the FS before the Service. Also, with my Apache configs, which I am copying from, I have the VIP start before the webserver, but in the guide I am following for SQL, it has the VIP start first. Any thoughts?

pcs -f clust_cf constraint order promote MySQLClone01 then start mysql_fs01
pcs -f clust_cf constraint order mysql_fs01 then mysql_service01

I Will test and get back if it fixed it! Thanks

This seems to have fixed the problem, fail-over happens as it should, but I still get the error, but like I said, its working good! Don't like seeing errors, but fail-over time is like 2 seconds.

pcs constraint order promote MySQLClone01 then start mysql_fs01
pcs constraint order mysql_service01 then mysql_fs01

Let me know if there are some logs to help, I am not the best at debugging logs, but I can tail them ! — FreeSoftwareServers, Commented Jun 14, 2016 at 0:57
You haven't added any ordering constraints between drbd, the filesystem, or mysql. Only between mysql and the VIP. There's nothing preventing pacemaker from trying to start mysql before the filesystem is mounted. Or to prevent the filesystem from mounting before the node is converted to DRBD master. — phemmer, Commented Jun 14, 2016 at 1:25
Does the order you input constraints effect the outcome? Or does it just do it by logic? — FreeSoftwareServers, Commented Jun 14, 2016 at 3:09
From what I have been reading (here)[clusterlabs.org/doc/en-US/Pacemaker/1.1/html/… Groups are more then an OU as I originally though, they infact are ordering constraints, so I have conflicting Ordering Constraints I think — FreeSoftwareServers, Commented Jun 14, 2016 at 3:24

Matt Kereczman · Accepted Answer · 2016-06-14 04:22:15Z

Groups imply both ordering and location. So your group is saying, "start mysql, then mount the filesystem, then start the VIP". Not only is this incorrect ordering, but it is contradicting your ordering constraints.

You should just put everything in the group besides DRBD, and then place a single ordering and single colocation constraint that ties the group to where DRBD is Master.

The order in which you add constraints to the cluster has absolutely no affect on the outcome.

Based on what you had up there, it would look something like this:

# pcs -f clust_cfg resource create mysql_data01 ocf:linbit:drbd \
  drbd_resource=mysql01 op monitor interval=30s
# pcs -f clust_cfg resource master MySQLClone01 mysql_data01 \
  master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 \
  notify=true
# pcs -f clust_cfg resource create mysql_fs01 Filesystem \
  device="/dev/drbd0" directory="/var/lib/mysql" fstype="ext4"
# pcs -f clust_cfg resource create mysql_service01 ocf:heartbeat:mysql \
  binary="/usr/bin/mysqld_safe" config="/etc/my.cnf" \
  datadir="/var/lib/mysql" pid="/var/lib/mysql/mysql.pid" \
  socket="/var/lib/mysql/mysql.sock" \
  additional_parameters="--bind-address=0.0.0.0" \
  op start timeout=60s op stop timeout=60s \
  op monitor interval=20s timeout=30s
# pcs -f clust_cfg resource create mysql_VIP01 ocf:heartbeat:IPaddr2 \
  ip=192.168.1.215 cidr_netmask=32 nic=eth0 op monitor interval=30s
# pcs -f clust_cfg resource group add SQL-Group mysql_fs01 \
  mysql_service01 mysql_VIP01
# pcs -f clust_cf constraint order promote MySQLClone01 \
  then start SQL-Group
# pcs -f clust_cfg constraint colocation add SQL-Group with MySQLClone01 INFINITY with-rsc-role=Master
# pcs cluster cib-push clust_cfg

When I do pcs -f clust_cfg constraint before pushing the CIB, without adding any "ordering" constraints it shows nothing, thats why I thought that it was just an OU. But the article I read made it seem different, so a group DOES control ordering, but doesn't show the same output when looking into constraints on the system? — FreeSoftwareServers, Commented Jun 14, 2016 at 4:07
I will definitely try your suggestion, as one Group and One Co-location sounds wicked to me, much more simple! — FreeSoftwareServers, Commented Jun 14, 2016 at 4:08
Whow, that is WAY cleaner, you are a life saver, I had it working, but it is just soooo much cleaner now. Thank you! — FreeSoftwareServers, Commented Jun 14, 2016 at 4:14
And to note, the second time I checked constraints, the cib showed the group as an order constraint, I must have made a mistake first time around. — FreeSoftwareServers, Commented Jun 14, 2016 at 4:19

FreeSoftwareServers · Accepted Answer · 2016-06-14 03:57:21Z

pcs -f clust_cfg constraint colocation add mysql_fs01 with MySQLClone01 \
INFINITY with-rsc-role=Master
pcs -f clust_cfg constraint colocation add mysql_service01 with mysql_fs01 INFINITY
pcs -f clust_cfg constraint colocation add mysql_VIP01 with mysql_service0 INFINITY
pcs -f clust_cfg constraint order promote MySQLClone01 then start mysql_fs01
pcs -f clust_cfg constraint order mysql_fs01 then mysql_service01
pcs -f clust_cfg constraint order mysql_service01 then mysql_VIP01

Thank you to Patrick for the correct path to point me!, took me a moment to figure out, it seems the order the constraints are input effects the output, I was using the above constraints, but inputting them in different order.

The order you add the constraints to the config is irrelevant. — Matt Kereczman, Commented Jun 14, 2016 at 4:07

Stack Exchange Network

MySQL Server monitor_20000 on node1 'not running' - HA Cluster - Pacemaker - Corosync - DRBD

2 Answers 2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
mariadb
pacemaker
drbd
corosync
pcs
.

Hot Network Questions

MySQL Server monitor_20000 on node1 'not running' - HA Cluster - Pacemaker - Corosync - DRBD

2 Answers 2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged mariadbpacemakerdrbdcorosyncpcs.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
mariadb
pacemaker
drbd
corosync
pcs
.