7

Using pacemaker in a 2 nodes master/slave configuration.
In order to perform some tests, we want to switch the master role from node1 to node2, and vice-versa. For instance if the current master is node1, doing

# crm resource migrate r0 node2

does indeed move the resource to node2. Then, ideally,

# crm resource migrate r0 node1

would migrate back to node1. The problem is that migrate added a line in the configuration to perform the switch

location cli-prefer-r0 r0 role=Started inf: node2

and in order to migrate back I have first to remove that line...

Is there a better way to switch master from one node to the other?

  • Have you tried moving the resource instead of migrating? – YoMismo Dec 2 '14 at 9:56
  • Do you have active/active or active/passive setup? if it is active/active you can just bring one node to standby and check if the resources are moved to the other node. – YoMismo Dec 2 '14 at 10:26
  • @YoMismo Active/passive. What is the difference between migrate and move? – Ring Ø Dec 2 '14 at 23:44
  • It's been a long time since I had my cluster running, I've been checking crm's man page and migrate/move appear to be equivalent, but clusterlabs.org/doc/en-US/Pacemaker/1.1-plugin/html/… uses moving instead of migrating, for documentation I would use clusterlabs, since it is (or was when I set my cluster up) project's main page. If you have active/passive then it is easy for you, activate the second node and put in standby the first one, resources will have to move automatically. Have you checked the documentation? clusterlabs.org/doc – YoMismo Dec 3 '14 at 10:08
5

I know this bit old; but it seems like no one answered this satisfactorily, and the requester never posted if his problem was solved or not. So here is an explanation.

When you perform:

# crm resource migrate r0 node2

a cli-prefer-* rule is created.

Now when you want to move the r0 back to node1, you don't do:

# crm resource migrate r0 node1

but you perform:

# crm resource unmigrate r0

Using umigrate or unmove gets rid of the cli-prefer-* rule automatically.

If you try to delete this rule manually in cluster config, really bad things happen in cluster, or at least bad things happened in my case.

  • I know this is quite old, but what "bad things" happened when you deleted the rule? Does using "unmigrate" move the resource to another node, or just cleanly remove the prefer rules? – Michael Firth Jul 16 '18 at 10:40
  • Unmigrate removes the location constraint that is keeping a given resource from running on the "migrated from" node - it does not perform any move operations of its own, but just make them possible. If a resource has a preference for that node, it will try to move back if ordering/collocation constraints allow the resource to do so. – Spooler Aug 16 '18 at 21:48
1

Oh the joys.

This all boiled down to wether using migrate commands with or without target nodes. From the current cli documentation via crm resource help migrate:

If the destination node is left out, the resource is migrated by creating a constraint which prevents it from running on the current node. For this type of constraint to be created, the force argument is required.

In the past this could also be done without force, and when you ran a two-node-cluster (two nodes never constitute a proper cluster but lets not digress here), you ended up with a statement in pacemakers running config telling it explicitly to not run on the node it ran on when the migrate command was fired, so pacemaker would force it to move away onto another node.

If you ran a crm resource migrate you ended up with a location cli-... statement up in your pacemaker config. There are two types of such statements, which can be checked for rather easily:

crm configure show | grep -e cli-prefer -e cli-ban

For serious business, the command above should also be part of nagios/icinga/whatever monitoring check, as these manual constraints should never be present in running configuration in the long term anyway.

If you migrated again back to another node, you ended up with yet another such statement, marking the last node ALSO as a "non-runnable" location for the resource. If you had eventually all your nodes 'outmigrated' in such fashion... your resource in question would not start anywhere and you were out of luck, which is likely the reason this question was asked in the first place.

Solution was to ALWAYS have an crm resource migrate command being followed by an crm resource unmigrate command when omitting target nodes, to not shoot yourself in the foot eventually later.


If you however always specify a target node, all this does not matter:

Upon rerunning a crm resource migrate RESOURCE NODE / pcs resource move RESOURCE NODE the cli-prefer... statement within the pacemaker config gets adjusted, so no worries.

So you can run crm resource migrate / pcs resource move as often as you like without running into trouble of resources not being able to start anymore.

To get rid of any of these manual constraints for a resource in particular, run one of these:

crm resource unmigrate RESOURCE
crm resource clear RESOURCE

pcs resource clear RESOURCE

This however will not make your resources magically to their intial hosts, if you did not create location constraints in the first place - unlike the other answer here suggests.

  • In recent crm crm configure show cli-\* also works (no need for grep -e cli-prefer -e cli-ban). – U. Windl 2 days ago
1

One solution is to remove those cli-prefer constraints manually before you try to migrate back, or (if you don't care about inactive constraints lingering around) add a time period like 5 minutes:

crm resource migrate r0 node2 PT5M

After 5 minutes the rule is no longer effective (and r0 might migrate back), but it is still visible.

Also note that in a two-node cluster you do not need to specify the other node.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.