I have a configuration with 5 nodes, 4 able to host resources and 1 resource. What I want to do is to make the resource stop completely after migrating N times from node to node.
So a sample scenario would be: resource VIP1 running on node
one; its monitor fails,
migration-threshold is reached, it moves to node
two; then it fails there again, and moves to node
three; and then it fails again, but some kind of rule (that I'm searching for) prevents it to migrate to node
four. I need some kind of migration counter, I guess.
This is a test configuration, in reality I'll have more nodes and more resources. So I'd like the configuration be as simple as possible. I figure that I can achieve this using an opt-in or opt-out location preferrance strategy for each resource, but it's neither simple nor maintainable easily. Is there a better way?
P.S. With current configuration, if I flood my VIP with
hping, it will cycle through all nodes before it stops completely (because it has no more nodes to run on, until I cleanup it or introduce a
[~] corosync -v Corosync Cluster Engine, version '2.3.6' Copyright (c) 2006-2009 Red Hat, Inc. [~] crm --version crm 2.2.0 [~] cat /etc/debian_version 8.5
[~] crm configure show node 1: one node 2: two node 3: three node 4: arbitr \ attributes standby=on node 5: four primitive VIP1 IPaddr2 \ params ip=10.10.11.230 cidr_netmask=22 \ op monitor interval=1s timeout=40ms \ meta migration-threshold=1 target-role=Started location preferOne VIP1 50: one property cib-bootstrap-options: \ dc-version=1.1.14-70404b0 \ cluster-infrastructure=corosync \ cluster-name=debian \ stonith-enabled=false \ last-lrm-refresh=1474984960 \ have-watchdog=false
EDIT: The context and the story of the whole configuration is this: we have two load balancers with LVS that proxy traffic to our frontends. High-availability used to be achieved with a simple keepalived configuration and 2 VIPs (an active-active configuration; in a normal situation, each LVS has 1 VIP). But some guys lately started to DDoS us regularly, and the keepalived setup no longer works as expected: even if they DDoS only only 1 VIP, i.e. 1 host, its keepalived loses inbound traffic from another host, grabs the second VIP and cripples the whole system.
So we thought that we a) find something to manage VIPs with quorum, to avoid split-brains; b) at least double the LVS hosts.
So I tested pacemaker+corosync, and its configuration is really simple when we have two active and one standby nodes. Just as expected, when one of the VIP is DDoSed, that VIP first migrates to another node, and then shuts down completely. As we add nodes, that 'resource walk' becomes longer, which is not quite what we want: we want the VIP to be shut down as soon as possible, but also we of course want to leave a chance to automatically recover from an inner hardware problem (if the node suddenly goes down, FE). So I thought... maybe there's a way to migrate a resource just ONCE, and if still fails, then be done with it. Saves us from hardware failures; saves us time when we're under partial DDoS.
That's pretty much it, sorry if it is too elaborate. If we chose the wrong instrument to deal with this from the start, please let me know. (Of course, we do defend from DDoS with other methods, but if the attack manages to get through, we want to minimize the damage).