1

I have a configuration with 5 nodes, 4 able to host resources and 1 resource. What I want to do is to make the resource stop completely after migrating N times from node to node.

So a sample scenario would be: resource VIP1 running on node one; its monitor fails, migration-threshold is reached, it moves to node two; then it fails there again, and moves to node three; and then it fails again, but some kind of rule (that I'm searching for) prevents it to migrate to node four. I need some kind of migration counter, I guess.

This is a test configuration, in reality I'll have more nodes and more resources. So I'd like the configuration be as simple as possible. I figure that I can achieve this using an opt-in or opt-out location preferrance strategy for each resource, but it's neither simple nor maintainable easily. Is there a better way?

P.S. With current configuration, if I flood my VIP with hping, it will cycle through all nodes before it stops completely (because it has no more nodes to run on, until I cleanup it or introduce a failure-timeout option)

Versions:

[~] corosync -v
Corosync Cluster Engine, version '2.3.6'
Copyright (c) 2006-2009 Red Hat, Inc.
[~] crm --version
crm 2.2.0
[~] cat /etc/debian_version
8.5

Pacemaker configuration:

[~] crm configure show
node 1: one
node 2: two
node 3: three
node 4: arbitr \
        attributes standby=on
node 5: four
primitive VIP1 IPaddr2 \
        params ip=10.10.11.230 cidr_netmask=22 \
        op monitor interval=1s timeout=40ms \
        meta migration-threshold=1 target-role=Started
location preferOne VIP1 50: one
property cib-bootstrap-options: \
        dc-version=1.1.14-70404b0 \
        cluster-infrastructure=corosync \
        cluster-name=debian \
        stonith-enabled=false \
        last-lrm-refresh=1474984960 \
        have-watchdog=false

EDIT: The context and the story of the whole configuration is this: we have two load balancers with LVS that proxy traffic to our frontends. High-availability used to be achieved with a simple keepalived configuration and 2 VIPs (an active-active configuration; in a normal situation, each LVS has 1 VIP). But some guys lately started to DDoS us regularly, and the keepalived setup no longer works as expected: even if they DDoS only only 1 VIP, i.e. 1 host, its keepalived loses inbound traffic from another host, grabs the second VIP and cripples the whole system.

So we thought that we a) find something to manage VIPs with quorum, to avoid split-brains; b) at least double the LVS hosts.

So I tested pacemaker+corosync, and its configuration is really simple when we have two active and one standby nodes. Just as expected, when one of the VIP is DDoSed, that VIP first migrates to another node, and then shuts down completely. As we add nodes, that 'resource walk' becomes longer, which is not quite what we want: we want the VIP to be shut down as soon as possible, but also we of course want to leave a chance to automatically recover from an inner hardware problem (if the node suddenly goes down, FE). So I thought... maybe there's a way to migrate a resource just ONCE, and if still fails, then be done with it. Saves us from hardware failures; saves us time when we're under partial DDoS.

That's pretty much it, sorry if it is too elaborate. If we chose the wrong instrument to deal with this from the start, please let me know. (Of course, we do defend from DDoS with other methods, but if the attack manages to get through, we want to minimize the damage).

  • @JeffSchaller, thanks for edit, didn't think of 'cluster' tag. – Pavel Gurkov Sep 27 '16 at 14:46
  • Judging by the available resource meta options available this may not be directly possible: access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/… – Centimane Sep 27 '16 at 16:53
  • @Centimane, I've browsed all the docs I could find and yeah, no luck. I thought maybe I'm missing something. If you can think of a viable workaround that's better than opt-in/opt-out locations strategy, please let me know. – Pavel Gurkov Sep 27 '16 at 17:02
  • @PavelGurkov Unfortunately, I think your opt-in/out idea is probably the only way to achieve this. Otherwise Pacemaker will try every node until it has at least one failure marked on all. Perhaps there is some context for your project that you can share that might inspire some other solutions? – Matt Kereczman Sep 28 '16 at 14:36
  • @MattKereczman, thanks for your comment. I've edited the question to add the background and the context. – Pavel Gurkov Sep 29 '16 at 9:40
1

That is definitely an interesting use case, thanks for sharing...

While you're VIPs are getting DDoS'd they probably can't ping out reliably. Perhaps you might take a look at the 'ping' resource-agent for Pacemaker.

The clusterlabs documentation mention it briefly here: http://clusterlabs.org/doc/en-US/Pacemaker/1.0/html/Pacemaker_Explained/ch09s03s03.html

More information can be found by checking out the resource-agent's info via your preferred cluster configuration management tool:

# crm ra info ping
--or--
# pcs resource describe ping

Hope that is helpful.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.