Unless I'm misunderstanding something, this is a very common scenario, and well supported in linux.
Currently the best tool for the job is Pacemaker. Pacemaker sits on top of Corosync and runs on all the nodes in the cluster. It sends heartbeat messages around, and when it detects that a node has failed, it migrates all resources off that node, and can optionally kill the node (referred to as fencing).
Resources are things like a VIP, or a MySQL database. Some resources support master/slave modes, in which if the node running the master fails, pacemaker will promote the slave to master.
Resources are driven through scripts, so if it can be scripted, pacemaker can support it. There are a large number of resources already created, so you likely do not need to do this.
It also supports ordering so that when resources are moved around, it can do things like make sure that MySQL has been promoted to master before moving the VIP so that no clients connect to it before it's ready.