Beside our internal IT infrastructure, we've got around 500 Linux machines hosting our services for the on-line world. They are grouped in a bunch of clusters like Database A-n, Product A-n, NFS, Backoffice and so on. Furthermore, they are administered by an external provider, according to our specifications and requirements.
However, we face a lot of trouble during (web-) software development, roll-out and deploy - especially because the dev- and staging-environments have almost nothing in common with the live systems (I spare out the nasty details..).
Thus, I've tried to create virtual machines, copied the various live-systems as exactly as possible and prepared them to connect to e.g. the development-databases instead of the "real" ones transparently for developers (they aren't
root). This works pretty well, but...
I was wondering how one could administer those systems remotely and in bulk? Is there some software family I'm not aware of? Or, at least, some techniques or principles one should be familiar with?
We would provide every developer with a bunch of images to be run locally (VirtualBox). The QA dept. would get virtual clusters (XEN or Hyper-V). If I need to provide an additional server-module, re-route a new database connection or just want to update everything provided by the package manager... how could I possibly do that without being forced to log on to every system and/or ask my colleagues to download and run some fixture-script?
I believe there are plenty of solutions. Well, somehow I'm too stupid to enter the correct keywords into the search engines... Or isn't this issue as trivial as it sounds?
For the record:
- Almost all systems are running Debian GNU/Linux 6.x "squeeze"
- No developer is forced to use a particular OS at his/her workstation
- The budget is limited, of course, but not too small to buy proprietary software
- A solution that would involve our aforementioned provider is preferred