12

Beside our internal IT infrastructure, we've got around 500 Linux machines hosting our services for the on-line world. They are grouped in a bunch of clusters like Database A-n, Product A-n, NFS, Backoffice and so on. Furthermore, they are administered by an external provider, according to our specifications and requirements.

However, we face a lot of trouble during (web-) software development, roll-out and deploy - especially because the dev- and staging-environments have almost nothing in common with the live systems (I spare out the nasty details..).

Thus, I've tried to create virtual machines, copied the various live-systems as exactly as possible and prepared them to connect to e.g. the development-databases instead of the "real" ones transparently for developers (they aren't root). This works pretty well, but...

I was wondering how one could administer those systems remotely and in bulk? Is there some software family I'm not aware of? Or, at least, some techniques or principles one should be familiar with?

We would provide every developer with a bunch of images to be run locally (VirtualBox). The QA dept. would get virtual clusters (XEN or Hyper-V). If I need to provide an additional server-module, re-route a new database connection or just want to update everything provided by the package manager... how could I possibly do that without being forced to log on to every system and/or ask my colleagues to download and run some fixture-script?

I believe there are plenty of solutions. Well, somehow I'm too stupid to enter the correct keywords into the search engines... Or isn't this issue as trivial as it sounds?

For the record:

  • Almost all systems are running Debian GNU/Linux 6.x "squeeze"
  • No developer is forced to use a particular OS at his/her workstation
  • The budget is limited, of course, but not too small to buy proprietary software
  • A solution that would involve our aforementioned provider is preferred
Improve this question

4 Answers 4

Reset to default
17

It depends what exactly you need and what you are looking for. But in general there exists multiple solutions for "configuration management like:

  1. puppet
  2. chef
  3. cfengine
  4. ansible
  5. salt

etc. I personally would recommend puppet as it has a big community and a lot of external provided recipes. This allows you to configure and manage systems automatically. If you combine this with own repositories and automated updates via e.g. unattended-upgrades you can automatically update the system.

Another solution is just to provide your own packages like company-base etc. which automatically depends on the necessary software and can configure your system automatically.

You should also look into automates deployments (barebone and virtualized). If you combine this with configuration management or your own repository you can easily automate and reinstall your systems. If you want to get started with automated installation have a look at theforman which supports libvirt as well as bare bone installations and has integrated puppet support. If you want do do it yourself you can look into kickstart (redhat et. al.) or "preseeding" to automatically configure your system. For Debian you can also use something like debootstrap or a wrapper named grml-debootstrap supporting virtualized images.

To help providing the VirtualBox images for your developer have a look at vagrant it allows you to automate the creation of virtualized systems with VirtualBox supporting chef, puppet and shell scripts to customize your virtual environment.

If you want to use the solution by your existing provider you should ask them how they manage your systems but it will probably be some kind of configuration managment. It may be possible to run their agent on your systems if you can access the configuration server.

For google keywords look into devops, configuration management, it automation and server orchestration.

In short automate as much as possible and don't even think about doing stuff manual.

Improve this answer
5
  • 1
    Thank you! That's a lot of stuff to read through, but looks pretty promising.
    – mjhennig
    Jun 9, 2012 at 23:12
  • @mjhennig i just added some more information also re. deployment. There are plenty of resources available but most important you should not really do stuff on your own via ssh/distributed shells but have some kind of system for it.
    – Ulrich Dangel
    Jun 9, 2012 at 23:20
  • 2
    There is nothing wrong with doing stuff on your own, especially if the available tools do not fit the purpose. Systems like Puppet are largely system administration tools, not configuration management, per se. A large majority of the systems that I use are not even accessible from a central server, but from users' laptops over (of all things) vpn - been trying to change this for three years. Our IT department is split by regions because each region can't access the other properly. Homegrown scripts are necessary in some situations. That's also where innovation starts.
    – Arcege
    Jun 10, 2012 at 0:47
  • 3
    @Arcege i just upvoted your comment, scripts are necessary and you don't have to convert your whole infrastructure at once. The most important part is to automate things and make it repeatable. But the descriptive nature of puppet and chef have some unique aspects, e.g. you can test&verify puppet classes with cucumber-puppet. Of course you can develop/grow your own framework reusing existing components but it sounded OP has nothing in place currently and if you start from scratch i think it's best to use an existing framework.
    – Ulrich Dangel
    Jun 10, 2012 at 1:20
  • Yes, I agree with automated/repeatable operations. I have a number of automated or push-button home-grown scripts to interface between existing systems like Jenkins/Oc4j and Subversion/Bugzilla integration. I just didn't agree (strongly) with your comment "you should not really do stuff on your own". Sometimes existing frameworks are not applicable, as in my situation I described.
    – Arcege
    Jun 10, 2012 at 1:47
3

Ulrich already gave the answer regarding software deployment and automated server setup.

The principles behind this are

  • Define how your servers should look like - this includes common software that is installed by default, the partitioning scheme and the filesystem-layout
  • Production, staging, test and developement servers should not differ regarding these basic standards (else you will run into problems later on - as you did)
  • Use a proper change-management to document ALL changes you did (including tiny one-line-changes in any configuration)
  • Always do you change first in test, then in developement, then in staging and last in production

You asked for a handy tool to manage masses of servers - my personal favorite is cluster-ssh (cssh). Type once and do changes on many servers simultaneously.

If you discover a problem and have a fix for it that removes the problem:

  1. Apply the fix to Test/Dev/Staging/Prod (see above) if it really works
  2. Apply the fix to your virtual templates so future VM-clones will not have that bug
  3. Apply the fix to your physical installation process (kickstart/autoyast/whatever)
  4. Apply the fix to ALL servers

If you are facing masses of servers to fix this is a process that has to be well documented and at the end a different team should check if the fix has been completely applied.

We employ Mantis (open source, PHP) for that purpose.

Improve this answer
2

I manage about 30 products and a few hundred servers in multiple countries. I'm the software configuration manager, so I do not have root access (by design), don't touch the databases or their servers (again, by design) and have to jump a lot of hoops because of corporate security. But I do manage the configurations in test, staging and production, including database links and changes. I have a number of scripts that go out to servers using combinations of ssh, python and shell scripts.

The primary things to think about are:

  1. What kinds of interactions are you going to have with your servers? Just file uploads? Running command-line programs? Running remote X clients?
  2. What level of security is needed to access these servers? Firewalls, secure networks, vpn? Is ssh sufficient and from a central secure location?
  3. How much can be automated on each server? Can you install a program on each server and run it, or do you need to stream the program through something like ssh to run it remotely? Can you script it with expect or just a command-line invocation?

VirtualBox gives a lot of command-line tools that you could administer through just ssh or systems like puppet as Ulrich mentions.

Improve this answer
3
  • 2
    Just a small suggestion re. virtualbox, have a look at vagrantup.com it can simplify and automate the creation of virtual images.
    – Ulrich Dangel
    Jun 10, 2012 at 1:23
  • Unfortunately, even getting simple network access between remote test environments is near impossible. Setting up a virtualbox farm would be even harder. I have trouble just asking IT to update standard software with what's been out of date for more than are year because it is not part of the 'standard RedHat' repositories.
    – Arcege
    Jun 10, 2012 at 1:57
  • What helped in my experience wrt outdated software is to either show that the software is either EOL or there are security problems. The network setup/connections are often much harder to achieve, maybe try to emphasize how connecting the different test environments help save money, simplify processes, save QA time or make the test environment more realistic. It could also help if you get people from the different branches on-board.
    – Ulrich Dangel
    Jun 10, 2012 at 2:24
0

Over the past couple years a new set of minimalist systems have appeared that focus on the ability to build a configuration using scripts. Because they are frameworks for writing scripts they tend to excel at tasks that require sequencing.

pyinfra

pyinfra automates infrastructure super fast at massive scale. It can be used for ad-hoc command execution, service deployment, configuration management and more

http://pyinfra.com/

Judo

Simple orchestration & configuration management

http://github.com/rollcat/judo

rset(1) : pln(5)

Configure systems using any scripting language

http://scriptedconfiguration.org/

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .