NEWS & BLOG

Maintaining Linux systems includes making changes to them – removing a user, adding a new application, updating a certificate and so on. Logging in to each server in turn and updating it is one way; using some kind of automation is another way. In this article, we are going to discuss automation using a Configuration Management System.

Buzzwords and Terminology

In a discussion of Configuration Management Systems, you may come across the following terms:

• Configuration management system: the term we will use
• Configuration management software: the software that implements a configuration management system
• Software configuration management: more accurately applied to the tracking of changes to specific software rather than the system as a whole
• Orchestration: the creation and management of computer systems, often including billing, and typically used in a cloud computing context.

We will not look at the subtle differences between each of those terms, but rather look at the central control of system configuration as a whole. For clarity, in this article the abbreviation “CMS” refers to a Configuration Management System as opposed to any other meaning.

What Is A Configuration Management System?

The principle of a CMS is straightforward. There is a central “CMS server” (possibly more than one), and there are the systems that it maintains, the “CMS clients”. Periodically, each client sends details of its configuration to the CMS server. The configuration of the client is compared against a template held on the server. If the actual configuration of the client differs from that defined in the template, the necessary changes are made to the client such that it conforms to the template.

Almost any aspect of the client configuration may be controlled by a CMS server; for example:

• the installation (or removal) of a software package
• the creation (or removal) of a user
• the addition and configuration of a new website
• a change to the firewall rules
• system tuning settings
• setting passwords

Why use a CMS?

The key ingredients of effective Linux infrastructure management include consistency, accountability and cost management, and a Configuration Management System can help greatly with all of those.

• Consistency: if we define a change that must be made to all web servers, we know that the same change will be made on each. The alternative, logging into each server individually, leaves us open to human error. The change may be made slightly differently on one server, or one server may be missed and not updated at all.
• Accountability: an audit trail of changes (who, what, when) can easily be created if the configuration data on the CMS server is stored in a version control system.
• Cost Management: the more servers that are supported, the more expensive it becomes to update each one manually. At the same time, the likelihood of human error increases, which in turn has cost implications.

We will look at each of these in more detail below.

How A CMS Might Be Used

Most CMSs allow the client systems to be considered as groups, and any one client may be a member of more than one group. For example, webserver1.example.com may be a member of the following groups:

• Web servers
• Red Hat 7
• Manchester data centre
• Dell

By way of example, suppose we want to change which DNS server is used by all systems hosted in the Manchester data centre. We update the necessary files on the CMS server, and we indicate that they should be applied to all servers in the Manchester data centre group. Over the next few minutes, the updates will be rolled out to those servers. Similarly, when a security update is released for Red Hat 7 systems, we can trigger that update on just the Red Hat 7 systems.

The granularity of server selection can be very fine. It’s possible to target a change on one server only, by membership of various groups, or every server. As well as groups, some CMSs support a hierarchical organisation of clients, and allow changes to trickle down from a particular level. One possible hierarchy might be:

• all systems
• the client the server belongs to (eg, “Acme Ltd”)
• the role of the server (eg, “web server”)
• the site where the server is hosted (eg, “Manchester DC”)
• the site-role (to allow selection of all “web server” systems in “Manchester DC”)
• the system name (eg, “webserver1.example.com”)

There is crossover between the use of groups and the use of a hierarchy, but the end result is a very flexible mechanism for targeting the systems any one change should apply to.

Abstraction is also possible. For example, to install the Apache web server on a Red Hat system, the package name to install is ‘httpd’ whereas on a Debian system it’s ‘apache2’. It’s possible to specify to the CMS that all servers in group ‘web servers’ should have Apache installed, and the CMS can take care of both the exact package name needed as well as the appropriate commands to install it on a given system.

Consistency

It can be seen that the use of a CMS can improve consistency across the server estate. No more manually logging in to fifteen web servers to install a new Apache module, and trusting that each and every installation will be the same. Now we just update the list of packages that should be installed on every web server, and sit back and let it happen.

Accountability and Audit Trail

All of the files that control client configuration may be – and should be – kept in revision control system. A simplified workflow might be:

• a request for change is logged on the ticketing system
• the required changes are staged and tested using a test branch of the revision control system and a test client system
• once finalised, a request is made to merge the test branch back to production
• the proposed change is independently checked and merged to production, and a comment or tag relating to the original ticket is added

A year later, when someone asks why systems are configured in a given way, the revision control system will show when the change in question was made. The comments in the revision control system point back to ticket XXX in the ticketing system, which in turn shows the source of the request. The answer can be given: “Joe Smith requested that be changed on ticket 23144 in May last year”.

Cost Control

There are cost savings to be realised with the effective use of a CMS in that changes to multiple target systems only have to be implemented once. The value of that cost saving will, of course, depend on how many servers are affected, but the cost saving isn’t limited just to avoiding someone logging in to half a dozen servers to install a new Apache module. With a CMS, you also know that:

• all servers have been updated, as opposed to one being omitted by mistake
• they are all updated in a reasonable timeframe, typically within half an hour (and it can be a lot quicker when needed)
• they are all updated in the same way. Any little extras that are needed – a tweak to a configuration file, the reloading of an application – have all happened on all servers.

The value of the last point should not be overlooked. Inconsistency between servers is not always easy to detect, and once it has been detected there’s a need to check that all systems have been correctly updated. A CMS gets rid of all of these points: all systems will be consistent, and even if a step has been omitted, it can be corrected in one place and rolled out to all servers.

Implementation

There are lots of CMSs available (don’t confuse Configuration Management Systems with Content Management Systems, used to manage websites). Wikipedia lists over 20 Open Source CMSs. The key players at the time of writing include:

• Ansible
• Chef
• CFEngine
• Puppet
• Salt

Here at Tiger Computing we have used Puppet extensively since 2010.

Planning

The implementation of a CMS is decidedly non-trivial. Whilst each of the systems listed above will have its pros and cons, broadly speaking they do the same thing. Arguably more important than the system used is the deciding how to organise the client servers into the right groups and/or hierarchy.

In reality, both experience of a CMS and an understanding of your organisation are required. Here’s how to get started:

• Define some requirements. What are the top ten jobs you’d like to manage via a CMS? Look at the configuration language used by various CMSs: which fit your organisation best? How scalable do you need your CMS to be – will you be managing tens or hundreds of systems, or will it be multiple thousands?
• Pick a CMS. Don’t spend hours on this; just pick what appears to be the best for you right now.
• Start small. Set up one test client and use the CMS to carry out simple tasks (install a package, create a user, update a configuration file). Each “simple” task will present challenges that need to be overcome.
• Get the first five of your ‘top ten’ jobs implemented in a test environment.
• Add a “real” server – perhaps a development or test machine – and run the CMS in “dry run” mode on it. Examine the changes the CMS wants to make. Modify the CMS configuration if needed.
• Run the CMS for real on the development system and evaluate the results.

You can continue in this way, gradually growing the number of clients and the reach of the CMS. You’ll come across issues, of course, and you may realise that, with hindsight, some of your earlier decisions weren’t the best. That’s OK: you need to go through a learning curve to understand which is the right CMS and hierarchy for you. You may end up re-implementing almost from scratch. To quote Fred Brooks in The Mythical Man-Month, you should plan to throw one away because you will anyway.

More Advanced Applications

Here’s a real-life example of how a CMS, in this case, Puppet, has been used:

A high-tech chip design company has a need, at various times during the design phase, to employ a significant number of temporary engineering staff. Each of those staff has a Linux workstation, and requires various utilities to be installed (editors, IDEs, compilers, debuggers, testing tools, and so on). They also need access to network resources, such as printers, code repositories, file servers, email, etc.

When such staff are employed, a workstation is built from scratch for them. The IT staff carry out a network (PXE) boot of the workstation and answer three questions. They then leave the workstation and the boot image installs Linux and the CMS client, and then runs the CMS client. Twenty minutes later, it is fully built with all utilities, X windows, network resources, etc. It only remains to add a user login to the LDAP server, and that workstation is ready for use.

It’s trivially easy to change the configuration of a new workstation: it’s just a matter of editing files on the CMS server. Gone are the days of a “gold” CD that would build a workstation, and which was complex and costly to maintain and update. The workstations are configured exactly as required, all are consistent, and it Just Works.

Gotchas

It’s not a perfect world, and whilst a CMS is, in my view, an essential component of a Linux infrastructure, there are pitfalls.

Consistency: as discussed, a CMS is an excellent tool to ensure consistency across multiple servers, but that means that mistakes are consistent, too. There must a be a robust process whereby changes are both tested and reviewed before being deployed.

Planning: making major changes to how a CMS is implemented within an organisation is not trivial, and as the CMS embeds itself deeper into the infrastructure, the change becomes harder still. A CMS is not alone in this respect, of course, but it can be deceptive: what starts as a mechanism to only ensure all necessary packages are installed on servers can quickly become something much more. There are three approaches that can help reduce the problem:

• buy in experience of implementing a CMS, preferably from an organisation that has a reputation at stake
• define clearly what you expect the CMS to achieve, and do not allow “feature creep” to expand that remit without proper consideration
• plan to throw one away anyway. The lessons learned will make the next implementation considerably better.

Security: if the configuration of all of your servers is managed from one CMS system, you need to be sure that the CMS server is secure and that changes to that system are carefully managed.

When should you start using a CMS?

The glib answer is “before you need one”. As soon as you have more than one server, you will benefit from a CMS in all the ways mentioned: consistency, an audit trail, reduced system management costs. However, it’s unlikely that an organisation with two servers would be prepared to implement a CMS (but be aware that the phrase, “If I knew then what I know now” may crop up in the future).

Realistically, the best answer is “sooner rather than later”. When IT staff are repeatedly logging in to systems to make the same changes, or when the irritation factor of different servers behaving in different ways starts to rear its head, that is the time.

Could This Article Be Improved?

Let us know in the comments below.