24/7 Proactive System Monitoring

System Monitoring

Effective system monitoring, with automated alarm notifications to support staff, can preempt many problems and thus improve system availability. Such monitoring will typically include:

• Free diskspace available
• CPU utilisation
• Memory utilisation
• RAID health
• Checking for disk errors
• Checking system logs for potential problems
• Ensuring essential services are running
• The status of the backups
• Whether security updates need to be installed
• Routine system security checks
• Checking the validity and lifetime of any SSL certificates be installed
• Routine system security checks
• Checking the validity and lifetime of any SSL certificates

Business Process Monitoring

It is relatively trivial to extend the monitoring to cover key business processes. For example, an online shop may know that, on average, it takes 10 orders an hour. One check might be to look at the time of the last update of the “sales” table in the database. If it was more than 10 minutes ago, an alarm could be raised.

This is by no means the only test that should be run on such a server, and it wouldn’t be very helpful in diagnosing the cause of the problem. It would, however, alert staff who could check that all is well. If an issue is found, it may be appropriate to add further checks to more quickly identify future similar issues.

The point is that IT is there to support the business. Monitoring its effectiveness in doing so is a worthwhile approach.

Configuration Management

The configuration of every server we support is stored on a central Configuration Management server.

Each server checks its configuration against the master server periodically. If the actual configuration of the client server differs from that defined on the master, the necessary changes are made on the client to bring it into line.

It is possible, for example, to ensure that certain software is installed (or even that it is not installed), or that certain user accounts are or are not present on all or a defined subset of servers.

This ensures consistency between the servers as required, and makes the building of another server “just like server X” very easy. It also makes it easy to manage configuration changes across multiple servers in a consistent and controlled manner.

System Tuning

Effective system monitoring can also be used to improve system performance. The following example considers a web server that uses a MySQL database; however, the principle applies equally to other areas of system performance. Here, the trend graphs are used to monitor the impact of the changes that were made.

After examining the system, the decision was made in week 9 to change a number of MySQL parameters relating to caching (in other words, keeping more of the database in memory). In the graph below, a dramatic reduction in disk reads and writes can be seen because less data needs to be moved to and from the disk:

As a consequence of more database operations taking place in memory rather than requiring disk access, the number of database operations per second has tripled:

Below, it can be seen that the network traffic on the web server has increased significantly after week 9. This is because the database is now working faster, and is able to handle more page requests. (The additional network traffic in week 3 was unrelated to this work, and not typical of this application’s normal performance.)

User Data Backups

User data backup is available as an additional service. We use online backups to two independent remote backup servers. These are hosted in geographically disperse data centres, and each backup is independent (we back the data up twice rather than copying data from one backup server to another). Each backup includes a customisable preparation phase where, for example, databases may be dumped or custom application functions may be called. Each backup is monitored, checking that:

• the last backup completed without errors
• the last backup is not too old
• the current backup has not been running too long