In my experience, the vast majority of Linux servers are not kept up to date. There are always reasons, but they are seldom good ones.
Types of Updates
We know that software has bugs, and we know that most software will improve in both reliability and versatility over time. But, to get from yesterday’s version of the software to today’s, we’ll need to install a new version.
We can divide software changes into two groups. “Security updates” are released to fix, or at least mitigate against, identified security vulnerabilities. “Upgrades” are releases that introduce new functionality together with, typically, more bug fixes (and possibly new bugs).
What’s Included?
Linux distributions vary in how strict they are in ensuring that security updates consist of only security fixes. Some – for example, Debian – are very strict, and no new functionality (even to fix bugs that are not security related) creeps into their security updates. Others are not so meticulous.
In a development environment, there is often a pull by the developers to install new versions of software that include new functionality or functional bug fixes: it makes their job easier, or their software can do better, sexier, faster things.
By contrast, the system administrators want a stable system, one that is constant, unchanging.
Both should agree, though, that fixing known security vulnerabilities is a good thing to do.
Security Standards
The standard of security updates from the major distributions is very high. It would be wrong to say that there are never issues, but they are very rare (perhaps one update every 2-3 years has an issue), and the issues tend to have a minor impact.
Many of the security vulnerabilities that the updates fix are theoretical and have not been seen “in the wild”, but once or twice a year a serious vulnerability is identified and fixed.
Ignore Updates or Ignore Testing?
There are three approaches to security updates: you can ignore them altogether, rigorously test them before installing on production systems, or just roll them out.
Ignore
Ignoring them altogether is, I would suggest, not a sensible plan. Typical arguments for this policy include “the system is not exposed to the outside world” and “it’s too risky to update them”.
The system may well be intended to be available only internally, but it’s always possible that an error or another bug exposes it to the outside. Even if that doesn’t occur, you may have a nefarious or resentful employee, or even an office cleaner who is more interested in breaking into your systems than emptying the waste paper basket.
Some organisations are very wary of security updates, possibly having been bitten in the past where a “security” update did more than just fix a security problem and, in doing so, introduced other, unrelated, problems. If you consider it too risky to update a system, you are choosing to accept the risk of the known, documented security vulnerability over the truly minimal risk in updating the system.
Think carefully before deciding upon that as your strategy.
Test
Rigorous testing before roll out is appealing on the surface, and it’s certainly possible to introduce security updates in the same way as functional updates, via a testing phase before being rolled out to production, but be aware that that will incur a lot of overhead.
If you choose to adopt this strategy, put in place a defined acceptance testing procedure, preferably fully automated. For the software installed on a typical server, one security update per week would not be unusual, so time spent minimising how long it takes to test the updates is time well spent.
Just Do It
Simply rolling out updates without testing may sound cavalier, but in reality for many servers, it’s not inappropriate.
When a security vulnerability is identified, it will usually affect all Linux distributions. Although distribution A may view version 1.2 of a package as the current version, and distribution B may see version 1.3 as current, very often a security issue will affect all “current” versions. There is therefore often consensus amongst the various distributions as to what the best fix to the vulnerability may be, and that in turn makes for more robust bug fixes.
For many, choosing a distribution that is more regimented with respect to only including security fixes in the updates, and possibly installing the updates 24 hours after they are released to allow any issues to surface, is a reasonable strategy.
External Security Scanning
Even if you keep your systems fully up to date, some external security scanning services may (incorrectly) identify your systems as vulnerable.
Consider Apache, the web server software, as an example. At the time of writing, the version of Apache distributed by Debian Linux is based on version 2.4.25 – that is, major version 2, minor version 4, patch version 25. That’s the version number that would be reported, by default, to an external scanning tool. However, the full version number of the Debian package is 2.4.25-3, the “-3” indicating that Debian has made some changes of its own.
At some point, it’s possible that a security issue will be identified in version 2.4.25 of Apache, and in response the Apache Software Foundation may release version 2.4.26. The Debian security team may not want to release 2.4.26 to their “stable” release because it include some functionality enhancements, so they “backport” only the security fix to 2.4.25 and add a “-4” to their version.
An external tool which only considers reported version numbers will report that your version of Apache is vulnerable – but it isn’t. The better scanners do more than just look up the reported version number – after all, it’s trivial to suppress version information or even fake it.
CVE numbers
Vulnerabilities are often identified by a “CVE” number, which is a reference to the Common Vulnerabilities and Exposures system. This system is managed by the non-for-profit MITRE Corporation, funded by the US government, and lists publicly declared vulnerabilities and exposures in publicly released software, including beta and other test versions. It’s an easy way of unambiguously referring to a known vulnerability, and the major Linux distributions will maintain their own databases that show which version of a given software package (if any) resolves a given CVE number.
Let’s consider an example: a third-party scan of your web server reports that it is running Apache 2.4.25, which is vulnerable to CVE-2011-4415. If you want to know more about this vulnerability, you could look it up in the CVE database, and find it at https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-4415.
That gives a brief summary, but if you wanted more detail you could follow the link to the National Vulnerability Database, which gives some technical detail, and also assess the risk (“LOW”) and the impact (“Allows disruption of service”).
What you really want to know is: “How vulnerable is my system?”. You’re running Debian, so you can go to the Debian Security Bug Tracker and search for that specific CVE. That will take you to https://security-tracker.debian.org/tracker/CVE-2011-4415, which in turn tells you that the “stretch” release of Debian is not vulnerable, and that this issue was fixed in version 2.4.25-3 of the Debian apache2 package. If you’re running that version (or later), you’re not exposed to this vulnerability.
Monitoring
Your system monitoring regime should include the availability of any uninstalled security updates. The installation of most security updates is very straightforward, although updates to services (databases, web servers, etc) will usually involve a restart of that service to pick up the updated version. That typically results in the non-availability of that service for a couple of seconds, and for that reason installing security updates outside of key business hours may be desirable.
Kernel Updates
Kernel security updates are not so trivial. In order to make the new kernel live, a reboot will typically be necessary. That means a few minutes of downtime, and is a slightly higher risk than the more run of the mill updates. An evaluation of the risk/benefits of updating the kernel is sensible, and in some cases deferring some kernel updates may be appropriate.
Non-native Software
All the above applies to software packages that are part of the current version of the Linux distribution in use. Things change when that’s not the case, and there are three special cases:
- running the “Long Term Support” (LTS) version of a distribution
- running packaged software from a repository other than the current version of your chosen distribution
- running software built from source
Long Term Support (LTS) Releases
Some Linux distributions release new versions very frequently; for example, Unbuntu has a new release every six months. That may be fine for an individual user’s desktop, but upgrading servers or corporate desktops every six months would be a huge undertaking.
The idea behind “Long Term Support”, or LTS, releases is certain distribution releases are supported for a considerable time despite later, short-lifetime, releases.
Taking Ubuntu as an example, the standard release cycle is every six months, in April and October, with each version being supported for nine months following release. The even-year April versions, though, are LTS versions, and are supported for five years from release.
Not all distributions embrace the concept of Long Term Releases: Red Hat, for example, supports each major version of Red Hat Enterprise Linux for at least ten years, although the level of support declines towards the end of that period.
The LTS Gotcha
The issue to be aware of with some LTS versions is that security updates apply to only a subset of the software packages. This is a commonly misunderstood aspect of LTS releases. If you elect to use LTS releases, check with your distribution exactly what is included with LTS support. You may find that you are running a business-critical application that is not covered by security updates.
Non-Distribution Software
From time to time, it may be considered desirable to run software that does not form part of the Linux distribution in use. It may be that a later version of the software than the distribution supplies is required, or simply that the required software isn’t packaged for your distribution at all. There are a number of options.
Use Your Distribution
The first is to install the required package from a later version of your chosen distribution, although that won’t always be practical. Some distributions make this easier than others. Debian, for example, supports a “backports” repository, which contains software versions that will appear in the next stable release of Debian, but which have been packaged for the current stable release. They won’t be as well tested, but if you must use versions that are not currently packaged for your release, the backports repository is a good place to start.
Use The Vendor
The next option is to find whether the software vendor has packaged the version you want for your distribution. If they do, that’s an easy option, but a word of warning. The quality of the packaging varies enormously, and by “packaging” I mean the installation and system management scripts that are bundled with the software to facilitate installing and removing the software from your system. Ironically, in the case of commercial (paid for) software, the general standard of packaging is considerably worse than the packaging done by the distributions themselves.
Any Port in a Storm
A third option is to search the Internet to see whether someone has packaged the version you require for the distribution you are using, but we are getting into “not recommended” territory here. The integrity of the software you download from an unknown source is itself unknown: don’t do it.
From Source
Lastly, the software source may be downloaded and built. Where possible, build it as a package to install on your system using the package manager (ie, for a Red Hat system build an RPM file). Do not build the software on the target system – production systems, in general, should not have compilers installed on them – but rather on a development system running the same distribution as your production systems. Ideally, add the resulting package to an internal package repository, which makes it much easier to manage packages across various systems.
In all cases where you are installing software other than from the current version of your chosen distribution, consider how security updates will be handled. Certainly if you build the package from source there will be no automated update or even notification of vulnerabilities, although most software projects have an “announce” mailing list that will keep you informed (but you need to subscribe to it).
Other options may have some kind of security support; for example, the Debian backports repository implements security patches on a “best efforts” basis – but there are no guarantees. In short, if you can avoid installing software that does form part of the current release for your distribution, then do so.
In Closing
Keeping Linux systems up to date is a necessary element of IT hygiene. There can be challenges in doing so, but that is not a good reason to run software that has known security vulnerabilities.
Could This Article Be Improved?
Let us know in the comments below.