Linux servers have clocks inside them and, like almost all clocks, they are not 100% accurate. We’re going to focus on one clock, the “system clock”, and look at some of the issues around Linux system time and how you can prevent them.
What’s The Problem?
Most of us don’t worry if our watches are a minute or two fast or slow; some even set their watches fast deliberately. That isn’t usually a problem, but in the Linux server world having a clock with the wrong time can be a problem.
External Events
Let’s imagine you had server problems overnight. The logs on your server show that it was down from 21:25:03. The building management say that the power failed at 21:31. The question is: did your server actually have a problem before the power failed, or was the clock simply six minutes out? If the former, you may have a real problem to solve; if the latter, it’s just irritating (but there are ways of fixing that problem) . The issue is that you don’t know.
Other Systems
When multiple IT systems are working together to provide a service – for example, a database server and a web server – it can be very helpful to be able to tie events that happened on server A with those that happened on server B. If both system clocks are accurate, that’s not too hard to do, but if they are different it may be very hard or even impossible.
Authentication
Some servers refer to another server to authenticate users (that is, confirm that a username and password are valid). The remote system, the one doing the authentication, will validate the login for a defined period of time, typically until five minutes into the future. If your system clock is more than five minutes ahead of the authenticating server, no one can log in.
When troubleshooting the inability of a user to log in, few would think initially that the problem may be the time set incorrectly on one of the servers. This problem, however, isn’t uncommon when a Linux server is using a Microsoft Active Directory server for authentication: it’s critical that both servers’ clocks are synchronised.
What’s The Solution?
Time Servers
Around the world, there are systems known as Time Servers. These make the current time available to other systems, and use a highly-accurate reference clock to ensure that they have the correct time. It’s a little more complex than that, but essentially the time servers provide a “speaking clock” service for other servers.
So how can you use those time servers?
Brute Force
There is a Linux command – ntpdate – that will force the system clock on your server to synchronise with a time server. It works, but it isn’t ideal:
- it has to be run manually, which is almost never a good thing
- there may be unintended side effects. For example, if there is a job that is supposed to run at 10:00 each day, and you reset the system clock from 09:59:34 to 10:00:17, will that job still run?
Regular Resets
It has been known for people to set up a regular job to reset the time, perhaps every night. This isn’t ideal either: the concept of using the clock to manage a job that resets the clock is fraught with problems. If the clock reset runs at midnight, and as a result the clock is reset to 23:59:33, the job will run again 27 seconds later. If the clock is set forward, the second problem outlined above may occur.
The Network Time Protocol
All of these problems can be handled automatically by running a program that manages the local system time. Such programs are typically referred to as “NTP”, after the protocol that is used to determine the correct time. Such programs are clever and they can:
- find the time from a number of remote time servers (typically around 3)
- calculate how long it takes for the time information from the time servers to reach them
- calculate the average “current” time from the remote servers
- compare that time with the system clock
- make multiple, small changes to the system clock until the time is correct
- Again, it’s slightly more complex than that, but this technique avoids sudden jumps in the system time, and ensures that it is kept accurate.
Testing
It’s not easy to check the accuracy of your system clock from a non-technical perspective. You can look at the date/time of files created or mails sent, but that will only indicate accuracy to within a few seconds at best.
There is a Linux command (ntpq) that can list the names of the remote time servers and the difference between their version of “now” and your system clock, and this can be useful in diagnosing problems.
Best of all, however, is to have the accuracy of the system clock included in the regular monitoring of your servers. That way problems can be investigated and resolved before they impact your business.