Channels ▼

Embedded Systems

Testing and Verifying Time Synchronization


NTP does not have native support for synchronizing with a PPS signal which is required to achieve accuracy of greater than roughly 1 millisecond. Instead, NTP uses external software to provide that capability. There are several programs available to do that but we limited our tests to software that does not require modification and recompile of the Linux kernel. Our assumption was that few end users will be willing to modify the RedHat kernel and lose support from RedHat when setting up a time server. Two programs are common for this -- GPSD and SHMPPS. We selected SHMPPS because GPSD had multiple problems during our tests.

The latest release of the GPSD software did not work correctly with some PPS sources and this was noted on the forum for GPSD users. A patch for this behavior did exist in the development version (beta) of GPSD version 2.38rc1 which is the version that we tried. Some tuning variables are required for each model of GPS and the tuning parameters for the Garmin 16HVS came from the same forum. The documentation for GPSD and NTP state that there is a tuning variable required to be set correctly to get an accurate fix on the PPS signal. The offset for the Garmin 16HVS that we are using is 42 milliseconds and is put into /etc/ntp.conf. As we tested we found a 42 millisecond offset from remote NTP servers being used to cross-check the time offset. This offset shows up in our tests no matter what we set this tuning parameter to. We tested with 42 milliseconds, 0 milliseconds and -42 milliseconds with no change in the behavior of the system. Without in depth testing like this one could have just followed the installation instructions and accepted the values produced by NTP and assumed all was functioning correctly when it wasn't. We did not find a way to overcome this problem of the 42 millisecond offset when running our NTP server tests. We also found that GPSD regulary lost its lock with the PPS signal and after some debugging we found that this is a known problem with a race condition that exists in GPSD with no avaiable fix at the time of writing this article and so we discarded GPSD and instead used SHMPPS for these tests.

SHMPPS uses a remote NTP server to compute an absolute time -- time to within a second to establish "absolute time". Once that time is reasonbly well established -- to within 250 milliseconds -- it begins to track the local PPS source to refine the estimate of time.

Each NTP test was started with the "drift" file removed except where noted. This file is used to store the local clock speed difference from the reference clock between runs. This caused each NTP start to begin without a calibration of the local oscillator.

Figure1 shows a NTP server synchronizing with a GPS displaying the offset from the GPS as reported by NTP itself (using ntpq -p), the NTP reported offset from a remote NTP server and the offset from the PPS signal measured by a dedicated Linux kernel module.

[Click image to view at full size]
Figure 1: NTP server offset..

The left side of the graph shows that NTP was detecting a large offset and applying a large correction of nearly 0.5 seconds occasionally. This shows up as sawtooths in the graph. Any programs running on the system at this time would have experienced an instantaneous jump of time by this amount which is undesirable for applications that require stable and predictable time. This happens while NTP attempts to synchronize with the absolute time source (NMEA string provided by a GPS or remote NTP server) so that an approximately accurate value of time is established before attempting to refine the synchronization by using the PPS via SHMPPS. After these corrections SHMPPS would begin providing PPS offset data to NTP (seen as the NTP reported offset from the PPS and remote server tracking one another) but before NTP had applied correction and begun tracking the PPS the offset would drift beyond the 250ms threshold and SHMPPS would stop providing data to NTP. These updates can occur at any time during the initial synchronization period for NTP until it begins applying gradual correction. Depending on the local oscillator frequency (clock on the chip or motherboard) being faster or slower than correct then this synchronization period can take quite some time.

Figure 2 shows that the NTP self-reported offset from the PPS signal eventually (after about 5 hours runtime) has reduced the offset to below 1 millisecond.

[Click image to view at full size]
Figure 2: NTP server offset; first 4 hours.

This shows that even with an extremely accurate external timesource (the GPS) NTP requires a great deal of time to converge to the external time source. After a reboot you may have to wait for hours for NTP to adjust the time. In many environments this would not be acceptable since accurate and synchronized time is required shortly after boot, especially when acting as a server and providing this time to a number of remote clients.

[Click image to view at full size]
Figure 3: NTP server offset; hour 23 to 24.

These figures also show that NTP required a great deal of time (nearly 45 minutes) before it begins adjusting the time so that system time offset from the PPS signal begins moving towards 0, overcoming the inherent local oscillator inaccuracy and causing the system time to correct. This would mean that after resetting the server 45 minutes would be required before the server would even begin to stop the local clock from accumulating errors that are provided to clients.

[Click image to view at full size]
Figure 4: NTP client offset - 24 hours .

NTP Client

The NTP client was start shortly after the NTP server was started (about 30 seconds). A Linux kernel module was used to monitor Linux system time offset from the PPS.

The start of the run -- the first 4 hours -- is shown in Figure 5.

[Click image to view at full size]
Figure 5: NTP client offset; first 4 hours .

There are a number of sawtooth curves in the graph. This is due to NTP adjusting the clock instantaneously by 1.5 seconds at times. This effect is visible to applications running on the system. There are more of these corrections in the client graph than the server graph since the client is attempting to synchronize with a server that is itself applying 0.5 second corrections while providing time to clients. This measurement is more interesting than the server measurement since it shows how well NTP is able to synchronize remote clients with an accurate time source -- a GPS and PPS in this case. We can see from this graph that a NTP client once started up requires a good deal of time to "settle down" and apply smooth correction but during that time radical changes in time and unreliable offsets from a time standard are to be expected.

Figure 6 shows the graph above at hour 23 through 24 of NTP running.

[Click image to view at full size]
Figure 6: NTP client offset; hour 24.

The time offset is slowly moving towards 0 when viewed over 24 hours but this more detailed view shows that time is moving towards and away from 0 in leaps and bounds. In this one hour view there are several direction changes in the curve showing that time is not being adjusted smoothly. It is, in fact, moving away from 0 offset and being corrected abruptly occasionally. Even though this adjustment does not show the same time jumps as in the initial synchronization these changes in the rate of the clock could mean that applications attempting to sleep or pause for a precise amount of time would see time progressing at very different rates depending on when they execute the wait.

Figure 7 shows the NTP client offset from hour 8 to hour 9. Time is trending towards 0 offset from the server and PPS but not smoothly or consistently. This shows that even correction is not predictable and time on the system may not be entirely reliable.

[Click image to view at full size]
Figure 7: NTP client offset; hour 8 to 9.

This shows that even with a quiet and short connection to a remote server NTP takes a great deal of time to converge to the correct time. After a reboot you might have to wait for 24 hours or longer for NTP to adjust the time. In many environments this would not be acceptable since accurate and synchronized time is required shortly after boot.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.