Atomic Clocks & Time Synchronisation 101

[00:00:00]Hello and welcome to Atomic Clocks and Time Synchronization 101. I'm Julian.

[00:00:05]While this is a long talk, giving everything its dues is impossible. Many of these topics, every sentence I talk about could be and often already has been someone's PhD thesis.

[00:00:18]There's a lot of complexity. This is high level. I'm almost certainly getting a bunch of stuff wrong.

[00:00:25]Sorry about that. Please use this talk as an inspiration to go do research, find what these interesting topics are and talk about them. By day, I'm a site reliability engineer at Google in Sydney. Been around there for 15 years. In fact, today as of this recording, I specialize in human factors and networking. Have done a bunch of stuff. And while Google does have probably the broadest fleet of commercial atomic clocks in the world, I'm not talking about Google things today. I have worked with the time lords back in the day, usually explaining to them why their idea for better time synchronization wouldn't actually work with the realities of intercontinental networking.

[00:01:12]I'm a music lover and studio engineer ser photographer and have a deep history with the Australian open source community which connects me to a bunch of things here as you'll see and yeah I've been around time for a while. Uh this meme came up just did the rounds recently and it just felt seen.

[00:01:34]So my background into time not goes back to about 2006.

[00:01:37]I got interested in making NTP work better than just pointing at random serve with open NTP. The NTP pool as it is today and some of the better public services simply didn't exist and I was on dialup ISDN or congested ADSL depending on the site and that really didn't help. So I picked up a few GPS sourced NTP servers made them work. Uh, several of these have been since permanently loaned folk and are contributing NDP pool join the time nuts list. I bought my first cium clock in 2018. That's a originally DATIM model 4000 that actually lives under my desk at work and is quite literally a footrest. Um, I was given a HP561 cesium for around Christmas time in 2020 by a friend and I have heaps of Rubidium clocks.

[00:02:36]It's at least six different model families from just the quick pile through and I wouldn't be surprised if it's more. I've designed my own uh integrated GPSDO OCXO.

[00:02:51]Um never quite got around to doing the integration test for it. There are some Linux Confu talks where there's video up for the single board computer and there's another one somewhere about the oscillator and testing and characterizing it. On the bottom left there is an FRAM FRK in a custom enclosure that is about a meter to my left as I record this. And on top of that is a Spectra 8140 frequency distribution system.

[00:03:23]On the right are my business cards, my personal business cards, which are actually uh a modernized version of the uh pod that is used on that Spectrum 8140 system. And it's in fact a pure single-sided layout with no jumpers.

[00:03:48]Asterisk, you need to connect the top and bottom ground on the connectors. So, I'm kind of proud of that one. Modern single-sided layouts in modern stuff takes effort, but this layout happened to work. On the left, we have the FTOM FRK that is my home standard, and sitting on it is a Ublocks that was done as a test of that module and custom oscillator. So, that's actually running at not 10 MHz.

[00:04:15]on the right uh is in fact the commercial uh test module for that module and my custom oscillator that I designed. So first let's get into a bit of history. Let's talk about how timing and clocks developed before we get into the atomic clocks. So we start out with a concept of time versus frequency. All clocks that are more than just sund dials are actually counting cycles of oscillators.

[00:04:45]Even sun dials are kind of counts of earth's rotation.

[00:04:50]And with the exception of me mechanical watches, we make the use of resonance to help keep us on frequency. And even watches, it's kind of a thing, too. Uh, but it's more complicated. If you want to know the history of watches, highly recommend the book Longitude. And should you ever be in London, visit the Royal Observatory to learn more about the prize and see one of the watches that would ultimately won it. The first real solid clocks uh are pendulum clocks. You also hear them as grandfather clocks for their larger ones at least. They're also known as gravity clocks because that's actually what they're doing is they're measuring against gravity which doesn't change in a location.

[00:05:41]Um the weight provides the power for the timekeeping mechanism and keeps the pendulum moving. Uh so in this clock the example is actually electrically powered so you don't see uh separate weights but classic movements you do. These were the best clocks we had until electronic clocks arrived in about the 1930s. You get errors from temperature, pressure, humidity, and they have to be calibrated to the local gravitational field.

[00:06:13]So, reference clocks are isolated. In this case, uh this is a doublewalled construction. I don't think it's fully isolated, but it is much more isolated than just a normal house clock. the local gravitational field does actually change. It is not a perfect constant.

[00:06:33]There are maps of this. Uh this image is from a survey of this year. Um you expect a few more significant digits. So I don't quite see the point, but So next off we have what might seem an odd degression into tuning forks. So tuning forks are a metal fork with a pair of tines of length controlled to resonate at a target frequency. Now the actual precision ones, you don't control the length as much. You then add weight and shift that weight to ensure accurate resonance.

[00:07:11]Uh as implied by the name, they were used until very recently to help orchestras tune. They will in fact still be in use in some places. I'm certain of that. But I don't have any examples.

[00:07:24]Temperature, humidity, air pressure.

[00:07:26]Again, all errors. In those use cases, that is acceptable.

[00:07:32]Once we're into the 20th century, you had powered versions, electrically powered versions. And until high quality quartz oscillators, they were actually our best available portable frequency standards. These are just some random tuning forks I picked up. a 440 in the front and a 300 and something behind.

[00:07:53]Larger size, lower frequency. This is an electronic tuning fork reference uh that was used as a power grid standard uh for the Victorian power grid. Uh the labels there reference that this was from the Hazlewood power plant in the LRO Valley in Victoria. Imported from London. Uh, it runs off only a 1.5 volt battery. I have had this running using just a pair of D cells to provide the current because it does actually have a pretty heavy inductive load, but I didn't try and optimize it beyond that.

[00:08:30]The sensor at the top right there is a modern standard where modern probably means 70s 80s possibly uh inductive sensor that replaces the original electrodes for use as output. That way you don't actually have to have physical contact to do it and you get partials.

[00:08:50]That was a modification clearly done while this was an interesting reference but probably no longer in service. The other interesting uh tuning fork to mention is the Bova Acutron series of watches, the original ones that actually use a tuning fork mechanism uh as essentially a stop gap before we got to quartz watches.

[00:09:15]They're really quite neat in person. The really neat thing about these is you hear them. They're extremely quiet. If you've got any background noise, you can't. But in a quiet environment, you can hold one up to your ear and you will hear it sing. It's quite amazing to hear in person as one of those things that people will tell you this is true and you go, "Yeah, sure, whatever." And then you actually hear and it's really quite neat. If you ever get the opportunity, give it a try. Another semi-digression, uh, grid clocks. Once large scale power grids became a thing, we started to get clocks powered by synchronous motors.

[00:09:51]And to keep them roughly correct, grids would actually install a pair of grid clocks. And unfortunately, I couldn't find a recent photo of this. I've seen them before, but I didn't make an archive where you would have an asynchronous clock display and a classic grandfather movement and they would speed up and slow down the grid slightly.

[00:10:18]Well, the speed up and slowdown of the grid that happens during normal operation, they would then often try and correct it at night so that the average time kept was correct.

[00:10:31]Um, modern grid clocks absolutely exist as modules in GPS reference clocks, although many grids have stopped trying to keep long-term time stability and instead simply focus on frequency stability. This is the best I could do.

[00:10:46]This is a display unit uh that has a tuning fork style resonant display of grid frequency. If you love the look of these, and they do look awesome, Adopseek did an amazing video on them, and it's worth digging up because they are really neat to see in person, particularly the units with dozens or of tines that have much lighter frequency variation. And you can see there that uh in this case in Australia on 50 Hz we see uh tines from 48 to 52 Hz and the 49 Hz tine is almost not going nor the 51 the 50 is bang on. And now to quartz crystals or crystal oscillators. These are where you have a thinly sliced piece of quartz used as a pazo electric element. When used as an oscillator, this means exact size and shape um of the cut along with the angle of the plane that the cut is made along the bulk element or matter with an electrode on either side. They then act as an extremely sensitive filter in an oscillator. So for lower frequencies, the fundamental is used above about 30 megahertz. Overtones are used much above that fifth overtone. You actually use electronic frequency mult uh multiplication to get the desired frequency. Capacitance is used to tune the oscillator slightly.

[00:12:22]And it's worth adding that the tuning fork style actually comes up again because standard timing crystals for lowerend devices. every commercial uh basic clock, every time crystal in a computer uh as a real-time clock or your microwave oven, your wristwatch are a tuning fork style crystal running at 32.768 kHz. And all the computer people immediately go, "Wait, that's 2 to the 15." Yes, it's precisely so you can divide down to a one cycle per second reference. And no, I don't know why that it's always references 32.768 kHz even though it's an exact number of on the right there is some early uh Soviet in this case crystals in glass envelopes so you can see them nicely. Uh here are a couple of other crystals. So on the far left and right are just crystals and packages, although the one on the left is still a quite high-end crystal unit and high quality. The two in the middle are crystal oscillators.

[00:13:41]So the one in the middle is a temperature controlled crystal oscillator that is disciplined and the one on top of it is a full-on ovenized crystal oscillator again running at the slightly odd frequency of 26 MHz. Uh here is a Trimble GPSDO unit. The interesting thing about this one is that package of the oscillator there. You can see the ovenized entry. This happens to be the exact same size as the modern uh small scale atomic clocks.

[00:14:18]So it is 2 in x 2 in by an inch half an inch depending on the packages. So now reducing sources of error. The biggest source of error we still have is the environment. So again, temperature, pressure, humidity, all modern uh crystals are hermetically sealed.

[00:14:40]Even just cheap consumer stuff, all hermetically sealed to eliminate most of it. It might not eliminate all pressure effects, but it's probably enough for general purpose environments.

[00:14:53]Temperature remains the big error. So for higherend users, the crystal is or the full oscillator is temperature compensated or run at a constant temperature by being ovenized and that being corrected long term. There is still some long-term drift from aging. So crystals are often selected after burn-in for critical applications and that can be thousands of hours. Drift also happens when the c when devices are soldered. So final products are burnt in as well. Once all these are handled, crystal oscillators are the best frequency sources we have for periods up to a few hours. Exactly what that time is depends on the exact circumstances. Now we hit the wonders of the atomic age. So the key concept of atomic clocks is what if instead of measuring based on construction of a device, we measure based on attributes of the underlying matter. Here we have a quote from the book cesium beam atomic time and frequency standards uh referencing how the 12th conference of weight measures in 1964 designated cesium. The committee then defined a concept of the hyperfine transition which I won't even try and explain uh when unperturbed by external fields which includes magnetic fields which includes the earth's magnetic field. So this is actually relevant and at 0 Kelvin which is also a impossible but b relevant for a standard. So how do atomic clocks work at a basic level? So you start by taking a good tunable crystal oscillator. Again over the short term they are the best thing we have.

[00:16:37]You use it to discipline a PLLbased synthesizer other ways of doing it but in practice around the hyperfine transition of a material. So hydrogen that's 1.4 4 gig, rubidium 6.8 gig, cesium 9.1 gig and other elements are used in some research clocks.

[00:16:57]We excite that material in a way that lets us measure its resonance.

[00:17:02]Now we measure the frequency that happens. But because we have defined what frequency that happens, we are measuring the input oscillators error in practice.

[00:17:16]We use that to tune our input oscillator so it gets closer to the expected value.

[00:17:21]We repeat the output of our crystal oscillator then becomes our standard output. So this can happen after multiplication division. It doesn't have to be 10 MHz. It usually is. That's all great. But it does sound a lot like the classic write a distributed map produce in Erlang, which it kind of is because at a high level it's actually pretty simple.

[00:17:46]But in detail it is quite complex. So again for the simpler clocks the rubidium standards the chip scale atomic clock cesium standards you take a bulb of elemental rubidium or cesium metal as a gas. You heat it until it glows.

[00:18:01]That's about 160 C. You pump in RF at the right frequency. So that hyperfine transition frequency as mentioned earlier. the light output changes or RF power changes. You slew this and this is one way of doing it. You can slo the synth from slightly below to slightly above to get the right frequency.

[00:18:25]You measure where we find the peak or the dip depends on the design.

[00:18:30]The peak is that peak or dip is at the known frequency. Now you discipline the oscillator degrees and your control loop repeats. Uh you can also do this as a pure analog pigfinder.

[00:18:43]Uh the nice thing about the modern system is you can run it at low power modes where you only do this every uh every hour even for a cack in low power mode and you can still actually get extremely solid uh time.

[00:19:05]So the early commercial rubidium standards like the HP5065 in the '60s were a three rack unit deep heavy thing.

[00:19:14]In the 80s you started getting roughly half liter volume modules. Uh the effort fakk for example that I showed photos of earlier.

[00:19:24]Then you got down to about 200 ml volume. The Ephrto L Pro was deployed by the 100 in cellular sites for CDMMA which required extreme timing accuracy.

[00:19:38]Then you got uh the modern volumes the Symmetric SA35 and similar.

[00:19:48]These are small board components as shown earlier and now with cesium in the 2010s with the chip scale atomic clock. So again they're about 2 in by 2 in 1/2 in for the modern ones. These are actually smaller and lower power than high-end OCXOs which is amazing. They're also price-wise comparable, so you really can be choosing an atomic clock over a crystal or the reverse. Depends on your exact use cases. Uh, story time. So, here's actually the shipping case for one of my units, which ages it both by having a TX label and then on the unit itself, made in West Germany, which uh definitely demonstrates that it uh is an older unit. This unit was made by the Bull Corporation who made the underlying oscillator in West Germany. This is what the inside of one of those standards looks like. This was in fact the unit from that previous photo.

[00:20:51]Uh the actual cesium uh sorry rubidium bulb in this unit was bad, but it had other problems, so it needed some repairs. So here's some uh units from the ' 90s and 2000s.

[00:21:07]You have a newer Ephrom unit on the bottom left, a Symmetric X99, which is actually a Symmetric X72 with different firmware for certain embedded uses, and a Stanford Research Systems PRS10. I believe the PS PRS10 is actually a derivative, a descendant of that Ephroto unit, and the Symmetric is probably a descendant of something else.

[00:21:32]Now again that bull logo the Americans might recognize because yes that is the same ball of mason jars.

[00:21:40]It was the 1960s everyone had an aerospace division. Uh some people might know Bendix known for their brakes to this day had built uh computers both air data computers on a bunch of aircraft and actual computer computers which ended up being a division of control data. Gore, you know, from Goreex, uh, one of the makers of the best microwave cables you can buy. I have a set of cables from Gore for my VNA that I'm sure originally cost BEA systems, tens of thousands of pounds, and I got for an absolute fraction of that. Uh, Monsanto Agro Company actually was one of the early pioneers in LEDs. So, there are still a bunch of remaining sources of error. You have everything [clears throat] from crystal oscillators partly because we need one at the core, partly because pretty much everything shares those same error margins, orientation, gravity, earth magnetic field. Uh, all of these are issues with crystal oscillators, but they're harder to measure. The cesium beam clocks actually do have manual compensation for this. We'll get to season beam clocks in a minute. And there's a bunch of more subtle stuff. This is a deep area of active research and as always there's a relevant XKCD for everything. Floating point errors digital systems continue to be things with their errors. Now cesium clocks when you think of an atomic clock you almost certainly think of a classic seesium beam standard.

[00:23:13]These are what people think of when they think of commercial atomic clocks not the modern super miniaturized stuff.

[00:23:21]The way this works is you get a beam of seesium atoms excited ejected from an oven filtered to those in the right ground state then excited by a cavity at the right frequency filtered again and measured. So this is similar to the simpler standards. The power is used to provide servo feedback and discipline a quartz oscillator. This is the HP571.

[00:23:45]It was originally the HP571. It was then the Agyant 5071, Symmetric 5071, Micro Semi55071, and now the Microchip 5071.

[00:23:56]As HP spun off their measurement division to Agyant, Agyant sold their time division to Symmetric, Symmetric got bought by Microsemi, who got bought by Microchip.

[00:24:06]This is the best commercial clock in the world. And the reason it has been sold under those names is because it was a really good design in the late '9s that hasn't consistently been bettered. I'm sure there are commercial clocks now that are better, but it really is only the last couple of years. This on the other side is the Atomicron, the first commercial atomic clock. Uh this specific one is owned by Tom Vanbark of leapsecond.com.

[00:24:32]Uh photos from when I got to visit his lab in 2019.

[00:24:37]Unfortunately, these are the best photos I have because I didn't find any better photos that Tom has of it. And all the other photos I get are classic photos from the era that have been badly newspaper scanned and aren't high quality.

[00:24:54]This unit is about a full rack in size.

[00:24:57]So, it's about six, seven foot tall. Uh so taller than nearly every all humans and couple of foot square.

[00:25:09]It dates from the late 50s to 1960 and it was estimated about 50 were sold.

[00:25:14]This on its side here is a HP 5061 uh a I think. Yeah. Uh this is the unit I was given for Christmas a couple of years ago. Thank you to Hugh for providing that when he had to move and shrink. It has controls here. That amazing little double dial in the center there is the C field which is the magnetic field compensation. Inside the unit we have uh controls for cable delay. So you can actually make the time appear at the right second. At the top right you see the crystal oscillator that is at the core of the unit. At the bottom right is the beam tube. This is my uh symmetricon branded CS4000. As I said, that's a datam design which is a modern cesium beam standard and it's just all digitally controlled through computers because there's no reason to do otherwise. So the next tier of clock up is the hydrogen maser. Instead of cesium, hydrogen mazes use, as you would expect, hydrogen as their working material. And the word maser is in fact where we get the word laser. The word maser was chosen first and then laser with light became popular and has very much overtaken mazes. So instead of cesium, hydrogen mazes use hydrogen as their working material. Fairly obvious from the name. And the commercial mazes are actually better than cesium clocks up to a few days. So they can actually end up in use in critical science situations as reference clocks or part of a reference fleet in large scale national lab timing and for some critical science applications where you care about frequency lock in over a couple of days. Commercial units are about sort of mini fridge size. Um the perspective on that one makes it look a lot bigger but it's it's about the size of a mini fridge. more advanced clocks.

[00:27:14]You get the lab grade units, the cesium fountain clocks, quantum logic clocks, optical lattice clocks, and research continues into actually redefining the second on optical standards within the next decade.

[00:27:29]These involve a lot of cryogenic temperatures, including laser cooling, which is a really neat thing.

[00:27:35]uh you then get optical latis atom trapping which is the next level of that and materials like aturbium, mercury, alum, aluminium and strontium.

[00:27:48]I spend too much time around Americans.

[00:27:50]All right, time for some random aides.

[00:27:52]So the best voltage references we actually have now rely on a thing called the Josephson effect in superconductors.

[00:28:00]So we actually do frequency to voltage conversion and therefore extreme quality of clocks is critical for those actions as well.

[00:28:11]This is a NIST experimental Joseph standard. There are commercial voltage standards on this basis these days as well. How do we compare oscillators to measure their standards? We do a thing called Allen deviation.

[00:28:25]So this is a plot of a bunch of oscillators referenced against other oscillators.

[00:28:31]And one of the key things is at the quality standards of high quality atomic clocks, you need to be measuring for hours or days to be able to tell which is better because in the short term they're both pretty good and you actually end up in the error margins where you need about a day of data or sometimes weeks of data to really measure. So that's the source clocks.

[00:28:55]Now let's talk about how we distribute that time information to all our modern devices or all our ancient devices. Let's talk before the computer era. We started out with master clock systems. That's where you take a very good clock, add a trigger output to it. That might be once per hour, once per day, once per minute. And when it fires, you send that clock to slave clocks. The slave clocks don't need to be very good. They are usually actually configured to run slightly fast or slow depending on the exact design and synchronization.

[00:29:31]Or for minute clocks, they might simply be pure displays only moving when the sync signal comes in.

[00:29:37]These are extremely common in schools, universities, larger business or government sites. And there were multi-level systems so that your sight's master clock could be fed fed by a master clock in the city. And although these were usually electrical with telegraph style systems, Paris actually had a metrowide airpowered system in the 1880s.

[00:30:01]These were significant business lines for Western Union and IBM amongst others.

[00:30:07]and modern versions very much still exist, but usually talk NTP, the standard for network time sync, which we'll get to later.

[00:30:15]On the left, we have a photo from Paris of one of their street clocks that was synchronized. And on the right, we have one of the Western Union clocks.

[00:30:28]The interesting thing on the Western Union is that it is stopped a minute before the hour. That is because they would run they are a system that would run slightly fast. They would intentionally stop a minute before the hour and the sync signal would trip them into the hour. So without a time sync when they were discon when they were disconnected from their time signal that is why you will find them almost always stopped a minute before the hour. So next you had uh widescale radio systems that would provide timing or timing as a side effect of location. So in the US you have WWV plus the Boulder and Hawaii versions. In Germany you have DFC77 that covers essentially the continental Europe. What you get in those regions are cheap atomic clocks that are basic clocks that are synchronized and set by those systems.

[00:31:28]You even get watches for some of those.

[00:31:31]The next tier up was a thing called Lauran C, which was a US military global navigation system and also the VLF uh system for submarine navigation.

[00:31:48]There was also a very common thing of using CDMMA. So, as previously mentioned, CDMMA cell sites had atomic clocks in them because they required such solid timing. You could then derive back timing from these. And not only did people do this, there were commercial clock receivers that would get their signal based off the cellular. The unit at the bottom right is a WWV uh based receiver unit. These units are no longer usable. Next, you had uh via telephone service. So in addition to speaking clocks which people of a certain age will remember uh those are vaguely hilarious because they were often synchronized to the grid who would occasionally call the speaking clock.

[00:32:32]This is how you ended up with until the large scale national continental international time standards.

[00:32:42]time really wasn't that well synchronized and you could end up in these local uh control loops that weren't aware they were synchronizing off themselves.

[00:32:56]So in addition to the speaking clocks there were a modem based system called acts that was available in many locations. Now, inconsistent transport delays and no ability to correct for them made this worse than radio in general. And with the death of PSTN, this is an obsolete technology. Not really strictly time, but there are a bunch of protocols used to transmit time data in media production. In audio and video, we have linear time code and simply time code in audio and the video interle time code in video production.

[00:33:33]Later we had the MIDI version with MIDI time code and video sync actually started as just a black burst video signal that you would gen lock off and now is the video interle time code.

[00:33:47]These will handle time at less than a day of interval. And in fact for a bunch of reasons you always start at at a time above zero. It's a classic one uh classic question to ask people who've done real media production is what is what hour does your day start? The standard answers are either 1:00 a.m. or 3:00 a.m.

[00:34:11]but it can be anything.

[00:34:15]And just for completeness and because there's nowhere better to put it, there is a still occasionally used standard from the interrange instrumentation group. Uh the range in this case being the White Sands missile range for countdown count up timers. IRG uh B122 is the most common version of it. And this is what drives every countdown timer you've ever seen in a control room. All the NASA timers, that's all IRG, at least classically. They may well be network controlled these days. So what does time distribution look like today? Big winner here is GNSS, global navigation satellite systems, of which the global positioning system was not the first, but was the well-known system to this day. And now we have competitive uh satellite constellations like Gloness, Galileo, and BU from Russia, European Union, and China.

[00:35:14]The preg system it's worth knowing about is a thing called transit which you can find information about online.

[00:35:21]And these effectively uh while these are navigation systems they work by determining delay from several satellites and you end up needing very good timing as a side effect of this.

[00:35:36]There were some time only systems goes was one example but once GPS was fully operational in the early 90s all the systems have been GNSS.

[00:35:50]The modern receivers from companies like Ublocks combine the multiple constellations to estimate if a constellation is lying and by using multiple frequencies they can uh remove some of the error cases. This even includes for GPS, the military signal that is an encrypted signal can still be used as a reference because you know a bunch of its attributes and can measure them even if you can't read the data in that signal. So the short version for how a GNSS works is satellites announce the current time and their orbital parameters. You take the time received and when where the satellite should have been at that point.

[00:36:38]At that point in time, you then repeat.

[00:36:42]You need at least four satellites for an initial position, but more is better. You can now use that time from four satellites to estimate what your delay from those satellites must be to get a consistent behavior.

[00:36:57]There's some radio errors, atmospheric conditions and such that will cause slight errors. This is why more satellites is better. Plus, some of the satellites is not perfect. My friend Cestera would at this point normally interject with exactly which GPS satellite he had a particular hating for when he worked for a GPS company because it was notoriously bad from its inception from its uh inception up until when it was de-orbited. He was very happy the day that satellite got de-orbited. So in addition to pure delay, you actually also need to include relativity corrections based on orbital parameters or that things don't make sense. This is one of the simplest and clearest proofs of relativity that we have is GPS needs to include those corrections for the data to make sense.

[00:37:48]And as part of this, GPS receivers are the easiest and cheapest ways to get high-quality time and frequency references for a low cost. For 20 bucks, you can get a really good time signal.

[00:38:04]For $500 US, you can get an incredible time signal that is only beaten by atomic clocks. And even those atomic clocks, you need to do good work to do them.

[00:38:16]So for 50 bucks, you're getting something that is only really beaten by $5,000 if you're willing to accept the dependency on satellite networks. So actually before we get to the satellite constellations, we had synchronous wide area networking. This derives from uh early digital telefan which needed an agreed shared clock for reasons. It basically comes down to It was infeasible to do otherwise in the 60s. And because of compatibility, this meant until the 2010s, we were still doing this for essentially all longhaul data transport.

[00:39:00]This started with what we now call a plesiosynchronous digital hierarchy, the Tarrier. Might have heard of a T1 or a T3 at 1.5meg, 45meg. In Europe, we have the Earrier, 2meg or 34meg.

[00:39:16]And then we got into the synchronous digital hierarchy uh known as set in the US SDH everywhere else. The difference between the two is literally one bit in the header. They are otherwise actually identical.

[00:39:29]This started at the OC3 level or STM1 which was 155 meg and ran up to OC768 which was about 40 GB.

[00:39:42]really it stopped being used at the 10 gig level. As we got to OC192, that was deployed. It was definitely in wildcale use and there were some transport systems built. Uh Google ran optical transport from I think a company that I think it was the older Nokia transport division if I recall that was actually uh OC768. So 40 Gbit wavelengths that were then marxed to give you four uh 10 gig lines. But at that point everything switched over to Ethernet because there weren't any real benefits and with essentially the de essentially with the death of classic tele voice telefan the need for the time synchronization aspect of it had gone away and everything is easy without it.

[00:40:40]If you talk to anyone who has run intercontinental uh set links, they will tell you about the hell that could be clocking. Google at one point had a line between Atlanta and Taipei, I believe, where it was a set of wavelengths. Uh might have been four of them. It might have been 16. And I I don't actually recall, but one of those wavelengths we could never get stable. It would just never clock correctly. And in the end, we abandoned it in place because we just couldn't use it.

[00:41:18]Eventually, we transition to actual Ethernet uh using a thing called well first using a thing called WANFI which is a whole separate discussion.

[00:41:28]Uh then transition to normal Ethernet at 10 gig and 100 gig and onwards.

[00:41:35]But with the increasing density in cell sites, a solution was needed for base station reference clocks and a synchronous version of Ethernet was developed called Sinki, which has now taken over from the classic systems and is widely deployed in cell networks.

[00:41:53]The reason for this is, as weird as it sounds, not all cell sites have access to GPS because some cell sites are indoors in buildings. So, they can't rely on GPS.

[00:42:05]They also do need very good timing and they would actually need quite a good ovened oscillator for a bunch of reasons. This lets them use cheaper oscillators that aren't as power hungry.

[00:42:22]Next, NTP. The network time protocol.

[00:42:24]NTP is the main protocol used for time synchronization over the internet. Uh, it is not related to the daytime protocol or time protocol which came before them. NTPv1 came out in 1985 and NTPv3 came out only in '92 which seems a long time ago but it was one of those many protocols where we learned enough to build in sensible extension mechanisms. So the NTP of today is not the classic NTPv3 from the early '90s.

[00:42:59]We have standards for authentication, encryption and a bunch of other things.

[00:43:02]These days there is the public pool NTP or give general users a way to synchronize against good clocks without annoying their owners.

[00:43:12]If you are looking to deploy a product that needs time sync, you should talk to either one of the fleets that is otherwise existing, Cloudflare, Google, other companies that provide a time service or you should engage the NTP pool as a vendor. That way you can guarantee that you won't hurt people.

[00:43:39]And the sync logic for these is all actually very simple. You send a packet tagged with your current time. The far end then adds in the time when they handled the packet.

[00:43:50]And if you assume that packet delay is symmetric by adding in when you received the packet back, you can work out the time difference. Uh packet delays are never symmetric. And of course, load has a bunch of variations. There are many reasons why there's classes of errors.

[00:44:06]This is why the control loops for these it can take hours or days to really lock things in. Next is the precision time protocol.

[00:44:16]This was a new protocol created with the goal of being better than NTP with sub microscond on lands. The initial version one was released in 2002 and is now almost entirely obsolete only supporting multiccast lands. There are still some uses in production deployments today. I happened to be talking to a tech from the Sydney Opera House just a few months ago and they actually still have it deployed in some cases for uh presumably Dante but I don't actually know. Dante is a audio transport protocol. Version two in 2008 is the generally deployed version and can absolutely deliver on better than NTP when configured absolutely right.

[00:44:58]Uh there was an implementation by CERN that's called White Rabbit that delivered sub nanocond timing and the software elements of that are actually now standardized in PTP v2.1.

[00:45:10]The actual timing accuracy with PTP starts to run into issues where you've got things like the length of a piece of fiber is not a constant because temperature changes its length. You get inconsistent timing by direction within a single link and the speed of light is actually wavelength dependent which matters. There is a wonderful triple C talk from 2025 on PTP.

[00:45:41]Uh the title is uh it's from 39C3.

[00:45:46]Excuse me, what precise time is it?

[00:45:49]It doesn't get into some of the classes of errors that do exist. So, it's a little bit optimistic in reality in my mind, but if you want to know what you can actually achieve, it's a pretty good overview. And now for some bonus content that doesn't really fit in this talk, but also kind of needs to be in this talk. Leap seconds. Now, leap seconds were a thing. They kind of don't really exist anymore. Uh Google somewhat pioneered the use of a smear to ignore the leapse second yet still synchronize with the rest of the world.

[00:46:25]If I didn't include the slide here, people would object, but I don't really want to include it. Uh strata.

[00:46:33]So you might have heard of a stratum one or a stratum zero.

[00:46:37]This is generally a reference to clock quality but it can mean different things in different scenarios. So in the telco world in the classic synchronous networking world we have a concept of stratum that is derived based on clock quality. Primary reference clocks secondary clocks uh set equipment clock is that last one there. In the NTP world, we just define stratum zero as the master clock, which does not itself speak NTP. So, stratum one means something that is directly connected to a master clock.

[00:47:16]And then stratum 2 is connected to a stratum one, etc., etc. A stratum 16 is in theory slave to a stratum 15. In practice, if something is saying it's a stratum 16, it's saying it's unsynchronized.

[00:47:33]Uh in practice today, you wouldn't go beyond about a stratum 5 for an end device in a pretty complex setup. Uh time standards, there's actually a bunch of time standards. So GMT Greenwich Meantime is the original name for UTC and is still the name for the UK's time zone. UTC is the civil standard. There are a whole bunch of things like UT0 astronomical time, UT1 mean solar time. Uh with times smoothed out with season smoothed out.

[00:48:09]There is also TAI which is without leap seconds. So TAI is the same as UTC minus about 30ish seconds.

[00:48:21]In practice that number of seconds is now constant. GPS actually transmits TI plus leapsec counters. Time zones. So the time zones have a canonical database called the Olson database as we know it.

[00:48:40]Weirdly this is maintained by ICAN of all people.

[00:48:45]Uh taken over by Ayanna. It's used by just about everyone in the modern world except Microsoft. Uh that might have changed. I don't know if Microsoft still maintain their own.

[00:48:57]And I say as usual with GE geographic things, you might not be aware just how political this gets.

[00:49:06]Uh updates here are surprisingly frequent and can come with little notice. They can even be retroactive.

[00:49:14]This is a political thing. Time zones are set by governments. Governments do things that make no sense.

[00:49:21]There are also some silly time zones. So being from Australia, it's worth calling out Adelaide is one of the notable halfhour time zones. India being the much larger one.

[00:49:33]We also have a semi-standard zone.

[00:49:40]It's dubious how much this is observed in Uclar which is on the 45 minute.

[00:49:47]Other than that pretty much everywhere sticks to the hour or at most the half hour.

[00:49:54]We did used to have time zones. Uh Singapore was pre-1941 on a 20 minute, Dublin on a 25 and then other parts of the world would be on even weirder minute boundaries or even second boundaries.

[00:50:13]In practice, as I say, everyone these days basically runs on the hour or the half hour. Daylight savings time, where it's observed, is also weird. It's not always just an hour forward. Lord How Island uses a 30-minute offset. There have been 20 minute and two-hour offsets in the past and many systems actually need to be configured with negative daylight savings time because summer in the southern hemisphere goes over the new year and those systems can't handle it properly. So, you actually configure them to be an hour offset and run negative daylight savings time. Uh, I had to do this for a lot of Voiceover IP devices back in the day. What do people need timing wise?

[00:50:54]Users complain if time's more than like Timoth Keraros uh network protocol for authentication used by Windows but also many other things actually has problems with more than about 5 minute offset.

[00:51:08]Now users really complain if anything's more than a minute off.

[00:51:15]Uh why did this email take so long? Why am I getting the phone call from the future? Sorry. Why am I getting this email from the future? The clock on my PC is wrong.

[00:51:25]Even if you synchronize, whatever they're compared to might not uh log analysis. If you're working on a distributed system trying to identify an event, if your clocks don't agree closely enough, this can be a real pain to find out about 30 milliseconds.

[00:51:48]uh and there are plenty of scientific widecale synchronization needs in scientific projects that need 100 milliseconds across hundreds of kilome sorry 100 nanoseconds across hundreds of kilometers. There was a recent incident in I believe Vienna where a experimental measurement was seeing results that indicated the speed of light had to be wrong. They were getting faster than light transmission and it turned out they were getting a 60 ncondish error because of an incorrectly mated fiber connector. Now this being a good science project what they did was call out we are seeing what should be impossible. Please help us eliminate sources of error and it took a while but they did find the error.

[00:52:39]Financial markets want submicroscond uh video frame sync for some live video walls. They now need 4 millisecond time lock. If you're wanting to do audio sync to the sample accurate level, that can be 10 microsconds.

[00:52:57]And scientific needs are getting much much worse than that. There's a really great paper by the square kilometer array people on their time synchronization needs. [clears throat] where they had needs that really wanted a hydrogen maser in each antenna unit which for a start you can't afford to do that. The second problem is you're in the West Australian desert for the part of the square kilometer array in Australia and you've got thermal fiber problems.

[00:53:29]You've got a lot of problems here where that doesn't work. So they uh wrote up what they are doing and it's a wonderful paper and you should read it. And that's it. Thank you very much. I hope you learned some interesting things.