Germany's Top Level Domain Completely Died Yesterday

Added: 2026-05-11

[00:00:00]Germany was temporarily deleted from the internet.

[00:00:03]I don't mean that literally, of course, but.de, the TLD, top-level domain, that thing you see at the end of a URL like.com,.net,.gov, or in this case, a cc-TLD, a country code top-level domain, like.au for Australia,.uk for the United Kingdom,.jp for Japan, or in this case, the problem.de.

[00:00:26].de was almost completely inaccessible for a period of a few hours from around about 10:40pm to about 2am Germany time, so it didn't cause any major disruptions amongst most people.

[00:00:43]However, for the night owls out there, yeah, this was a pretty big deal.

[00:00:48]In the biggest duck-up since Germany has internet, the root DNS for.de is unavailable.

[00:00:54]Everything is down here.

[00:00:55]I'm sitting in a train and can't show my ticket, but the conductor also can't check it.

[00:01:01]Pretty much everything that used.de was completely broken, completely inaccessible.

[00:01:07]I feel kind of bad, you know, I feel very bad for any of the sysadmins who got a very angry call late at night because whatever corporate website you're managing was no longer working.

[00:01:20]But there's one very important question with two very different interpretations.

[00:01:25]How did this problem happen?

[00:01:29]As in, I thought TLD were just some letters at the end of a URL.

[00:01:35]How do they just stop working?

[00:01:37]And on the technical side, what actually caused the problem here, as put by the one and only Jeff Geerling, it was DNS.

[00:01:48]Now, that's a good surface-level answer, but what does that actually mean?

[00:01:54]What actually happened here?

[00:01:55]DNS stands for Domain Name System.

[00:01:58]When you want to connect to a website, typically you enter what is known as a URL.

[00:02:03]This is a string of characters that indicates the resource you want to access.

[00:02:08]For example, www.youtube.com, the site you're probably on right now.

[00:02:14]This is a URL.

[00:02:16]But on the back end, this is not the language the machines are speaking.

[00:02:20]They speak the language of IP addresses.

[00:02:23]So if we go and ping this, we can see the IP of 142.250.124.93.

[00:02:31]However, YouTube being a very large website, if we go and run that again, we get a different address.

[00:02:39]192.178.187.91.

[00:02:42]Let's try again.

[00:02:44]That is different from the original one.

[00:02:47]So a third IP address.

[00:02:49]Unless you've got it behind something like Cloudflare, your random little blog probably only has a single IP address.

[00:02:56]But when we are looking at a big website like YouTube, there is a reason they want to have multiple and you could just enter the IP address and connect directly to the site.

[00:03:06]But there's probably a good reason they have multiple of them.

[00:03:09]So there needs to be a system that can map the YouTube URL to that series of IP addresses.

[00:03:14]That, at least in part, there are other things that go into it as well, is DNS.

[00:03:19]One of the initial ideas here is this was a problem with the name server.

[00:03:24]This is a part of DNS, the domain name system, and Cloudflare has a really good post on this.

[00:03:30]A name server is a type of DNS server.

[00:03:33]It is the server that stores all DNS records for a domain, including A records, MX records, or CNAME records.

[00:03:40]What these are, are out of the scope for this video, but I will leave this page linked down below if you want to go check it out for yourself.

[00:03:46]Almost all domains rely on multiple name servers to increase reliability.

[00:03:51]If one name server goes down or is unavailable, DNS queries can go to another one.

[00:03:56]Typically, there is one name server and several secondary name servers which store exact copies of the DNS records in the primary server.

[00:04:03]Updating the primary server will trigger an update of the secondary name servers as well.

[00:04:08]Whilst it was an initial theory that there was a problem with the name server, this would also cause the sites to be inaccessible, that ultimately ended up not being true.

[00:04:17]Instead, it was a problem with something known as DNSSEC, Domain Name System Security Extensions.

[00:04:24]Whilst it might seem like they do, TLD don't just exist in the ether.

[00:04:30]You can't just say, oh, I'm gonna use.com, or.net, or.au, or.jp, or.li, or in this case,.de.

[00:04:39]All of these TLD have some sort of body that is managing that TLD.

[00:04:45]All of them do it in various different ways, but there is some sort of body that manages who uses the TLD, what the rules are, how it operates, all of this stuff.

[00:04:56]In the case of.de, this is done by DENIC.

[00:05:00]They operate out of Frankfurt, Germany.

[00:05:03]And what some technical users started noticing, likely some of the sysadmins who are awake very late at night trying to deal with this problem, is they were seeing malformed RRSIGs.

[00:05:13]These are cryptographic signatures assigned to these.de domains.

[00:05:17]So the point of DNSSEC, again, that is Domain Name System Security Extensions, is basically to ensure that the DNS records you're seeing are authentic.

[00:05:29]They've not been messed with in any way.

[00:05:31]So there have been cases of things like DNS hijacking, where you basically redirect the user to a page of the hacker's choosing.

[00:05:40]So you think you're going to www.youtube.com and there is a man in the middle and they redirect you over to you've been pwned, hacker's domain, whatever place they want to send you to.

[00:05:53]And maybe they make it look like YouTube.

[00:05:55]Maybe they make it look like, oh, you're supposed to run a command or whatever it is they want to do.

[00:06:00]They've now got you on their website.

[00:06:03]Also, there is DNS spoofing.

[00:06:05]Where it's basically the same idea.

[00:06:07]You think you're going to www.youtube.com and what they do is change out the IP address to where you're going.

[00:06:16]So you still think you're at YouTube.

[00:06:19]But they've changed where the destination is.

[00:06:23]And these are your textbook definition of a man in the middle attack.

[00:06:27]Now there are various other pieces of technology that stop these attacks from happening.

[00:06:30]DNSSEC is just one of them.

[00:06:32]And the way it does this is by providing cryptographic signatures those RRSIGs and public keys, DNS key to compare the signature against.

[00:06:40]So if the signature is invalid, then you're obviously seeing an invalid domain.

[00:06:46]Something has been done here.

[00:06:48]Someone has messed with this.

[00:06:49]For you as a user with a computer to actually make use of DNS, you need something known as a DNS resolver.

[00:06:56]Now most people are using the one provided by their ISP.

[00:06:59]Maybe you're using something like 1.1.1.1.

[00:07:03]This is provided by Cloudflare.

[00:07:05]Or maybe you're even going and hosting your own.

[00:07:08]But for DNSSEC to function, there has to be a source of truth.

[00:07:14]Each DNS zone is trusted because it's verified by its parent.

[00:07:20]But there has to be somewhere where that stops.

[00:07:23]There has to be, this is how we know this is valid.

[00:07:28]And in the case of.de, that is with DENIC.

[00:07:32]What appears to have happened here is through a bad update, through a bad config change, DENIC basically started generating malformed signatures.

[00:07:41]Those RRSIGs, those cryptographic signatures that are verifying that this DNS record is a legitimate DNS record.

[00:07:50]What's interesting here though is those public keys, those DNS keys, those appear to be correct.

[00:07:57]Nothing was wrong there.

[00:07:58]It was just the signatures themselves that were wrong.

[00:08:02]This resulted in any DNSSEC signed domain being basically inaccessible because if the signature is wrong, then you can't use the public key to validate the signature because it's always going to return this is a wrong signature.

[00:08:18]But it wasn't just the individual domains themselves that were failing.

[00:08:21]There are.de domains that are not DNSSEC signed.

[00:08:25]However, they were in a DNS zone that was DNSSEC signed.

[00:08:31]These zones have a DS record.

[00:08:33]And again, if that record cannot be validated, the whole zone is now invalid.

[00:08:41]This resulted in basically everything failing to resolve because pretty much because pretty much every sensible DNS resolver is using DNSSEC if DNSSEC is available.

[00:08:54]So pretty much all of them were just failing to resolve the signatures even though the sites themselves working perfectly fine, their hosting was fine, the servers were fine, all of that stuff was good.

[00:09:06]The DNSSEC signature was failing.

[00:09:08]So it just assumed, oh, something is wrong here.

[00:09:13]This is not valid.

[00:09:14]Someone is messing with this.

[00:09:17]You're not going there.

[00:09:18]But some.de domains were still accessible during the situation.

[00:09:22]That part is not yet fully understood, but is most likely due to caching.

[00:09:28]Whilst this was all going on, DENIC published a status page, DNSSEC disruption affecting.de domains.

[00:09:33]DENIC EG is currently experiencing a disruption in its DNS service for.de domains.

[00:09:40]As a result, all DNSSEC signed.de domains are currently affected in their reachability.

[00:09:46]Now, as we discussed, this was not entirely true.

[00:09:49]It was also affecting zones as well with domains which were not DNSSEC signed.

[00:09:56]If you'd like to dive more into that specific topic, there is this great thread over on Mastodon.

[00:10:00]I'll leave it linked down below.

[00:10:02]Be sure to go and check it out.

[00:10:03]Whilst the name web and the way we interpret the web might seem like this big decentralized thing, and in many ways, it really is, at the foundations of this technology are these very centralized pillars.

[00:10:17]And if these pillars go down, things go really badly.

[00:10:21]Now, in the case of.de, in the case of DENIC, this is a very, very rare occurrence.

[00:10:27]So rare, in fact, that people initially weren't actually blaming DENIC, they were blaming Cloudflare because, you know, Cloudflare has had, uh, quite a few outages over the past year or so.

[00:10:43]In this specific case, though, Cloudflare is entirely innocent, but they did go and publish something on their status page, resolution issues for.de domains.

[00:10:55]Basically, what they decided to do was temporarily disable DNSSEC validation for.de domains with their 1.1.1.1 resolver.

[00:11:05]So, if you tried to connect to any of those domains, basically, it wouldn't validate the signature and would just allow you to connect to it.

[00:11:13]Certainly not a perfect or a long-term solution, but with the goal of getting things functioning, getting websites back online, this was a good temporary thing to do.

[00:11:25]There's one thing which I wanted to save for the end, something which is probably not related, I assume, I hope, but this took place on the 5th.

[00:11:40]Seven days from now, there is a scheduled outage by DENIC.

[00:11:48]Do I think this is related?

[00:11:51]Are you excited?

[00:11:52]No.

[00:11:54]However, the timing is very amusing.

[00:11:58]With all that being said, my love goes out to the sysadmins out there who got called very late at night to fix a problem that was not your problem to fix because you had nothing to do with it, but your boss didn't know that, so you still had to come into work very late at night.

[00:12:14]So, get some claps, get some Ws in the chat for all of the sysadmins who had to stay up very late to fix a problem that they couldn't fix.

[00:12:26]Anyway, if you liked the video, go like the video, let me know your thoughts down below.

[00:12:30]Were you one of the sysadmins who got called very late at night to fix this problem?

[00:12:33]I would love to know.

[00:12:35]So, if you liked the video, go like the video, go subscribe as well, and if you really liked the video and you want to become one of these amazing people over here, check out my Patreon, SubscribeStar, Liberapay, linkd in the description down below.

[00:12:47]That's gonna be it for me, and don't let the DNS see you.