Two Raspberry Pis, one cable: Can they just talk?

[00:00:00]I am building a little internet out of Raspberry Pi's on this old desk of mine so that I can start to understand all of the amazing invisible layers that go on behind this technology I use every day.

[00:00:12]I'm going to consider this project done when I have three independent networks, one of which is operating as a transit between the other two so that any one device can communicate with any other device over BGP without knowing the whole map. That sounds like a lot and honestly it is, which is why I'm starting from the very beginning. I have two Raspberry Pi's connected over one ethernet cable and surely they can just start to talk, right? Here's what it looks like now that I've got it all figured out, but I want to wind that back.

[00:00:46]You would expect that if I plugged these two Pi's in together and pinged Pi 2, which is on the right, from Pi 1, that it would just work. The reality is that they don't and the reason why was not what I expected. I'm actually trying to answer three questions. First, is there a wire at all? Second, are frames flowing between the two devices along that wire? And third, even if the devices are connected, can they reach each other? Let's pop open some SSH sessions and take a look at the reality going on in these devices. So, in the beginning they're not connected by an ethernet cable and I can verify that with the IP and eth tool Linux utilities. There's a couple of really good dead giveaways like no carrier down, uh there's no speed and duplex and link detected is no. Next, I just have to ask what changes in these tools when I plug the ethernet cable into both devices. Aside from the lights on the ethernet port very delightfully signaling that something is going on, these tools also reveal that no carrier has changed to broadcast, down is now up, the speed and duplex are now present and link detected has flipped to yes.

[00:01:59]That leads us to answering the first question, which is is there a wire between the two Raspberry Pis? And the answer is yes. I plugged in both ends and the NICs came to life. This is a really good moment to introduce packet capture. I'm using a tshark to show the frames that are passing between the devices when I plug the ethernet cable into both of them. There's actually quite a bit happening off the bat. If we look at just the first of the two Pis, we can see frames related to things like ICMPv6 and we can see frames related to mDNS. And in those mDNS frames, we can see mentions of both Raspberry Pis, which again signals that they are communicating. With all of these frames flowing between the two devices, I'd expect to be able to ping one from the other using an IP address that I'm familiar and comfortable with like 10.10.0.2.

[00:02:53]But when I try that right now, the packets get fully lost along the way.

[00:02:58]And if we look where this IP address is supposed to get routed to, I see that it's supposed to go out over a wireless network, which which makes sense but not in the way that I want. These devices are actually connected to my home Wi-Fi network I can SSH into them and manage them. It also means that they are trying desperately to reach each other over the Wi-Fi. But I thought of this ahead of time and I put a firewall in place to block that. If they're going to talk, they're going to do it over the ethernet cable. The second question now, are frames flowing between the two devices?

[00:03:30]And the answer is yes, but over IPv6, which wasn't what I was looking for.

[00:03:36]Because honestly, I don't think about home networking in IPv6. I figured that two computers and a cable would just talk, but they don't. A Pi doesn't come with an address baked in. That's just not its a It waits to be handed one or asks for one itself.

[00:03:55]The moment that I linked the pies together, both of them actually did ask.

[00:04:00]Kudos to you if you noticed this, but I skipped over some pretty important frames when I showed off what happened when I plugged the cables in. And that's DHCP. That's the network protocol that hands out identities to devices in the form of IP addresses. Maybe you've seen that if you ever had the joy of logging into your router's admin page.

[00:04:21]What I didn't realize at the time was that both pies were looking for a DHCP server, but there isn't one. There's nothing to hand out new identities. I thought they would have sorted it out anyways, but in IPv4, they don't. The wire is live, but that's not the same as being able to reach each other.

[00:04:42]What I finally realized is that I need to step in with network manager, which allows me to create an identity myself.

[00:04:49]In these two commands, I'm assigning those IP addresses that I've been after directly to the ethernet ports.

[00:04:55]Now I can start up TShark again to be able to capture all the frames related to this magical moment in which I plugged the two pies in together over the cable and see them communicating in the way that I expect. When I ping Pi 2 from Pi 1, it actually works. TShark has very helpfully captured all of the new and different frames that are passing between these devices now that I've given them a proper identity. Each frame has an interesting clue as to why this connectivity works this time when it didn't in the past. And so I'm going to have to freeze frame my way through this so that we can take the time necessary to explain what's going on in each. The first frame here is the first Pi sending a message out to the entire network, which of course is just these two devices, but it is still the entire network nevertheless. And it's asking, "Who has the IP address that I'm trying to ping. Who has 10.10.0.2?

[00:05:50]Tell me your MAC address. Next, we have a reply from Pi 2 saying, "Hey, that IP that you're looking for is at my MAC address." In the middle, there's these two pairs of ping requests and replies.

[00:06:02]The first one is pretty clearly triggered by this call and response about IP addresses and MAC addresses.

[00:06:09]The second one though, doesn't create the same behavior. And that's the action of this mysterious thing called ARP, caching the information about where to find neighboring devices on the local network. The last two frames here are the same behavior, but in reverse. Pi 2 is asking where to find Pi 1. Pi 1 responds with, "Hey, here I am. Now you know too." And I think that's pretty cool, because it means that every device on a local network keeps its own independent record using the data collected by ARP of who else is out there and where to find. I could go on for quite a while about what ARP is, all of its nuances, why it's both absurd and incredibly elegant, but while I distract your eyes with these little OLEDs showing the ARP lifecycle.

[00:06:56]ARP, or the [snorts] Address Resolution Protocol, maps IPv4 addresses to MAC addresses. When I pinged Pi 2 from Pi 1, that first frame that passes along the wire is Pi 1 shouting to the entire network. And again, that's only these two devices at this time, but this idea also holds to your home network or a coffee shop or the airport Wi-Fi. The devices are constantly shouting, "Where can I find you?" And "I'm right over here." Every device's kernel then takes the information gleaned by ARP and writes it into a cache table that it can reference the next time it needs to send data to the same device. That way it doesn't have to use ARP every single time. That process is what you're seeing represented in these delightfully tiny OLEDs. When there's a successful ARP request and reply, then Pi 1 considers Pi 2 reachable. Pi 2 then does the same on its end, too.

[00:07:55]And after a few seconds, it's normally about 30 seconds with some jitter that the kernel introduces, but I made it really short here on purpose. That state falls to stale. There's still a record of it, but because IP addresses can be changed, it can't be fully trusted.

[00:08:11][snorts] And the next time I need to ping or send any traffic from Pi 1 to Pi 2, it's going to first use ARP to confirm that mapping of IP address to MAC address and make sure that the connection is back in a reachable state. This happens all the time across every network that every device you own has been a part of. And it's exactly what I was missing the first few times I tried to get those Pis to talk. I now feel pretty confident in answering why these two Pis didn't start talking right away.

[00:08:46]Connectivity, attaching things via a wire, that comes free.

[00:08:51]Uh but reachability requires devices to have an identity, too. They need both an IP address and a MAC address. Two Pis on one Ethernet cable is a very small network. Uh it's not particularly interesting, and there's no practical difference between one device shouting across the room, so to speak, and just shouting to its nearest neighbor, the other Pi. What I want to ask next is what happens when I throw a switch into the picture.

[00:09:20]That's for next time. But for now, I'm Joel, and thanks for learning about the internet with me.