Effective software debugging requires a systematic approach of continuously asking 'why' to understand the root cause of failures, rather than just addressing symptoms. This methodology involves gathering comprehensive data (logs, backtraces, timestamps), establishing timelines of when failures occurred, identifying what changed during those periods, and understanding the human intent behind code decisions. The process requires avoiding tunnel vision by considering context, checking assumptions at each step, and never giving up when stuck. Key tools include git for codebase investigation, print statements for debugging, and understanding that source code never lies while documentation may be unreliable.
Install our extension to search inside any video instantly.
LFNW 2026: CC 236Added:
You know that that might be more of the projector than either of these. You could try uh you could try bypassing the splitter and see if that changes it.
>> That's what they told us something.
>> Yeah, that's what the AB guys told me over in the other room.
Let's see what happens if I change it to only.
>> It does not like that.
>> It does not like that.
>> Take the splitter out.
>> The splitter sometimes does the artifacting and some computers don't like that. So, here try that.
>> See on the live stream.
>> There we go. Yep. Pardon me.
It's not sway, but it's okay.
>> Actually, let's see here.
No, not supposed to open the Libre office. Damn you.
>> I always PDF my slides.
You said it.
Oh, there it is.
I'm I'm supposed to introduce you and say what we're talking about, but I'm kind of a last minute room captain. The other guy ditched. So, what's your name and what are you talking about?
>> So, my name is Sean and I developed the Kalpa desktop, which is a open susa tumbleeedbased immutable.
Uh, and that's sort of what this talk about. That's why I call it what it is.
You know, immutable but not um so you know WTF is immutable.
A lot of people throw this term around and at the most basic uh all of us that fall under the immutable umbrella, it's a readonly base file system is really what it boils down to.
Fedora Atomics do it one way, we do it a different way. Uh Nyx has something similar. Vanilla OS, they're all I mean, everybody's kind of got their own version of it.
Um so all of the things that fall under the heading of immutable have a way to modify the base file system.
And it's actually a terrible description of what we do and it should die in a fire. And I wish people we would stop using the term. The only reason I use it is because people kind of know what I'm talking about.
Uh so you know why do we call things immutable? Well, most of us don't anymore. Uh Kalpa uh the term we use is transactional because of the mechanism we use. Uh Fedora atomic, they use the atomic moniker. Uh you're a NYX guy. What what does Nick call their model?
>> Oh, some pretentious academic thing.
>> Sure.
>> So, uh >> immutable reproducible.
Yeah, that's >> everybody's calling us and it's done that way honestly because not all of us are using the same technology to achieve what's going on.
>> Uh so why is a in my case a transactional system something you would want? Well, so on a traditional system, uh, which I'm talking about a a readwrite, fully rewrite system, you know, Tumbleeed, Fedora, Arch, Debbie, and pick your DRO. Uh, you all of your updates when you're getting your packages from the repositories are going to be running as root and have full access to everything they want to do to the file system.
And it can have unintended consequences.
uh Tumbleweed in particular being an open susa guy uh if you run up run a zippered dump on a live system it's going to update libraries underneath your running processes uh sometimes they will crash. One of the famous ones is the desktop crashes because it restarts services and if you haven't done the precaution of opening up a screen or a T-m session and running your update in that it you get a broken update.
Uh and then there's also weird edge cases with some hardware where hardware uh doesn't get brought back up when the services come back up.
So what we do with Kalpa is we never touch the running system. Uh we snapshot your running system using the Butterfs snapshot features and then we run a traditional zipper transaction in a new snapshot and then the snapshot gets set as your default snapshot. You reboot the machine and you've got your updates.
Uh and they're fully atomic. Uh if something goes wrong with the update, it just discards that new snapshot and then it tries again in the next maintenance window.
Part of Kalpa setup is we do do automatic updates of things. Uh I just had a fun one. I was over in the AV room checking and my computer decided it wanted to do a firmware update.
Great.
So, and in the rather rare event I I've been working on this for three years.
I've had twice this has happened where I've booted into a new snapshot and usually it's Mesa or the AMD GPU driver that goes funky.
All I have to do is roll back to the snap last snapshot and I'm fine.
Uh this is I don't have a a graphical tool for doing this yet. I'm currently developing one, but this is how I would do it if I needed to. You just go into the terminal, it's transactional update, roll back to the previous snapshot, reboot the machine, and all of those changes are gone. And actually, in the next cleanup, it will delete that snapshot and start over again.
So, you know, what is the big deal with this? It looks just like regular old Linux. All the files are in the same places. uh in the case of Kalpa, you know, etc is still read because you have to configure things. Uh I leave user local readr. So if you're doing things locally, you can still use user local like you always would um if you're doing development or whatever, but it isn't really uh zipper being the open source of package manager. You can see if you try and run zipper in I just picked neo chat off top of my head. It's going to tell you you're in a readonly file system. We can't do this.
Uh there is a way to do it which is through the transactional update command. And what this does just like you would a regular update. It will generate a new snapshot, install the RPM, and then set that as the as a snapshot to reboot into.
I don't have it well documented because I'm still writing some guard rails around it. You can do a live apply in this case if it you're just doing an application.
If you're doing something that installs drivers or changes a bunch of libraries, doing live apply is really not a good idea. Um, on any system, it's not just Calba.
But my p I I posit based on the way I do this, you shouldn't do this.
Uh most of your desktop applications for a desktop context don't need to be in the base system.
Uh and my primary application delivery is flat packs through flat hub.
uh because it does create a separation between the base system and the desktop where all of your applications are living in your home directory and you can just smoke your home directory if you break something which I've done multiple times and most of this is done through discover which is provided by by plasma uh by the KDE developers.
If you're on Gnome, it would be Gnome software.
Various different desktop do different things.
So, you know, if you need an application, you just use discover uh or if you really like the terminal, you can use the the flat pack command in the terminal.
Uh and everything that by default is installed user. So, but then you run into the problem, well, flatub doesn't have this application.
>> Please get it.
>> What am I going to do? Uh, I don't support app images out of the box because I don't want to deal with the old app images in the fuse 2 libraries because that's deprecated and go away please.
Uh, but you know, Dro Box is there by default. Uh if you've got an application that's only in the AUR, spin up an Arch Dro box works fine. If all you've got is a deb or something Ubuntu specific this is basically the the uh the the command line workflow for it. Uh you spin up a container in this case I did a a Debian container install my application and then you do distort box export and your application will be there in the menu. And from the user perspective, the first launch of it, yes, you notice a slight delay because it's bringing the container up. But I'm talking a couple of seconds and actually once the container set up, you don't have that delay anymore.
So that is that method for that.
So in summary, uh flat pack would be your first option. Dro box be your second transactional update last rarely and only if you really need it. Uh one of the applications that I have installed through RPM is VS Code just because the dev container setup is so much easier if I've got it installed as an RPM than trying to do it as a flat pack. Uh pretty much flat pack sort of right now falls on its butt anytime it has to get out of its sandbox.
So there are application if if you're uh in doing audio work and you need that direct hardware access, you would probably want to use transactional update install.
Uh oh, I actually got through that faster than I thought I would. Uh but I I get a lot of uh when I say the immutable thing like Fedora atomics they are modifiable through either layering RPMs that has a downside of making your updates a little slow because it's constant it's rebuilding the image locally basically or what you would do is you do like what the universal blue folks are doing with basite and bluefin where you just add what you need to Docker file and you just have an image that's already got your extra stuff built into it. Uh, and they've done a really good job of making that pretty painless. You know, if you like the Fedora ecosystem, uh, I haven't looked deeply into vanilla OS yet to see what they're I there systems using LVM provision uh, thin provisioning and an AB root setup which is very similar to what Android does.
Uh but there's a lot of misconceptions out there that I see constantly about, well, it's immutable. I can't modify it.
No, absolutely you can. You've got root access. Technically, you can take this and remount everything, read, write, and it doesn't care. It's just you're gonna potenti Open SUSA because that's what I'm based on has a pretty bad habit of if you start adding extra RPM repositories outside of the ones that go through testing. So like the old Ubuntu PPAs, you can run into some really weird stuff and update breakage. Uh, the other reason I'm set up like this and using the flat packs is this gets me around most of the problem with the video software patents for the codecs because those are coming those are in the run times from flatub.
Uh the one big sticker, and this applies not just to me, but to most of the quote unquote immutables, uh is if you need the Nvidia proprietary video drivers.
Luckily, I have a community member that wrote a very nice tutorial on how to get those to work. And that is on my website. Uh I'm trying to document as much as possible.
So, that's kind of what I got. Uh >> if anybody has any questions, I'm more than happy to try and answer them.
>> Sure.
>> Yeah.
>> So, so I'm pretty I'm pretty new to the land of immutable desktops. I recently started using Fedora Silver Blue and was you know doing like mix home manager for my flat packs and stuff. I um you know like for the epic proprietary drivers and such for the um like or and then you know for like the Nvidia drivers and things like that uh my you know sort of mental model is that it's like creating this you know commit and it's basically whenever you have to update the base system it's then trying to apply this like patch on top of the base system.
>> That's what RPM OS3 currently does. Mhm.
>> And so is that is that how it's going to work similarly in open like what are the differences in terms >> No. So as far as the systems concerned everything happening other than being on a readr file system it's all traditional RPM transactions under the hood. Uh I I like to use the term that we compose locally. There's no image up in the cloud that you're pulling as a blob and then you know applying.
We're just using the Butterfs snapshot capabilities to protect you from some of the things that can go wrong.
Uh, as I said, we do the last time I talked to the Fedora Autom guys, I don't think automatic updates are working yet.
>> Yeah.
>> Uh, they are working with Kalpa. There is a there is a counterpart to Kalpa called AON, uh, run by another developer that's the Gnome desktop. Uh there's some slight differences between what I'm doing and what he's doing, but we're based off the same setup.
And uh so we don't have the issue of having to do this compose where you're making this commit with the extra stuff.
And I think if I remember right, that capability is going away when this switch to boot C comes.
>> Uh I think what they're going to be there is using uh systemd system extensions.
which somebody will probably crucify me for saying it wrong because I don't it basically what you're doing there is you're building a system extension that gets uh mounted as an overlay uh and doesn't have the problem of having that commit and the recompose locally on Fedora atomics. So it should speed things up considerably uh if if that's the ecosystem you're in.
Anybody else?
>> Um, you're using uh Butterfs uh snapshots for this. Is there plans to support like other copy and write things like ZFS or Bcash or something like that?
>> There's not simply because I can't ship open ZFS >> because of open source of policies and that's where I'm working out of.
>> Uh, and I just don't want to touch bcash FS because I'm dealing with another router tree kernel module.
>> Uh, it's not that I dislike Bcash FS. I just from a maintenance perspective and and an enduser support >> some trauma associative.
>> Yeah.
>> Yeah. I I actually like Bcash FS quite a bit. Just it's problematic from a from a user support as the guy putting this together.
>> Yeah. I I was wondering what um so I'm a blue fin user um and >> so that you know obviously I rely heavily on flatp pack for um graphical applications and their solution for um things you might want to add that are terminal based is actually homebrew um I was curious >> that's actually funny because >> it has it has pluses and minuses uh you know but Uh >> I use brew.
>> Okay.
>> Uh >> mostly minuses.
>> The uh the the big the only reason that it's not there in default in Kalpa actually has to do with the policies on what I can and can't ship out of the open susa repos.
>> Ah okay.
>> So it works fine.
>> It's just something the user has to choose to install after after install.
Yeah.
>> Okay.
>> Yeah. I use it all the time.
>> Yeah. It's been it's been fine. Um, for me on Blue Fin, I think one of the things that they've done is they kind of in their build process, they they've they make sure that Blue Fin or that Brew is like the last thing added to the path, which helps avoid some of the path issues that can come up.
>> I actually stole their patches to make it work.
>> Oh, okay.
Yeah, I I I talk to to George and to Nol and Kyle pretty regularly. So, >> yeah.
>> And most of the time George is trying to convince me to switch to Boot C.
>> He He's a big fan.
>> He is.
>> Yep.
>> Anybody got anything else they'd want to ask? Yes. Derek, >> can I ask weird >> ask weird questions?
>> So, I'm curious. I learned recently that there's this weird systemd reboot thing that reboots just user space and I wonder can you has anybody started trying to abuse that to avoid the BIOS?
>> Uh I know that Richard who is my counterpart that runs AON has been playing with that.
>> Uh from what I understand he's not really thrilled with it yet. Uh, it's not quite doing what it claims on the 10 just yet, but it's also still in fairly active development.
Um, I think maybe it got pushed into system destable a little sooner than it should have, >> but uh I'll be honest on on relatively modern hardware, it's not saving you that much time over over just doing a reboot. Um, I could see use cases for it. I just I I it's not one that I've encountered enough that I' that I've really explored trying to enable it.
>> Yeah. One of my theoretical work projects was going to be looking into looking into ways to be effectively transactional updates and minimizing down time on a bunch of weird medical devices, lol crazy, and giving wreckage. [ __ ] is my career right now. I guess I'm not Yeah. Uh, a lot of what I'm doing here, uh, is leveraging work that others have done and just packaging it in a desktop context.
Uh, one of, you know, my personal design is with, uh, the what I'm doing here is when you install Calpa, you're getting a very minimal install. You get a web browser, you get a terminal because it's not my computer, it's your computer. Why would I load you down with a bunch of applications you're not probably not going to want? Uh, you know, if you like Google Chrome, go install Google Chrome. I don't personally use it.
>> Uh, but you need a way for people to go look up go look up help crazy things.
>> Yep. Oh, yes. And that is the other nice part about this is uh >> this system does protect you from an awful lot of the oh just use this curl command to install >> curl the bash.
>> Yeah. And Kalpa is going to tell you to go to hell because if it's trying to write to any of the readonly parts of the system which a lot of those scripts do.
I got I got saved from something like that recently where it was just like trying to modify some root set from a curl to badge. Yeah. Like, oh, thanks.
>> I've been around long enough that if I run into one of those, I just go package it in RPM and do it that way if it if I if it's something I really want. Uh or in this case, if I just want to check it out and I'm I use Brew to grab it most of the time, assuming it's in Brew. Uh the other one that comes up fairly often are snaps. Uh which I don't philosophically have a problem with snaps as a as an application delivery method. Uh the biggest problem with snaps are if you run currently if you en enable snaps because I use SE Linux and snaps are designed to run on app armor you're they will work on Kalpa but you're losing out on a whole bunch of the security and sandboxing features because they rely on app armor.
That's why it's not there by default. If a user wants to use them they can. I'm just not enabling it by default for anybody because I don't want that responsibility.
>> So, so on this system like I remember when I wanted to add like a new like when I wanted to add the slashnix directory I had to like on Fedora silverblue I had to like enable transient root and whatnot and then like do a bunch of nonsense in order to get that. What would that process look like with with this open? So, right now I'm in the I'm in the process of getting myself the I'm I'm still labeling this an alpha because I've got some installer issues uh with the beta release. The act the best way to do that is actually to go in to the open build service which is where everything's built. Copy my Kiwi definition and add the Nyx directory and generate your own image.
Uh, I would be much happier with Nyx and and be more comfortable with supporting it on Kalpa if I could put the store somewhere other than Nick's store and not break everything.
Uh, because VAR is always writable on Kalpa.
If I could put it in var store, it would be great. But I I don't think Nyx currently supports being able to do that.
I mean you you write to the NYX partition. Uh >> I don't think I understand what the problem is there. So >> So as I understand it and I could be I have not looked at Nyx in a while.
>> Sure.
>> Uh the Nyx store is in the root directory of >> yeahNix.
>> Yeah.
>> Thor.
>> You can do a sub volume of butterfs of a partition to do that. But yeah. So I have it >> that I actually can do. I would just have to look into I all all of my the file system is is butterfs sub volumes.
>> Uh generating another subv volume for nyx actually shouldn't be that hard. I just haven't done it.
>> Yeah. If you just gener you know I just generated nicks there then that's just by n config and just mount at a/ next sub volume there. It's pretty straightforward.
>> I I haven't had anybody really bug me about nyx or contribute patches to enable it. So, and since I don't regularly uh work with it, it's just not something I've >> We're very annoying. We don't bug you.
>> Yeah.
>> I early I had a bunch of of folks that were like, "What about Nyx?" And I'm like, "Patches welcome. Please, if you can figure out nyx, I think you can figure out the patches to enable it."
Okay. Anybody else have anything they want to ask?
Okay, good. That's about my time. And uh thank you for coming.
>> And there is that back for you.
>> Thank you.
Thank you for coming.
>> It's a little different.
charge with batteries.
>> You're free to go.
>> Yeah.
Where's the Xbox?
>> That is in the G building down that way.
I've stole various things and also you guys We do that.
I'm sure he's told It turns out yes.
hospital.
>> I think I heard Okay.
>> Right. But >> most of what >> that's got a good run in my >> Right. Right. Nice talking to you guys.
>> I guess I need box.
>> Yeah, good.
That's running.
Try the one.
What time is that? 10:30.
What is it that 10:30?
>> It's 11:30.
>> Oh, 11:30. Okay.
That's right.
>> Yeah. Whenever you're ready.
>> Whenever you're ready to order.
>> Yeah. I'm ready to help get up though, I think.
Hang on.
I was concerned about that chair going away.
speaker.
Hey Justin.
>> Hello.
>> You doing okay? You get all the announcements and stuff >> in the >> just like are you keeping up because I >> Yeah, I've been reading it.
>> Okay, good. Um yeah, so we're just waiting for the next one. captain shop.
Maybe he's doing the list of who needs to go where in case there's some weird overlap. So, um just stick here for another 30-ish minutes till the next session. Hopefully, the next they will be here the next person.
>> Do you want me to >> Oh, yeah. Fix the tripod now.
>> Something with that.
>> Yeah. I don't know where the adapter is for it. And I don't know why this thing doesn't want to spin if it's like locked or something.
tight.
>> I can just really That doesn't feel right. I'm going to break it.
>> Hold on. Hold on.
>> There we go. Yeah. More.
>> It might be tight.
>> Uh let me just really tight. Yeah.
>> Okay. Hi there. I'm the speaker.
>> Hi.
>> We're at 10:30.
>> Yep.
>> Adam Williamson. I'm in the right room at the right time.
>> Um yeah, you're early, so that's great.
>> Always a good start to the morning.
Yeah, we're going to switch out uh over here for you.
>> Okay, take >> just stick around here and take I'm going to ask you take this off again for me.
>> I'm actually keep this.
>> Give me a minute to here.
Oh yeah.
>> No.
>> Yeah, I think all of them have.
>> It should be okay.
I just want to get a good cover.
No window. Thank you.
>> Oh, right. I don't have a network. Do we have like Ethernet up here or >> Oh, you need internet.
>> I don't necessar I just I just want to open a website right now. Okay. Yeah. I don't have the Wi-Fi information.
>> Uh name is Linux Press and I'll get you the password.
>> Okay. Thank you.
There's a BTC hotspot.
>> No, it's called. Sorry. Let's see.
>> Oh, maybe it's not. Uh, >> not seeing it. I mean, it's fine. I can just tap it to my phone. Maybe the easiest thing to do.
>> Yeah, I think that's probably best.
>> Let's just go with that.
>> Um, and then do you have a HDMI connection?
>> I do. Yeah. Uh, >> let me get that set up. This >> We also have an adapter.
>> No, I I did the AV check. This works.
>> Awesome. Just need to put the Unfortunately, it's on this corner. I don't know if the one on the other corner works. I mean, we can try it. If it doesn't, it's fine.
>> Yeah. So, >> Oh, wait. There isn't even one on that corner. So, we got to go with this side.
>> So, is that a Are you see my mail on that end?
>> Uh, yeah. Uh, it's Yes.
>> And then, so >> it goes in the splitter. We tested it works through the splitter.
>> Oh, really?
>> They told us to They told me to in the in the media check, they told me to plug it into the splitter. So, they pulled I guess one of them. Yeah. Oh, sorry.
Make sure this is good. Let's try it.
And then there you go. Right here. Okay.
>> There we go.
>> There we go. Bingo. All right.
>> Make sure to put this in the box. I'll just put on top. You're going to We're going to put it back in system.
>> Um I'm just going to I'm getting cold.
Uh, okay. Good.
>> I'll be back before the 10:30.
>> Okay, that works. Excellent.
>> Oh, I should see if I can get power to should last just in case.
Oh, right. No, it goes up there, doesn't it?
Does that work?
I need to stop the presentation to find out.
It kind of does. All right. Excellent.
Good.
>> Yeah. Look, this is a friends of the presenter can. I feel like this is going to be a Friends of the Presenter kind of room.
>> Actually, actually got a shirt about a month ago says the five steps of debugging. It works on my machine and then and finally for work. It's like everyone that's done this is like, "Yeah, we Yeah, >> that's good.
>> But anytime you run into that kind of question, it's always a race condition."
>> Very commonly. Yeah. Yeah, that's true.
or it's a driver that you don't have on your system. You know, it can be a few things, but yeah, that's a pretty common.
>> Yeah.
>> Take your water.
>> Hi there.
>> Hey, >> that's I saw you take one of our stickers this morning.
>> Oh, yeah.
>> Wait, where are my notes?
So you just be done.
>> That's funny.
There's a thing in mathematics that's fascinating.
>> Oh, so are we recording through this mic or >> Yes, that is the um >> Okay. So, I need to stay next to it >> at least like relatively close. Cool.
Especially for guards and he's represent Let's go.
>> Yeah.
That's interesting.
Feel like I need a warm-up comedian.
>> Yeah. You guys opening act.
>> Yeah. Yeah.
>> How about that airplane food? Huh? Jeez.
hear music.
>> Yeah, right.
>> The guy in the >> I'll turn the light switch on and off.
>> My Uber driver last night had this like country duo I never heard of playing.
They were really good. I was like, "Oh, I got to take a note of them." I'm glad I did take a note because I've already forgotten their names.
Could play some of that.
>> It's funny that they make like obviously you have to come in early and get everything set up, but the second everything set up like well >> now what I do, right? Yeah. Yeah. You can never get that perfect time where it's like you finished setting up, let's go.
This is definitely better than conferences where you get a 5minute change over and everyone's like hurtling through the hallways trying to plug things in. This is definitely better.
>> It's the benefit of it being like a smaller, >> right? Yeah. Yeah. Yeah. No, I always like this conference.
It's a good vibe.
That AI transcription thing is scarily accurate the whole time.
>> It is. But isn't there hallucinates?
Like earlier it was saying like guys something Well, it's always there's a couple years that this building was online, but you >> where do you travel from?
>> Just Vancouver. Sorry. Yeah. Yeah. So, >> the Canada.
>> Yeah. The Canada. Vancouver.
>> There's two.
>> There are. Yeah. We don't talk about the other.
>> I've been through the other one on the Amtrak. It's certainly there.
There's nobody from Vancouver, Washington here, is there?
>> If there is, I feel terrible.
>> Is Fedora based in Vancouver or is it >> Fedora is based everywhere? Like Fedora is a Red Hat project, so it doesn't really have a home base.
>> There's We have We have people everywhere.
>> You never know when you're talking to a Fedora person. I guess like historically it was kind of based out of Boston.
That's where all the really old school Fedora engineering people are and some of them are still there, you know, >> being cranky Boston people.
>> I work I work from home. I've always worked from home like even before it was cool.
>> In Boston though.
>> Yeah. Yeah. People are always in like weird little satellite suburbs of Boston. Mass and then uh >> western Mass.
>> There is a Boston office now, but it's mostly sales and marketing stuff.
>> The engineers we still have there are technically attached to that office, I think, cuz they shut down the engineering center that was in Westford.
That's gone now.
>> Yeah. Oh. Oh, right. That's building.
>> You're right. Yeah, you're right. I had it wrong. You're right. There's a whole corridor up and down that traditionally was the heart of >> technology had data general ways >> all people off by MIT.
>> Yeah. Yeah.
>> Massachusetts just before the office shut down in West.
>> Oh, >> I'm also Red Hat.
>> Yeah, it's a whole internal history.
There's a fun I I always like the the sort of the fun Easter egg that the some of the older horri bits of the Red Hat build train are all named after obscure small towns in Massachusetts and it's like why is this thing named that >> all those are Massachusetts >> Massachusetts small towns. Yeah.
>> A bunch of the release code names were also >> right the early ones.
>> Well, Windows was always had games on Windows.
>> Yeah. Yeah. there like Washington >> places.
>> I can't decide whether I want to tuck the shirt in or not. Keep going back and forth on it.
>> The thing is is like it's hard to decide when it looks good both ways. Yeah.
>> Thank you. Thank you. I appreciate that.
Anybody else got any geographically based small talk?
>> Who's here from Black Diamond, Washington?
>> No, sorry.
>> From Michigan.
>> Okay, that's okay. Furthest travel.
That's always a good one. Who's the farthest travel? We got Michigan. Could anyone raise Michigan?
>> Virginia.
>> Virginia. Okay. Okay.
>> We're gonna have to go intercontinental next, I think. So, so, so there's the old joke of, uh, where where are you from? And the answer is, I'm from a place where they have in questions and prepositions. So, the response is, well, where are you from, [ __ ] >> You say it's an old joke, but I've never heard it before.
>> I'm older.
I don't know where prepositions.
>> Well, I I I did English in high school.
It was a long time ago.
I used to know what a subjunctive was, man. Those were the days.
>> I took Latin.
>> I did too. I did too. Oh my god.
>> You will have never needed to use >> Let me see. I used You had to conjugate things like pa porum. There's six of them. There's too many of them. I can't.
Oh jeez.
>> It's been so long since high school. My wife and I were picked up the Mad Lib one time. We're like, >> "What is that again?"
Just don't use any sort of >> adverbs and any of that >> like a like a language game kind of thing. Yeah. Yeah. Okay. Yeah. Yeah.
Yeah.
>> So long you just write code.
>> Yeah. All my all my cultural knowledge went out the window when I learned to code.
>> Sorry. code still hadn't burned out.
>> Well, that's true. That's true.
>> That's very true.
>> Here it is.
>> More people. All right.
>> Come in. Come in.
Oh, more people there. I tell you what, since we have all this dead time to fill, I was going to do this during the tour, but it's short, so let's do it as a preamble. Like, I'm curious, like, what's the level in here? Is are you all like super geniuses who are just here to pick holes in me, or are you like >> are people here like I know nothing about this and I want to learn? Like what's the vibe?
>> I mean, I'm banging my head against the wall a couple times.
>> Okay.
>> Yeah.
>> I graduated in the bottom half of my class.
>> All right. You're my people.
>> Let's go.
>> Very curious about all things QE since hole in the QE, >> right? Yeah.
>> Team number of reasons.
>> Reasons. Yeah. I think the reasons are public. Hi.
>> Sea level students.
>> Okay. I like it. I like it. That's cool.
We got a bit of >> Let's go. Sea level student with a student level enthusiast.
>> Oh, it's dark.
>> Oh, yeah.
>> It's not mine. I think it's >> Yeah, my the screen is >> needs to heat up a little bit.
>> Okay, I >> think it's getting better.
>> Oh, yeah. There it goes.
>> And I think what happened is overheated it. Shut itself off.
>> So, let's open some windows.
>> Yeah, I think >> I can feel a breeze. Yeah, this one's open. We can open a few.
>> Okay, >> I can open a few more. Yeah, I can smell a bit of a >> burnt plastic going on in there. Yeah, >> I don't think I get over there if I crawl on, which I wouldn't mind doing.
>> Hey, Adam.
>> Yep.
>> Were you impressive for when we played Jeopardy last night?
>> I was not. Oh, well, >> we really could have used a software quality assurance person.
>> Oh, yeah. What happened, >> Jeopardy? Well, the the game software didn't work well.
>> The the buttons.
>> Is it open source?
>> Yes, they did it all using Google Docs.
>> Okay. Oh god.
>> This talk does not cover debugging Google Docs. I'm sorry.
>> Holy.
>> I mean, if it runs, it runs.
>> This is This is true. This is very true.
Yeah, >> I mean it could have been worse could have been running off of uh PowerPoint.
>> I'm sure someone's done it.
>> I used to temp at the it's called the Care Quality Commission in the UK. I'm actually from the UK. I've been in Canada for 22 years. Um and I was there for like 6 months and I did they inspect care homes. So most of the people who work there are inspectors and they have the schedule of you know I've got to be here and do this step at this care home yada yada. They were tracking all of that in Excel and I did like a tiny macro which basically makes the the cells change color depending on how urgent it is and they were like this is the best thing ever and I got like an email from them like 5 years after I left saying we're still using that thing and I was like oh god UK government it could use some work I uh if it's dumb and it works it's not dumb.
What did everybody eat last night? Did you all do the lumbia >> at the social event? Yeah, >> I didn't eat there. I just drank the beer. That the beer slushie was actually pretty good. Yeah, I was going to get the cider slushy, but I left it too late and they said it was all kind of liquidy by then, so I missed out.
Shame. Shame. We went to the teriyaki place up the road. It was pretty good.
>> So, I did the what? Near the Four Points Hotel.
>> No, I mean just up the road from the from the cat from the brewery thing.
Yeah. Yeah, it was just across the train tracks.
>> Huge portions. Holy cow.
>> I went to a teriyak place near my hotel.
Same thing. Huge. I like it.
>> I think it's like a Bellingham thing cuz every restaurant I look up the AI summary is like portions are huge. Like every single restaurant.
>> I mean it wasn't raw.
>> Yeah. Yeah. No, this like I got I was like, "Oh jeez, this like thing weighs 3 lb." Holy cow, that was a lot of noodles.
Hi there.
>> Oh, are your slides going to be available after the presentation?
>> Um, that's a good question. I can put them up somewhere. Um, I have not yet.
Is there the conference putting slides up?
>> It's a good question.
>> We're super organized.
>> Is all the stuff getting recorded this year?
>> Yeah, there's a microphone here and I'm running through a splitter, so I assume.
And it says USB capture on top of it. So I assume there's some kind of recording going on.
>> Gets better with age.
>> Yeah, >> that's true.
>> True. Very true. But yeah, I could I will probably throw it up on my website or something. If you want it, I'll make a note.
>> I have emailed them to the conference, but I don't know what they're going to do with it. Actually, I should email them again because I kind of revised it a bit. I won't never mind.
Welcome. Welcome.
>> They're still coming. Holy cow. Oh, here comes here comes the Neil brigade. Oh, wait.
>> We This We We're just waiting for you, Neil. We're all just sitting here waiting for you.
>> Man, this is so tempting.
>> All right, I will I will totally do that. It's worth heckling.
>> You probably don't need to, honestly.
>> But look, man. Every time you do this, I get to heckle you. And it's more fun.
>> I I need you in the front row haggling.
I would be lost without it.
>> Seat in the front row. Derek.
>> Derek.
Well, you won because the other one I was going to go to is completely full.
>> Oh, great. So, I won by losing. This is the best way to win.
>> Is that how the wood works?
>> Kind of. If everybody wrote good code, I would be out of the job. So, please stop.
>> You wanted to lower the quality.
>> Yes, that is exactly No, maybe not. Just keep going. Steady state is good.
>> Right here has been paying my mortgage for nearly 20 years.
What does SPQE mean?
>> Uh, senior principal quality engineer.
Actually, I'm technically an SPQ or something now. They keep changing the job titles, but yeah.
>> Okay.
>> That the whole thing wouldn't fit in the slide template. So, >> you get initials.
>> You are.
>> Sorry.
>> You could be you could be the ro you could be Roman.
>> Yeah, >> that's true. I had not even noticed that.
Well, next time when you get a title bump, go ahead and make sure that your next your next would initialize as SPQR.
>> Sadly, it doesn't work. The next is DE, but I'm never getting that cuz it's a whole thing.
>> We can fix that.
>> I don't think you can.
>> I'll write a strong email.
>> Yeah, there will be a strong There will be strong strongly >> strongly worded emails. Purple crayon emails. Yeah.
>> Oh, I think we're on time. Yeah.
>> So, this is Adam Williamson here to talk about root failures.
>> Root coring. Yes.
>> Root causing failings. That is.
>> Okay. Thank you. Hi everybody. Thanks for coming out on this lovely sunny Saturday morning. Saturday morning.
Yeah. I'm Adam Williamson. I am the Fedora quality team lead for Red Hat. Um those are all my contact details. I will put the slide deck up somewhere when I'm done so you can look at all of that. in uh little bit about me. Anyone in here who's old and remembers, you know, the com boom may remember Mandre, which was a Linux Yay. Hi. Yes, I love it. Uh back in the day, I started on Mandre and then I for my sins got suckered into working for them for 4 years until the money ran out. So, that's where I got started in this industry. And then I got rescued by Red Hat calling me up and saying, "Would you like to come and work for us for much more money?" And I said, "Yes, that sounds like a great idea." and I have been there ever since. Um, I started out as the community monkey, which means I was sort of, you know, face of the quality team trying to get people involved in things. Did that for a while and since about 2016, I've been the quality team lead for, you know, reasons. Um, a focus of this talk is going to be that you can do it. Doesn't matter who you are. It doesn't matter what you know, you can do it. I have a history degree. I have never taken a computer science class. Um, I taught myself coding. I do not know what a pointer is. The traveling salesperson should probably get an Uber. Like that's a that's a joke for the KSKY people.
Yeah. So, you don't have to know all of the cool stuff to to root cause failures to find out why things are going wrong, to debug stuff. Despite that, I have contributions to well over 200 open source projects, including the kernel and [ __ ] Yes, despite not knowing what a pointer is. Gnome KD, CP, Python. I'm in all of your stuff, so be scared right now. Um, >> I think I'm quite the opposite.
>> No, you you you you probably got more than me, Neil. Come on.
>> Ah, no. There's no way I have more than you.
>> So, I also my slide deck is mostly just going to be this template and bullet points. I'm sorry. I'm not a super exciting slide deck person. I need more time to write my slide deck to do that.
So, sorry about that. Um, so yeah, I've spent a lot of my career investigating and fixing failures. That's what you do when you're in QA, especially uh in open source. Stuff goes wrong a lot. There's always something to fix. Open source makes it a really interesting job because you don't just have to stop at oh hey that's broken and send it to the team who can fix it. You can go and figure out why it's broken. You can always do this. Um you can just get there. So I came to give you my my insights into how how to go about this complex task after my many years of experience. So uh let's start digging into the details.
Dang it. Uh, yeah. I figured I should address the elephant in the room upfront. Um, AI is a thing. People have a lot of opinions about it.
Maybe just ask squad. Um, so I know a lot of people feel a lot of ways about AI. And if you, this is not going to be an AI talk. Um, if you are an objector to it, then stick around because this is all going to be about doing things the oldfashioned human way. people have ideological barriers, cost barriers, whatever. If you don't have any of those, I am going to acknowledge that LLMs are absolutely good at this kind of stuff and getting better at it. If you have access to them, if you're willing to use it, you can use them to help you.
You probably should. Um, the torque will still be interesting and useful because it will help you ask your LLM better questions, which is obviously very important when you're using one. But yeah, they absolutely are a thing. So, you know, I'll mention it in passing, but there we go.
Uh, I'm going to kind of split this. Oh, yeah. You guys are the alpha test audience for this talk. So, sorry about that. Um, my talks generally get better around the third time. So, let's hope this one goes well first time. Uh, but right now this talk is split into a theory section and a practice section.
Um, so the theory is more about like the approach, the mindset of how you go about investigating things. And practice is more detailed. you know, here are the actual tools I use. Here are some, you know, little tips and tricks. So, let's get started with the theory. Um, this is definitely born of experience.
The very first thing you do when causing a failure when debugging something is find out if someone else did it first.
This is very important. I have twice spent multiple days very painfully bissecting a kernel bug, figured out where it was, looked at the mailing list, found out someone else did it last week. So, don't be me. Don't do that.
have a look around. Um, a detailed thing on this is you may not be able to do this. Always try, but it it gets easier to do this as you go through the process because you will get a more specific idea of what's wrong. So remember to keep checking on this as you go along.
Like at first maybe you just know the thing crashed. You know, half an hour later you may know, oh, it crashed in this line of this application. Then you can go do another search and see if someone else has figured out why it's crashing there. and then you can skip the rest of the process and you know go have a drink. So always remember to keep in check keep checking in on this.
So yeah if nobody else has fixed it first or debugged it first. The first for me the key question the very first thing is to start asking why and never stop asking why. Um this is like my loop essentially is I'm always asking why until I've run out of why questions.
When you've run out of why questions then you have finished. Um, you know, act like a 5-year-old. That's what they do. They keep asking why. It's a great, it's a great mindset for debugging, for root causing. Um, so the very first question might be, you know, okay, what happened, right? What happened that shouldn't have happened? What is the bug? But the next question is, you know, why did that happen? Then once you've established the first why, okay, why did that happen? The application crashed.
Something fell over. Okay, why did it, you know, why did it crash? like go back a bit and then you'll keep working back and you'll be like, "Okay, it crashed because it got here. Huh? Why did it get there?" You're just going to keep saying why. Keep saying why. Keep saying why.
Um, and the why kind of goes two ways.
You're not just asking why. So, I this is going to talk a lot about source code. Um, but you're not just asking about what the code does, right? The why is okay, we went down the wrong branch of the conditional. Why did we go down the wrong branch of the conditional?
That's logical analysis of the code.
You're also asking why about the person who wrote the code. Like everything is done for a reason. Code doesn't just appear out of thin air. So every line of code a person wrote it for or an LLM wrote it at a human's instruction for a reason. You're always asking yourself, okay, there's a reason for this. It's never just there, right? So that's an important question. Um, so context is context is very important.
think beyond the problem you're immediately concerned with, especially about the intent question. Um, so yeah, people don't just, it happens, but it happens very occasionally that people just write outright wrong things like code that can never possibly do what it was intended to do under any circumstance. That's very rare. That's not going to be the case most of the time. So when you're debugging, it's easy to get tunnel vision on the exact workflow you're trying to, you know, that went wrong that you're trying to fix. But you have to ask yourself, okay, maybe this gets encountered a different way where this behavior is correct. You know, maybe there's a different code path through the application where when we get here, this thing behaving this way is correct. Um, maybe it gets executed against a different library and you know, it's only crashing when it runs against X version of the library which you have, but it doesn't crash when it runs against Y version of the library, things like that. So you have to kind of be aware of that and not just get, you know, tunnel tunnel focus on exactly what you're trying to debug right now.
Uh yeah.
Um other questions that you have to ask that are important, not as important as why. Well, you know, at different times um when and who. And this ties back to the human intention. It is um it's very important to establish a timeline when you're debugging things. Um failures don't just materialize out out of thin air. Almost that almost never happens.
You can almost always identify the last time the thing worked. Whatever you're trying to debug, when was the last time it worked? When was the first time it failed? It is crucial to get that and to get it as precise as you possibly can.
Like ideally you want it within you know minutes but you know at worst it might be a few days. Work with what you can get but get it as close as you can. It's very important. Um then knowing the when and also knowing the who. You know once you've identified okay this small period of time you can start figuring out okay what exactly was it that changed during that small period of time. And if you know what was the thing, who changed the thing, that helps you get to the why, right? Um, so look, you're looking for like a correlation between the failure and this this estab this time period you've established where it was working and then it failed. What happened during that period of time? Who changed a thing? What did they change? Why did they change it? These are all kind of interconnected questions. Um, and remember context again. Um, when you're figuring out the delta, it depends on how the thing is deployed, right? The thing you're debugging, you need to know how that thing gets to where it is.
Like, because you need to know, okay, um, to figure out what it was that changed, you need to know, okay, is this deployed out of an operating system, out of a distribution package? Is it just pulled down from GitHub? Is it deployed as a container? And then you can figure out what changed about that deployment method in the time period I'm looking at. Right? That's very important to establish. Um, you might establish that the failure showed up when it was upgraded from 1.5 to 1.6. So now you're looking for whatever change between 1.5 and 1.6 which is related to the failure.
Um, so yeah, that's those are the questions you're asking yourself. Those are the things you're trying to establish. How does how do I think about the actual process? You're basically you're continually trying to narrow your focus without getting tonal vision. As we talked about on the last slide, you start out with this. You usually start out with like a huge space, right? All you know is this thing crashed. You may not even know exactly what it was that crashed. You're just looking at a broken service or something like that. You're trying to get narrower, narrower, narrower, and narrower. Um, but remember to check the context of every step. Very important. Do not um run too far with an assumption. It's very easy to get an idea and be like, "Okay, this must be what's going on, right?" like you get an idea into your head, you know, the server's down. Okay, I think this is probably why it happens. And then you follow that out and you don't check it.
You should always check. There's almost always going to be a way to check each step of your, you know, reasoning chain.
You should always do that because quite often you'll find out it's wrong. You're like, okay, I thought it was, you know, because this service has fallen over, but actually that service is fine. It must be something else. Always check everything you do at every stage. Um, if you get stuck, rewind and try a different angle. Like don't don't get frustrated. Don't be like okay well I thought it was this and I looked at it but I just deadended. Ah I just don't know what to do now. Go back in your process and be like well you know where what was a step in my reasoning? What was the step that I made that maybe that was wrong? Maybe I can look at it a different way or maybe I can just use a completely different approach to figure out what's going on. Never just kind of get discouraged and give up. And as I said earlier, keep asking why until you can't anymore. You can for me I like to get to the very end every time. Neil can tell you this. I'm famous for writing these gigantic commit messages which explain the history of this problem back to like 2002 when the code was originally written. I like to do that because it gives you a feeling of satisfaction and it makes people who are reviewing you know your explanation your fix whatever more confident in it because they know you have explored every angle. So even if you know more or less what's happening, you think you can write a fix or say what should get fixed, you can keep going why you can just keep going on. Well, you know, why was it written that way in the first place and sometimes that will get you to a deeper understanding of the problem and a better fix, right? So you can always just keep going if you have the time to if you can. Sorry.
>> And write it down.
>> Yes, that's true. That's very good.
Always always take notes. Um this is a really important one. And I want to say these last two points are kind of the encapsulation of how I do this. Never give up. There is always something you can do. Um I'm going to go into more detail in the practice side, but it's it's very rare that you are actually completely deadended. I will mention a couple of specific techniques you can use, but there is almost always something you can do. Maybe it feels really dumb. It doesn't matter. You you can do it. Keep going and construct a story. This is maybe my arts grad side coming out. Like as I was saying, things don't appear out of thin air. There's always human intent. So there's always a story behind every failure. And that's to me that's what I'm trying to get to in the end is a story of, you know, how this came to pass, why it came to pass.
And the story kind of encapsulates the whole the whole problem.
So yeah, stuff anyone can do. This kind of could go in the practice side. As I said, we're alpha testing this test, but never mind. Um, you know, there are debugging pro these things will almost always get you to the problem in the end. And these are fairly simple things that everyone can either do or learn to do quite easily. Like one of the most simple things, most important things I do in debugging anything. I'm just searching just searching for interesting text. Like you start, you look at what's going wrong and you find little words in it that you're looking for things that you can Google and that won't give you, you know, a zillion irrelevant results, right? And that gets you a long way looking at things on the internet, looking at things in the codebase you're you're actually working on. get the code to print stuff out. Like you don't actually have to. I don't know C, but I can make CC code print stuff out. Like it it's not that hard to learn how to do that. I can print stuff in like an bewildering array of languages. It is not very difficult, right? Um bisect.
We'll go into more detail if you don't know what bisecting is. We'll go into more detail later, but bisecting is tedious. It's not difficult. You can learn to do it. And if everything else fails, bisecting will almost always get you to the problem or close to it. gross hacks. To debug things, you do not need to be a great engineer who is writing like code that people look at and go, "Wow, that's amazing code." You just need to make the thing happen. You can do it the dumbest way possible. Doesn't matter. So, you know, it's a great way to learn stuff and it's a great way to just mess around with stuff because nobody's actually going to see what you do unless you submit it as a fix later.
So, you don't have to write perfect code. Monkey C, monkey do. Um, even if you don't know how to write C like me or you don't know how to write Python or whatever, you're looking at the code behind a problem, you've got a whole sea of examples right there. There's probably something else in the file you're looking at which does more or less the thing you want to do right now.
You can just copy and paste it and fill around with it till it works, right?
These are very basic techniques that you don't need to go to KSKY school for.
>> Do you know how to print an assembler?
>> Uh, I do not. That one is one I give to the I give to the Ksky people. I'll have to learn that one. Uh that one I give to Peter Jones. Yeah. Uh okay. And yeah, write story. This is just going into more detail on my last bullet point. I Yeah, everything software doesn't appear. People write it for reasons.
Blah blah blah blah blah. I pretty much covered this. This is where I'm going to have to rearrange the slides a little bit. Uh okay, we got through that faster than I expected. That's awesome. So, let's talk a little bit about practice.
Um, first steps like data. More data is always better. Just gather everything you can. This kind of goes to what Neil said. Write everything down. Save everything. However you want to do it, doesn't matter. Use your own favorite workflow. Put it in Google Docs. Put it in text files. Whatever you want to do.
Just have all of everything you can gather. Have it around. Keep it, you know, related to this issue so you know where to look for it. You want back traces if you can get them. Back traces are fantastic. You want journal output, you know, var log messages, whatever your distribution is. Uh, you want other related log files you can find. You want console output from the thing. Like whatever you can get that tracks the execution of the thing that went wrong.
You need to have it. Um, when is very important. Don't just look at the exact point where the thing broke. Go back in history a bit and say, well, what does this look like when it's working? Right?
What do messages look like when it's not crashing? because you want that and you want to be able to compare them, right?
And you want timestamps if you can possibly get them. Time stamps are incredibly useful for figuring out, you know, your windows of when this was working, when this broke. And remember time zones when you're looking at time stamps, that can be significant. I've been caught out by that one a little.
Um, you're looking for a corner piece, right? Like you're doing a jigsaw. This is when you're getting your initial information. An interesting word is a corner piece. A back trace is a fantastic corner piece. like a pattern of you know recurring messages is a corner piece. anything that you know is a place you can start your investigation and you want as many corner pieces as you can because of the thing I said earlier about if you get stuck rewind start over if you get stuck go back find a different corner piece start the process again right um learn how to get a trace like this isn't mandatory but I actually have a again slide thing slide issues I have another slide about this later um for things that compile things that crash there's a thing called a cord dump. When a process crashes, it does a core dump. And there's a process you can use to get a back trace out of that. A back trace just tells you the execution of the program at the point it crashed.
What was it doing? What were the last like 10 things it did? You want one of those if you can get it. This is a simple, it's not very hard to learn.
This is a simple thing you can learn that will help you, but I'll go into this a little bit later.
Corner pieces. Yeah, like as I say, if you can possibly get a back trace, that is always the best corner piece. Um, but also it's not always the whole story. Like sometimes there's a why behind the crash, right?
Why are we getting to a point where this is crashing? And sometimes the crash itself doesn't tell you that. So you do also have to go back a little bit further and say maybe we shouldn't be in the code that's crashing anyway. You know, the crash itself is probably a bug, but it may not be the most important bug. So do remember to check that. Um, just knowing the thing that's failing. It might be a process, you know, it might be a certain thread on a web zone. Whatever it is, get as specific as you can from the initial error to the thing that's failing. That is a corner piece in itself. At least now you're not looking at the entire stack that you're debugging. You're looking at the one thing that is failing. Um, sufficiently distinct word or string in the logs. It's very important. It's very hard to search for, you know, home or, you know, but you you want even if it's not the meat of the error message, if it's a distinct string that appears in the message, that is a useful thing to have because then you can search for it. So note those down.
As I said, recurring patterns of messages that happen in little loops, those are useful corner pieces. And again, this is going back to the time thing. Specific change that coincides with the first failure. Again, if you can get that delta as close as possible, then you're looking in that time like what changed in our deployment during that time period. That in itself is a corner piece. Just like look at that set of changes and say, hm, what's what's important here? Um, reproduction is key.
It is way easier to investigate something you can reproduce. Just it's it's very very difficult to investigate if you can't. So, you should spend a lot of effort trying to make the thing reproducible. like maybe more effort than you think makes sense. It's just it will save you so much time later on trying to figure the thing out. Um it's not quite a debugging thing, but this is a story Neil knows which uh I it's very much related. I wanted to do some development on a a web application in Fedora and it was very hard to make this web application run anywhere except in production. Like it's called a development environment. we didn't have one and that you need that to be able to investigate bugs in it as well as to be able to work on it. I spent like two months building a development environment for it. Then I could write the thing I wanted to write and it took about 3 days, but that was still probably time well spent. It would have taken me longer to do it without having the development environment. And it's the same for debugging. You want to be able to reproduce the problem. You want your reproduction process to be as easy as possible. You want it to be as fast as possible and you want it to be as small as possible. and you want it to be fungeible, by which I mean you need to be able to change it because that's how you're going to debug things. So you you need to be able to just poke it and see the results. Um again, bullet point from earlier, this doesn't have to be clean.
It doesn't have to be great. It doesn't have to be beautiful. Um you can do stuff like for web applications, for instance, browsers can override client side JavaScript and CSS. So if you're debugging a front-end issue, you can just go into your browser tools and just mess around with it and then see what happens, right? Things like that. dumb, you know, messy things like that are fine if all you're doing is trying to figure out why something's broken. Um, another example, when I'm investigating some kind of compiled thing, I generally don't just build it out of the source checkout because I'm a distribution guy.
I tend to work with the RPM package for it because I know how RPM packages work.
I'm comfortable with that. So, when I'm bicting a kernel problem, I build kernel packages like over and over in a loop because I know how to do that. It's not the most efficient way, but it's what I'm comfortable with and it works. So, if it's dumb and it works, it's not dumb. same compile option.
>> That's true. Yeah, it does help with that. Yeah, >> there's actually some value in that.
>> But the point is do it the way you're comfortable with. You know how to do it.
Don't get roadblocked on I need to do this the perfect the most efficient way.
Just do what you know you can do. Um and refine your reproducer as you investigate. This goes back to the thing about you know go back and check if someone else has fixed it. As you go along with your process of narrowing your focus, you can probably cut down your reproducer to so it's smaller and smaller and easier and easier to work with because you know more and more what is it that's exactly gone wrong.
Uh so the tools kind of the meat of the talk I guess what are the things that I I use how do I do this process? Um you going back to my you know non-comsky background you don't really need a lot.
Um, I use a console text editor. I use gedit, you know, I'm proud. Uh, and a web browser and that's pretty much it.
Um, I've never I don't use an IDE. I have cursor for doing, you know, LLM things, but I don't use it for development. Uh, I barely ever use debuggers. It's probably more efficient to use one, but you don't have to. So, if you don't know how to use one, don't let that roadblock you. You can get a long way with just a text editor, a browser, and a console.
>> Well, a surprising large number of language environments don't have a debug.
>> Does that too? Yeah. So it helps to be able to do things with that one. Yeah.
Or not a very good one. Yeah. And GDB is very hard to learn. So and scary. So GP and Git GP are probably my number one most used tools. Um as I said, get a corner piece, get an interesting string, and then just g the codebase for it, you know, and then that will all on its own probably get you to where the thing is going wrong. And that's a huge focus narrowing, right? Git GP is nice because it only searches files that are committed to the repository. So if there are huge ephemeral files like minified JavaScript which gives you page long matches it filters those out. So that's nice. Um I use git for a lot of stuff. Honestly git is a godsend. Whatever you think about it for you know development for investigating things it's fantastic. My little pet theory about git is that apart from the super basic stuff like you know commit and rebase and blah blah blah everyone knows 10 git commands but everyone knows 10 different git commands. So these are some of the ones I know. Um, git log is, you know, obviously super important when you're trying to establish what changed, when did it change, who changed it. Um, some little uh, parameters for it. One line and name status gives you a nice condensed view of, you know, what was the commit message and what files did it change. Git blame is super useful for um, setting up to establishing who and when. So if you don't know git blame, you could do git blame a file and it shows you every line in the file and next to it it tells you what commit last touched that line and then you can just go and look at that commit and see what it did. Right? So if you found your corner piece, you've searched for, you know, you found, okay, it's this function in this, you know, line of this file that's broken, you can run get blame on it and be like, okay, did that line actually change during the time period I established? And if it did, then okay, that's great. We can keep going with that. If it didn't, you've got another why question. You're like, "Huh? So, we're crashing here, but this didn't change during the time period I established. So, why why are we here?
Why are we crashing here when we weren't before?" So, that's always going to be useful. Um, git log dash, that's a capital S. This is this came from when I did another talk very similar to this. I did not know about this and somebody in the audience said, "Hey, do you know about uh, you know, gitlogs?" And I was like, "No." And he was like, "Oh, you're going to want to." And I did. It is fantastic. Uh what gitlog capital s does is it basically searches the history of the git repository for a string and it will show you commits that changed the number of occurrences of that string. So it's basically like a search through the history of the repository. So it's it's like it's super useful for being like, okay, I know, you know, I've got my corner piece. Okay, what changed things with that string in them, you know, in the last x months, the last x weeks, the last x hours. Super super useful. And git bisect, um, if you don't know what git bisect is, uh, it's it's like the brute force way to do this. If all else fails, what git biseect does is it automates the process of you know when it was broken or you know when it worked, you know when it was broken, just build every state. Well, you don't build every state. Like do builds between those two states and note whether they pass or fail until you know the exact thing that broke it. So it'll identify you for you the commit that's the midway point and then you test that and you say whether it passed or failed and if you say it passed then it will you know go halfway and it'll just keep going down and down and down and down and down until you know okay on this one commit it passed this one commit it failed so that means that commit is the problem and that is kind of if all else fails you do a bisect. Um, it can take a long time if the thing if the build it reproduce it stage takes a long time. If that stage is fast, you might want to go to bisect sooner because you'll get to the the thing that caused the problem faster.
>> It also can depend on how many commits are between your >> Yeah, absolutely. That's why you want as narrow a range as possible, right?
Although it it's in a way it's because it's exponential because it's not it's not as important as you think it is, but it can save you like three or four steps. Yeah, >> it's a binary bucket search, but like it's still the buckets can be big.
>> Yeah. Yeah, it's true. Um, but again, this is all stuff that you can learn, right? It sounds a little bit technical, but it actually it's not hard to kind of get dissect. It takes you like 10 minutes to get the concept and then maybe a long time to do it, but you will get to the answer in the end.
>> Typically, I would say a lot of time you can augment that sort of with you do >> you get part of the way. Yeah.
>> Okay. It worked in this pack.
>> Yeah. Absolutely. Yeah. Yeah. Yeah. And then you only have to bisect the bit in the middle. Yeah. So like kernels is a good example for that. you can figure what was your distribution kernel the last time it worked your distribution kernel the last time the first time it failed and then you only have to bisect in the middle right so yeah that's a good tip uh shell small test scripts to verify behaviors anyone can write it takes you know an hour to learn how to do hello world in C but in any other language it's like five minutes and you can probably just pull out this is this goes to the reproducer thing you can just pull out the important bits put them in a standalone script that reproduces a problem that's not hard to learn how to Um, and some languages have shells, like Python has a really nice shell, which is just like a command line shell. You just you run Python with no arguments, and it gives you a little command prompt, and you can just type one line of Python at a time, and it'll tell you what happened. And it's it's a super nice way to play around with things. Other languages, some have this, some don't.
Yep.
Uh, yeah. So pulling it all together, what you're trying to do here is use the tools to answer the questions from the theory part, right? You're if you're answering why is this crashing, you know, use what you've learned to log input values, look for things that are null that shouldn't be, things that are the wrong type, things that are not in the shape the code expects them to be in. Um, if you're answering why are we here at all, log it. Like one of the dumbest, easiest things you can do in almost any language, this is why you learn to print. Just throw a bunch of lines that say, "Hey, I got here." Um, my personal tip for this, put xxx in them. I my my debug line is print xxx here one and just here two, here three, here four, here five. Sometimes I have like 5.1, 5.15, 5.16 because, you know, numbering is hard. Um, but xxx means it's super easy to search for. If the thing logs a zillion lines, you just get for xxx because nobody ever logs xxx.
And that just anymore.
>> Not anymore. Yeah. and that and that lets you know where you are in the code, right? And you can do that in almost any language, in almost any codebase. Just throw in a bunch of lines that tell you where you are. Um, always check which branch of a conditional you hit, things like that. Like if it's if this, else that, put an xxx in each one so you know which one you're in, stuff like that.
Um, and yeah, don't give up or panic or stress if you don't understand everything in the code that you're looking at. You don't need to understand everything in the code that you're looking at. I once fixed a bug in mattplot lib. I know nothing about matt or indeed plot, but I was able to focus on the bit that was broken. Figure out what was wrong with it part.
>> Yes, kind of. Yeah. Figure out what was wrong with it and fix it. I didn't understand the other 99% of that file. I don't know how to draw, you know, pretty pictures of data, but I fixed the bug.
Right. Just focus on the stuff you can understand. Do what you can do. Don't get don't get frozen by the bits that you don't understand. Yeah.
Uh, yeah, make things be more verbose is kind of like logging. Yeah. And read the docs, but don't trust them. This is something I've also found very important. I'm mainly talking about source code here because the source code never lies. The docs frequently lie. Um, they can be useful and you know, different docs are different quality.
Like the Python docs are fantastic. Some random app you're debugging, the docs may be terrible. Um, always check. And sometimes the docs being bad is actually the answer to a why question. The answer to why it does this thing break is because the person who wrote it trusted the docs and the docs were wrong. So, you know, be aware of that. Um, be a proud student at the University of Stack Overflow. Stack Overflow is fantastic.
Like, it's my alma ma. It's where I learned to code. Like, LLMs are just basically reading stuff from Stack Overflow to you. So, use it. It's fantastic. If you don't know the site, it you can ask any question about how to code and the answer is probably on Stack Overflow. It's it's brilliant. And another like really key point, act like you own the place. This is what open source is fantastic for. Um, don't be like, okay, well, I work on this thing and I've debugged the failure to the boundary of this thing, but now it's over in this thing and I don't work on that, so I'm just going to stop and file a bug for them and let them fix it.
Don't do that. It's open source. You can go and look at everything, right? I'm a kernel engineer, you know? It's you can go and look, you can follow the bug wherever it goes and you can mess around with the code. Nobody is gonna, you know, call foul on you for building this code yourself and figuring out what's wrong with it on your computer. It's what you can do. Nobody's going to call foul if you get it wrong a hundred times. If you get it right one time, that's it. That's all you needed. So, just be confident. You don't get in any trouble for, you know, messing around.
It's someone else's thing. Just follow it out all the way. It doesn't matter who owns it. Yeah.
>> Well, because you own it.
>> Yeah, you do. Yeah, you absolutely do.
And if you do this too much, you will wind up being the maintainer of it.
Yeah, ask me how I know. Um, this is, you know, minor things to learn. I'm running short on time, so I won't go through this. But yeah, this is basic stuff. You don't have to know this, but these are easy things you can learn. It won't take you a lot of time.
Dumbass history grad can do it, and they will help you with this process. So how to read a back trace basic concepts of coding conditionals functions signatures basic building compiled things like you can do it if however you're comfortable straight out of the compiler using a distribution package just being able to build a compiled thing is useful llms can teach you all this stuff this is one thing they're good at like asking questions they'll tell you the answers sorry I'll wait for someone to take a picture okay that was that was the end um thank Thank you for coming. I don't know if we have time for questions. Just a couple of things I want to call out on here on the right hand side. Uh I have a blog.
I'm old, so I have a blog. There's a category on the blog. I've written up in exhaustive detail some of the more interesting debugging things I've done.
Um and they're in a category on my blog.
So the QR code over there will take you to that category. The other thing very important, I don't know if anyone knows Julia Evans. Um she does these fantastic, she calls them Z, but they're like little mini guides to things.
They're fantastically illustrated, really well written. She absolutely knows her onions. Like they look cute and nice, but sorry, I I will in a second. I just wanted to show you the cover. Um, she has written a thing called the pocket guide to debugging, which um I had forgotten about it until I wrote my slides, but it basically says everything that's in this talk only better. Um, and that QR code on the left there will take you to the web page for that. And if you want more on all these topics written better, I absolutely recommend you go and buy that. It's like 10 bucks, but she's fantastic. You can kick her the 10 bucks. Um, I don't know her. She doesn't know me. This is entirely, you know, unofficial endorsement, but yeah, get that book.
It's great. That's everything I had.
>> Thank you.
>> Do we have time for questions or >> We have until 11:15, so about 10 minutes for questions.
>> Cool. Anyone got any? Um, sure.
Everyone's hand went up at once, so Um so so all my knowledge is all kernel so >> if you start adding debug or what you call printing it out and the problem goes away it's always a race condition.
>> Yes that's true. Yeah.
>> Um and the other thing is um a lot of times hardware has advanced like if you go from I'll date myself 486 586 and pros are showing up the bug has always been there. It's just >> or or you go from single processor to multiprocessor or your data is now on the cloud, >> you change the timing elements and you have a risk condition.
>> Yeah. So, I'll just reset it for the mic. Sorry, what was your name?
>> Abbron.
>> Abron.
>> Abram was saying um often if if you there's a thing that can happen where you're trying to debug the trying to debug something, you turn on debugging mode or you add a bunch of print statements and the bug goes away. That can be because it's a timing issue where it only happens if things happen in very precise order and printing out all those messages made it slower. Yeah, >> exactly. So, yeah. Yeah. So, that's and that's kind of part of being aware of context, right? As a general topic.
>> Um, and the same thing he said if sometimes a bug can be exposed by hardware changing and again that's context like well it works in this context doesn't work in that context.
Often these are timing issues which is something to be aware of. There's another question over here in the red.
>> Oh, I was just going to say we We didn't go to a party last night. We went to see Project Hail Mary.
>> Oh, that's a great movie. Yeah. Yeah.
>> Which is in essence everything you said.
>> That's true. Yeah.
>> About science.
>> Yes, that's true. So, yeah, you can read Julia Evans. You can also go see Project Hail Mary. Fantastic movie. Do it. Yeah.
>> And my son says the book's even better than anything else.
>> Okay. So, read the book.
>> Okay. Um back there.
>> Yeah. You mentioned 200 plus projects.
you don't understand anything which is great. Um now if you're going to submit your PR for example yeah to make this change um you're going to want to be confident you're going to break something else which brings in the question testing automated testing could you combine like get bicepect with your own >> you know test that verifies you're not breaking something else or at least your test is known >> that's a great question so he's saying um if you're working on something that you're not really a maintainer of you don't really understand possibly the thing you do you send as a fix might break something else how do you how do you avoid Yeah, that is part of the chain of why. That's the one big reason I keep asking why even even when I got to the initial fix is like that's why you say okay but why was this written this way? So if you don't understand the project research it figure out go back through that project and be like okay you want to understand like the context you're changing something in or pointing out a problem in you do want to work all the way back from that but yes test suites are incredibly important. It is always way nicer to encounter a project which has a really good automated test suite than one that doesn't. Like if I'm sending a pull request and you know GitHub actions whatever 4j actions runs 200 tests on it and they all pass I feel great about it. If I submit the pull request and nothing happens I'm like oh I don't know if that was right. So that's not something you can control but just be aware of that. And if the upstream doesn't have a test suite be more proactive yourself in making sure that what you're doing is and maybe even you know sometimes I've contributed tests to upstream so I was you know as part of the fix I was like well maybe you should have a test for this. So yeah, go the for me the answer is like go the extra mile to actually make sure to understand not just the thing you're working on but how it fits in context.
Uh over here >> regarding the statement of every line of code was written that way for a reason.
>> So in the last couple years right most >> uh most lines of codes that most companies are really starting to switch over to being AI generated.
>> Yeah. I I I've encountered plenty of lines of code that weren't really written for a good reason.
>> Yeah, I did I did qualify that statement.
>> Yeah, I'm wondering how that has affected your debugging and thinking about now in the context of much of this code.
>> That's a great question. So yeah, to repeat it for the mic, the question was okay, these days not every line of code is written by a human for a reason. It may have been spit out by an LLM because a human asked it to. and that that there's still an indirect intention there, right? Which is part of it, but and what's my experience with that and how this plays into the process, which is a great question. I would say that honestly it's still a little early. Like most open source projects don't have a huge amount of LLM code in them. Um, right now it's still quite easy to spot when it is. Um, I think that's going to get harder over time, but right now there are fairly obvious signs like it will be very verbose. um there will often be, you know, a gigantic markdown file with bullet points and icons and stuff all over it. That's a great tell that you're looking at an LLM codebase.
And when that happens, I guess first of all, I try and work back to the human intention behind it if at all possible.
Look at the commit message. Look at the whole commit that introduced the thing you want to change and try and figure out, well, okay, what what was the human who instructed the LLM actually trying to achieve when the LLM did this, right?
You just have to it's an extra step.
It's harder to do. Sometimes you may just have to go and ask, right? Instead of there is a that's a good brings us to a good point. Sometimes you can't get all the way through the process. You can't get all the way to the end of the story like right away. You might have to say, "Okay, I've worked it out to here.
I'm the like five layers upstream from where the original crash was. I need to file an issue on this project. Tell them this is where I've got to now. I'm in your project. I saw that this thing changed. I don't understand why. Can you tell me why?" Right? and then it can become an interactive thing. You're asking someone else to help you work the rest of the way in the process. I think that will be more common with LLM code.
>> Yeah. Thank you.
>> As a as a former used to be very involved, >> right, >> and developers on the social side, I guess I >> um writing good um >> posts about exactly what the problem is.
Yeah. are really valuable. Great. Even if you don't have it fixed.
>> Yeah.
>> Because if you say, "I tried to do this.
It worked yesterday. Today it doesn't work."
>> Right. The change was blah blah blah.
>> Yeah.
>> Then they can make >> Yeah. Absolutely. Yeah.
>> And maybe it's a really easy fix, >> right, >> for the person who knows all the context. Yeah. Yeah. That's a great point. It's I I've presented this like it's kind of a solo process, which I sometimes do. It's isn't always great.
Of course, open source is collaborative.
So if at any point you really are kind of stuck, you can't rewind, do more, you can take what you've got so far, put it in an issue, take it to a chat channel, whatever the support system is, and just say, "Hey," and the fact that you've done all that work gives you credit with the upstream. You're not just saying, "Hey, my thing crashed, fix it." You're saying, "Okay, I figured out this was crashing here because this changed, yada yada yada." That makes the work both easier for someone else to help you with and it gives them much more motivation to do it because hey, you did half the work. Now I'm going to Another hack that I have there is um sometimes you can send a really wrong pull request. This is evil but it works. If you've more or less figured out what the problem was, you don't totally know how to fix it.
Send a dumb fix cuz this nerd snipes programmers. Programmers hate bad fixes.
So they will look at that pull request and they'll be like, "No, that's wrong."
And then they'll fix it the right way. I have done this multiple times and it does work. It's evil, but it works.
>> Last respect.
>> Yeah, that's my evil tip for the day.
Uh, any more questions?
>> Don't make it.
>> I mean, you know, don't do it in a malicious way, but you and you can just say, you can be upfront about it. You can be like, "Look, I figured it out. I know where it's going wrong. I know this pull request probably isn't the clean final fix, but it's what I got to, you know. Can you look at this and Yeah. Can you look at this and do the proper fix, right? That's the less evil way to do it.
>> Please do it the less evil, >> right? I just felt the evil came over me for a minute, Neil. I couldn't control it.
>> I was doing so well.
>> You were.
>> I worked for Red Hat. I've got to be a little bit evil. There's another question in the back.
>> I just I just want to let you know this.
I I managed a lot of young developers and I found out that one of the best uh hiring routes is to get uh young people who don't actually have CS degrees but have come up through the QA route like skills.
>> Yeah.
>> They can narrow down and find a problem in code, >> right?
>> Learn how to fix that and then you can ramp them up to >> to the rest of Yeah, that's how I that's how I learned. And I've learned coding just by fixing little things. And then at some point I realized, oh, I can actually write a whole thing now. It just kind of happened without any intent. Yeah. Cool. I think we're mostly at time now. So >> thanks a lot, guys.
>> You know, she sends a weekly newsletter.
>> I I follow her mastadon, so I get it all on there. Yeah. But she's great. I love doing it. She's awesome. It was I finished I worked like four hours on this slide deck and then I was reminded that this existed and I was like dang it I should just have pulled up that yes my slide. you you and I have the same technical training, right?
>> I want to become a lawyer, but um it was actually a strength with my team and you probably find it a strength too because I was you never taught how to do something. So I have a different toolkit that everyone >> absolutely having diversity of approaches is super valuable and I found like history like get history and historian is great like it makes it great. It makes it >> Yeah. So having a diversity of viewpoints definitely it absolutely helps on the team and you need the ksky people the people who can write compression algorithms you need them too but it helps to have other people I can't hang out but but my >> that's not where you came into I've only have one class in like thousand years but anyways it's a strength >> yeah I agree I totally agree >> yeah all very good thank you >> thanks >> so >> I've always want to contribute to like major projects but like it's always an analysis process where do I begin how do I identify a problem that needs to be fixed like how do you go about like even finding a first thing you need to work on.
>> Honestly, I just just itch scratching is always the way I've done it. I think being a community person is useful because you wind up scratching other people's itches. So, I think that might be an interesting way to do it. Just go into like user communities that you're already a part of, you're already interested in, and try and help people out. Got it.
>> Like that's that's probably how I how I got started with it. Yeah. So, yeah, it's my best tip. Appreciate it. You're welcome.
Okay, I should clear out for the next presenter.
>> I just want to shake the hand of an other early Mandre user.
>> Oh, thank you.
>> I just loved Mandre.
>> It was it was it was fun >> cuz I started out in Linux Chicks and somebody said, "Well, have you tried that?" Because that is because if you don't like asking your son for advice all the time, >> we do have We did our best. We tried.
>> Yeah. And so I I could ask the bear, >> right? It was a super good community. I didn't like that about the magic team.
People were really nice.
>> Well, we're not going to do it anymore.
>> We There was never any money that was problem. It was constantly It was basically subsidized by the EU government by the EU. They were constantly trying to get like projects in the EU cuz nobody bought it.
Honestly, the sales were terrible. Like people don't want to pay for stuff. It was a shame. But it was >> fantastic logo.
>> We had Yeah, the logo is really cute. I don't know who drew the logo. That was before my time. I don't mean to put in but I bought so >> I bought Mandre Linux in a box at a Best Buy in 2004 that >> you are one of the few people who paid my >> I downloaded it. You know my first computer was a start in person.
>> Oh Jared.
>> I thought you were talking to us at the same time.
>> Uh no it's a two.
>> Oh no it's the same time as another talk I wanted to see. That's it. Yeah. Yeah.
Nice. Not that exciting. Finally.
>> Likewise.
>> Been co-workers for what?
>> I know, right? It's so weird. We never actually got into each other.
>> 18 years. Yeah. I just moved back to the West Coast from Seattle area, but there I'm in Tacoma, Washington.
>> My uh daughter is a sophomore at UDub.
>> Oh, cool.
>> And that was part of why I moved out.
Awesome.
>> I'll be down in Seattle in I think August for a whole steady concert. Yeah.
Just for overnight. So, it's cool. That was nice to meet you in person, finally.
I'll come to your talk.
>> Yeah, I'll do it. You'll be >> impressed and terrified at the same time.
>> Maybe. I >> hope so.
It's okay.
>> Yeah, we'll get you set up.
>> Yeah. Sorry. I'll get out of your way.
>> I lost people. I met you here.
>> Well, not you, but I'll meet you later.
You all here for the local offline AI?
>> Yep.
>> Wow.
>> I haven't really done it. I haven't really played around with stuff locally.
>> Okay.
>> Great.
>> I just have a brain staking new Adele R 660 to use.
>> How much uh how much RAM do you get for that?
>> Uh 384 gigs of RAM. Jeez.
>> Uh, it was a it was a present because someone decided to throw away a perfectly good server that would need to be replaced.
>> So, uh, I got >> Wait, that's that's RAM, not VRAM, right?
>> That's RAM.
>> Okay.
>> VRAM is what you need.
>> Well, if you'd like, you can watch the live stream on YouTube as well. There's a link to a virtual space.
>> Okay.
Sorry, I just want to close all my stuff except the talk here.
>> Yeah, this is full. David, you want to make sure to let people know that it's full in here? No one can sit down anymore.
>> Yeah. Okay.
>> Is there a live stream?
>> There's a live stream.
>> Working on that now. Yeah.
>> Okay. There will be as well.
>> Yeah. Okay.
>> It's live streaming actively now.
>> Okay.
So, let me see.
I'm only seeing one display so far.
>> Is that expected?
>> Try again.
>> One sec.
>> First one of the day.
>> Um, >> what time?
>> Um, >> this one.
>> Oh, don't use the loop through. You want to get rid of this. I can't do it. And plug it in the input there.
and then do restart OBS. Oh, no. You're good. We got it.
>> Uh, just it's doing dual display mirror.
>> Okay. So, right side.
>> That's a good question.
>> Okay. I'll find I'll figure it out.
>> Uh, hey, excuse me. The room's full.
>> Sorry.
>> I'm going to get you a full sign. So, >> Okay.
>> Oh, that would be >> Yeah.
Okay.
>> Okay, it's working.
>> Thanks.
>> Okay, we're live. We're good.
>> So, we have five minutes until you clean your camera. There's all this green stuff on.
Okay, I think I'll do that over here.
>> Okay, this is weird. I need to figure out which side I'm on. Okay, that screen's going to be on my right.
Good.
>> Sorry.
>> No, you're good.
>> Oh, >> okay. I don't have to look up there. I can just look right here. That's pretty sweet.
>> Testing 1 2 3 for this room.
>> Is there's a live stream available on >> YouTube?
>> Yes, on YouTube. And there's a link on Linux restaurant quest, but we can't accept anymore.
>> Thanks.
>> Just all the space we need.
>> Maximum capacitance.
>> Oh, yeah. There we go.
>> Can I go down a little bit?
Um, I >> don't think so.
>> Yeah, perfect.
>> Full resolution.
>> How far did everyone travel?
>> Michigan.
>> Michigan.
>> Iowa.
>> Iowa.
>> New York. What was that >> back there? Okay. No. Anyone further than New York?
Oh, that's cool. Virginia. Okay. Yeah.
See, >> nice. Local. Local.
>> There'll be a lot of like local theme today.
>> I'm from Seattle right now.
And um I don't know if yall saw online, but I gave this talk a couple times, but now it's a new talk. So, this is really mostly about tools and MCP.
I have not started yet. This is not the talk.
>> MCP. So, are you digitizing this taking us into the Tron world?
>> Thank you. Okay. This is a reference to master control program from Tron. Thank you very much.
>> I appreciate a good Tron reference anytime.
So, is this this this mic and this camera? So, I should probably stand right here, not like walk all around. Is that a little easier for you guys?
>> If you go to the about the midpoint of the front of the Yeah, you're out of frame now. You just went out of frame.
>> Okay.
>> So, everywhere from >> So, if I want to hide >> Yeah. I go in the middle of the room.
Okay.
>> Yeah. Yeah.
>> Interesting. And then >> Oh, yeah.
>> We're full in this room.
>> I'm flattered. Really?
>> This room was full before the last class of you left.
>> Oh man.
>> Who was asking? Someone was asking if there's a >> Is anyone in the uh It's a Matrix chat.
Linuxfest Northwest.
No, >> we're full.
>> Yes. What? Oh, yeah. Would you let them know they're live streaming?
>> If you could. Thank you.
Anyone from Seattle? You heard of Seattle Matrix?
>> Oh, yeah.
>> I actually have.
>> Okay. Check it out if you haven't. Uh, just search Seattle Matrix. It's a kind of a let's see little group of folks who keep a bunch of stuff online that's useful and local and friendly.
>> Okay. Can I start?
>> Just to You pronounce your last name Monson.
>> Monson. Yeah.
>> Yeah. So today we'll be listening to Adam Monson on local AI. Here we go.
>> Thank you. Thank you, Jason.
Welcome. Welcome y'all. Uh good morning.
Thank you, Linuxfest Northwest, all volunteers, all sponsors. Uh it's an amazing conference. I'm happy to be back, honored, and uh really flattered.
Uh my main tech interests lately are self-hosting and local first. Um, those are just kind of two things on my hobby end that get me excited lately. Uh, sorry it's full and u, but it's online. Am I going to be doing that a lot?
>> Fine. So you think about >> So I'm self-hosting a bunch of stuff including some large language models with just enough hardware for me and my family. And part of what I want to demonstrate today is that it's doable even with 8 gigs of RAM because that's what I have and that's what I'm playing with um for smaller experiments learning and primarily that's what I'm hoping to do today is just impart on y'all a bit of uh what I've learned and maybe you already know and I mean you'll teach me it always happens as well but a little bit of knowledge of this stuff that's that's new and sometimes scary and interesting. It's full. Sorry, but it's online and it's it's worthwhile for us learning. My intent is that I I I would hope that you learn it and take it on to do some some more good in the world. Uh help other folks.
So lately, it's become easier as well because the models are getting more efficient, uh smaller. They're meant for mobile. They're trying to uh hit us where we are. So stuff runs on mobile a lot better, edge and um your own stuff at home. All of these latest developments you see in the news are quite useful for uh self-hosters, local offline data and compute.
Um there's two kind of distinct chunks of compute. I hope this is shouldn't be news, but your kind of basic stuff, a general purpose computer, your CPU will do anything. GPU special purpose, so just CPU and GPU. There's lots of other XPUs, but those are the two we're going to focus on today.
Any questions? It's all free. You can remix it. So, these are the two things I'm focusing on today. We'll start with tools and journey on to model context protocol. Not master control program.
I'm alo I'm a big fan of Tron as well.
Uh tools are functions described to a model. So you tell you you just imagine a text chat with an AI and you're saying, "Hey, also I got this this tool here. You can use it and here's what it does. Here's how you call it. Is this is this new to anybody? Have you heard of So who's heard of tools? Who's really heard of tools? Okay. Oh, that's cool.
Who's using tools? Who's written a tool?
Okay. Nice. Nice. Well, I I'm gonna call you up here in a minute. No, I'm just kidding. Um but let me know if I get something quite wrong as well. That happens, but not often.
More often than I'd like, but not that often. Okay. So, yeah. So, so tools, you you you're in a chat. Let's go back to our chat. You're describing to the model. Hey, I have a question and I have a tool. Um, let's let's do some good work and it and it responds helpfully.
We'll get to what it does. And MCP uh model context protocol is a convention um more than a protocol. Well, I don't know. It's conventions for reusable portable tool servers. So, if you got your tool, a thing you're going to tell a model during a chat, then your MCP is a toolbox where you're talking to the model as well, and you're going to say, "Oh, I have act I have access to this whole toolbox." Um, I don't One of the examples that I found was a Nextloud.
Who's familiar with Nexcloud? Okay, good. So, you're you're kind of self-hostable cloud and someone's built there's several um MCP servers for NextCloud that handle and integrate with many of the functions calendaring and contacts for example files. So, you can just plug this point this MCP server at your next cloud instance and start doing a lot of things through a text chat with an AI.
Okay, so yeah, tools are wrenches. MCP as a toolbox. Okay. General architecture.
Everybody seeing this? Okay. Okay. So, uh I'm running Olama at home. You need some kind of thing to run your models.
You can there's quite a few of these.
Ola is just kind of a nice easy open source, easy to get started. It's hard to mess it up. You flip it on, you pull a model, you start uh chatting. So, very easy. And these ones that I've highlighted here are all tools capable models. So for a model to be useful with tools, it has to be trained that way and taught to understand what you're going to pass it in text chat. So you give it a tiny prompt and then itself in this case it's full. Sorry, it's online though. It's full.
>> We're full. Um but you can say uh oh lama I got these tool and it's you you give it JSON and O lama converts it to something the model is trained to understand.
Okay any questions on these clients anyone's unfamiliar just like you can give me like reactions or something it helps a lot.
>> No. Okay. How is NextCloud a client if it is itself something that serves like this is your server and netcloud is a client of your server which is doing the AI.
>> Yeah indeed it can do both. Nextcloud really is just like a connection framework. So you'd you'd have a separate and I think what the way they recommend running models with nextcloud is if you have well okay there's a million ways it deploys nextcloud. I don't but it's so flexible it's both let's say is that enough for now? Okay, >> so here's my toy example. This is the the actual work we're trying to get done is right here. What is A time B? Big numbers, hard to do on your fingers, even kind of hard to do on paper. It take you a little while. A silly example to ask an LLM, but interesting because a lot of LLMs until kind of this year couldn't actually answer it. So, uh, that's the question I've posed. And initially here in this whole script, I'm just chatting with an AI. I'm just saying, what is this times this? Do your best or your worst in this case. Um, there's an intentional space right here because I'm going to add more code. Um, but but this is this is any questions on this kind of it's this is Python kind of pseudo code. Um, but I'm pointing it at my own server.
Okay, so here's what happens when you run that script. So I already have the right thing installed. So I can import Lama at the top and then I send it to that's import is the Alama SDK and I'm going to it the SDK is going to send JSON over HTTPS to the Alama uh to the Alama API.
So JSON and gets to a llama and then a llama transforms that into a text template and a text template is what the model understands and the text template in this case is just saying here's a chat message from a user answer it um takes a long time a lot of rumination who's familiar with thinking models or reasoning models okay so what they do is they talk to themselves it looks crazy um but it helps them come up with better answers I don't even know why. Does anybody know why? Enlighten me. Okay. Well, let's learn why for our next talk. Uh, but it does work. It gives out an answer which is technically correct. It's just approximate. So, we can do better than that.
And certainly we could just open up Python and multiply the two numbers and get the answer instantly. We we'll come back to that as well. But that's that's the whole that's the whole uh motivation here is I know I can do it easily with Python. glue it together for me with a very easy UI that I can talk with. Okay, so here's all the new code. Multiply function added and then I'm passing in that multiply function into the Alama SDK and then that is going to change what's sent to the model itself with me so far.
Exactly the same pipeline.
The only difference here is that this kind of thing, this isn't actually, yeah, this is exactly what the text looks like looks like when it comes back to the Alama. So, it goes like back and forth and then this comes back out of the Alama uh SDK into my Python script and then I get to decide what to do with that. Meaning me or my Python script.
Is that clear? So the mo I said to the model here, multiply these numbers and here's a tool if you think you need it. You know, it's like a pretty obvious hint right here obviously, right? Um but we'll build on this example. So that's it. Our simple script doesn't actually call multiply.
It only prints out the model response because that's what we want to see. So let me show you really quick. I'm going to bounce to the chat template.
Okay, so bear with me. There's a lot.
Um, but hopefully you can you can uh sus out. Sorry, it's full.
>> So, this is the useful part right here in my original and second versions of the scripts, right? What's new is tool list. So, this whole tool list is is JSON it's passing to the model, but it's within these special tags. So, the model is actually seeing this text. That's my understanding.
And I got to admit, my understanding is a bit cursory because I got to it just by like you search around some of the terminology is in flux and then I turned on debugging and I'm like actually turned on a llama lets you spit out some more stuff and so I'm like oh okay I could actually see that going to the model and this so what's interesting too is that the model didn't spit this out.
I told that to the model. So my script I added on to the script below there and I said okay if I get a tool call actually called a function and pass it back to the model again. And the idea is like I've done all this work. I've chatted with a model asked it what to do passed it a function to do that if it wants to.
It did and so I executed the function and then I got that result and I passed the whole thing back to the model and now I can talk about it with the model.
Is that all clear? So I could say like approximate this number or is it close to pi or uh do more multiplication or format in a nice little table like anything you could imagine doing with the model you can you can continue building on this. This adds up into what's called our context window. So that thing builds and eventually runs out because you have a limited amount of video RAM for your GPU GPU to deal with or integrated RAM or whatever. You have an you have a limited amount of memory and in the in the world of AI models, it's called your context window.
Okay, so I'm going to bump back to that tool example here. Okay, so remember this this was just back and forth. We got the tool call and considering the the original math problem we asked there's several ways to attack it and I'm not here to tell you one way is better the other. I I think I want to tell you pretty quickly to avoid this middle row but sorry just basically what we have here is the method we use to compute this math problem. Pros for that method and cons for that method.
Um yeah, I mean I don't I I love Python.
I'd rather do this problem with Python, but I forced myself to use AI for it. It was interesting, right? Um but programming is arguably a complex UI.
Talking to something or text chatting, easy UI roughly. Um too slow, but then of course we we um we can get we can make it happen with tools. This is our this is our it totally feels like a cluge to me. But then maybe compilers did with assembly. I have no idea. Um I mentioned Jivvon's par jevans sorry Jevans paradox here. Anybody heard of that? That's kind of a fun one. The they came from the coal mining days in the UK where the more coal we were able to get the more the more efficiently we were able to use coal the more we needed. So you'd think like oh yeah we you know it's just it's a little bit counterintuitive but it's quite intuitive once you think through it. And I think that's something that happens in general with AI and it's kind of my warning to you is use it responsibly. I mean, use it thoughtfully. Um, it makes sense in some places and not others.
Okay, so that's tools in a nutshell.
And then back to MCP servers, which are a box of tools, but it's also this standardized box that fits in a lot of fits on a lot of shelves. you can pass from from many different kinds of clients. Um, they're pretty much all MCP capable now. And I think it won't be long before our cars are MCB capable.
Our phones already are. I mean, anything that changes quickly, but you could imagine your oven talking to your fridge or something like that. I mean, it's it's all kind of everybody wants to do these kind of integrations and so it's it's gradually happening. I again I would if I'm putting my dad hat on I would say like do it responsibly but it's just it's happening across the board because it makes money.
So anybody use these clients? Any of them? Okay. Anybody heard of Open Code?
Okay. It's okay. Um I've been a little happier with OMCP.
Um Open Code is made for coding. It's just there's I think there's a number of cloud code kind of ripoffs that are um not ripoffs, they're better. they're improvements because they're open source. So these um OMCP is one that appears to be designed to work with Olama, but I found it works fine. You can tell it to use any MCP server. So I don't know what the limitation is there, but it's kind of built to work with O Lama. And since I'm I'm kind of all in on a llama right now, you know, I will say there's lots of alternatives to a llama, too. You want to run stuff locally. Um, there's a ton of backup slides in this deck that you can find where maybe you want to try something that's more can more useful for image generation or manipulation or other kinds of AI things. There's there's many things you can run from home and many of them are pretty easy. Uh, local AI is another one. Really easy to use and a lot more image capability.
So these are the kind of things you might um connect an MCP server to uh to add more functionality to an AI chat.
So in this example here, I'll go through.
So I've started up a tiny web server that serves one web page. This is intentionally gray and hard to read because you don't have to know what's in this other All you have to know is that I've got AI. I've got a client that I can use to talk to whatever I got and I got a web page with an internal memo that is only accessible when I'm inside the company and it has the secret information that I want to uh do work with.
Okay, so this is ba basic architecture making sense. The MCP server is that toolbox um I've given it uh the tool that I've specifically given is called fetch. This is one of the uh examples that comes with model context protocol.
Uh if you just search model context protocol, you'll get the GitHub repository with all the examples. This is one of the example MCP servers.
That's all it does. Fetches a web page, spits it out in a format that's useful for a model to digest and uh help you do work on with a chatbot.
So, I've started up Olama, started up my OL MCP. Um, sorry, is this clear? This this little thing is just a terminal app. You can talk to it. You say, you tell it what things to talk with, what MCP servers to talk with, and what individual tools you can give it as well. So, this is your terminal app.
That server is running, the MCP server is running separately. The web page is running separately. Llama's running separately. But all this is under I mean except for the the the models all this just takes up like you know a few hundred megs of RAM or something like that which is not much these days.
Okay. So this is my conversation.
I've I've told MCP about fetch and then I've just asked it to fetch a web page.
um you'll have seen this kind of this kind of uh interaction if you'd use you know the the big proprietary ones anthropic put the big models chat GPT they all use MCPN models but it's it's opaque so it's not challenging though and I would totally encourage you to start these few little things oh pushed the wrong button ah gosh that's not the right button okay sorry about the buttons going to go back to keyboard.
It's too tempting. Got to do it. Okay, there we go. So, ah, sorry.
No, it's not good. Okay, here we go. So, it just shows pretty clearly that you can talk to the model and it can incorporate a fetch. So remember the chat template MCP servers in this this MCP server is pretty simple. So the chat template doesn't grow too big but you have to always be aware of the more it appears to me that the more tools you add the more MCP servers the more you have to add to the context window because you have to tell it all the things it can do along with the question you're asking you you're asking. So when this was about as much as I could get working reliably, it would it would work every time. I would get a consistent chat and I would get consistent answers. But if you're asking bigger, more challenging questions, like imagine that that next cloud MCP server, I would I tried asking I I plugged it all into this the alarm I had running in my OOL MCP and I said look at all my appointments in this year and move them forward by one hour and here's your MCP server and it just immediately runs out of the context window for for the model I was using here um LFM 2.5 built for tools very lightweight. You have to have very brief uh a very brief list of functions it can do, but it can do them well and um and and you still get that nice chat interface. So, I think mobile is like an interesting target here for for LFM 2.5 and a small amount of tools or a simple MCP server.
So this is this is kind of my takeaways. When you're thinking about MCP, it appears to be it's sticking. It's not going away, which in the world of AI today, stuff is coming and going. I mean, I did this talk six months ago and it was all about rag and it's like it's changed since then. So rag is still a thing. You can look it up, but I'm not even going to cover it today because MCP is so much more interesting. Um, I mentioned the context window and just I mean when it comes to security, you know, it's you hear about, oh, I've let open claw do everything with all my things. I mean, don't do that. You don't need to. I mean, maybe you're trying to make a a great YouTube video or something, but uh I think this is a great chance to go back to tried and true tools like uh Open Snitch.
Little Snitch was like this really popular one for Mac and Open Snitch and I mean more and more like there's we have all the tools we already need in a standard GNU Linux install like it's got it's got everything you need and with an AI you can chat with like you know which man page should I look up for the system calls related to security and it's just like pointing you right to those things.
So I think the current advances in AI are a huge opportunity for us to do more uh at home locally, small communities, small groups, families, churches, sports teams. You can do more with less today.
So I think it's kind of a side product of what you know people are trying to make loads of money and they are with with giant foundation models. You do everything. I mean who wants that?
Please raise your hand. Do you want computers to take over everything? I mean, I don't know. It's It's one world, but uh if we're thoughtful about it and careful and take this knowledge and do it for good, I use it for good. I think we can we can do some amazing things.
So, I had some questions for y'all.
Anybody have any any like anywhere any of the slides? Yeah.
>> You you listed several models there. Are you running all of those at the same time or are you swapping them out as needed depending on what you're doing?
Just thinking from the practical standpoint of how much do you need to go spend on GPUs?
>> Yeah. So the question was sorry repeat questions. Okay. So the question was about model size and do you run multiple models at once? Um especially with 8 gigs is is a low amount of VRAM these days video RAM. So my GPU is it's a small little gamer card. Um I can run Quinn 34B and LFM 2.5 at the same time but that's about it. Um, there's another thing in the background that runs on my server.
It's a jellyfin transcoding thing, and that's tiny. That's like 175 megs or something. That's all it needs of GPU VRAMm to do video transcoding and drastically speed up video streaming for my home movies and stuff for my family.
But yeah, they they swap out automatically. Ola handles that for you.
Local AI does as well. So, I imagine it's kind of a general thing. Local AI actually gives you more control. You can you can boot out, you can eject models, you can force load them or mess things up, but O Lama seems to just do the right thing. That's another reason I've kind of switched to OAM and I recommend that these days is because to get started to learn to flip it on. You can start it on desktop as easily on a server is on a server. Um the Docker stuff is really clean for containerizing it stuff or other I think it's you can containerize it other ways as well as Docker. So yeah, model size is a big issue though. your your number. You can ask me as you want. I just the VRAMm is going to be your biggest consideration.
Um if you're going to like >> Yeah. Go ahead. The next question.
>> I'll try not to monopolize. But one other question is um from angle cost.
>> Yeah.
>> Power consumption. Are you doing things to try to >> tamp that down or are you kind of just letting it run and not worrying about it?
>> What I'm doing right now with power consumption is the question. What am I doing with that? Well, I you know, it's interesting because you go and ask Chad GPD something, you have no idea, right?
It could be a small slice of power in a big farm or it could be boiling water in Oregon like they claim on the news. I mean, I I just don't know and I don't think we will know unless they decide to tell us. But at home, it's really fun because you can compute exactly how much power is used. So I can watch my card pull. I think it's aund 130 watt uh idle. So it's like a incandescent strong incandescent light bulb. Then when I flip on my GPU, actually I got a nice slide with that power consumption.
Can you guys see that? I'll try to make it a little bigger. But uh what I'm doing now is in short just monitoring it. So I'm curious and the cool thing about running it locally is now I have a way to monitor it, right?
Okay. Oh, so here's like I'm generating an image with uh this system is called uh invoke AI. Of course, it's right there and you just it's kind of neat because it shows you the intermediate steps towards making an image. But then I have this screencast also of what um power is used during it. So uh yeah, so it's it the card can use up to 130 watts. My server also idles at about 130 watts. So you know 5 seconds at 130 watt. Um so 130 times 5 watt seconds which is negligible. And home power is always cheaper than data center power or whatever I think. Um, but that's as far as I've got with it. You gotta have another question. Come on.
No. Yeah. Front.
>> I have lots of questions.
>> Thank you.
>> As someone who's ever run anything locally, like is or what's a good learning resource if I want to dig deeper or >> lots of links in this deck, especially the take-home notes. So, yeah.
>> Will that be available on your site?
>> Yeah.
>> Okay.
>> Uh, can I get to it?
Yeah, there you go. So that adammonson.com the the highest post today is links to this deck and um sorry this uh this is like just a giant HTML doc that you open locally of course offline and it has backup material is just all the previous versions of this talk I've done plus everything from today.
>> Okay.
>> All free. You're welcome.
>> Thank you. Thank you.
Oh, can I mention one more thing? My uh my biggest client right now is Meos Initiative. They're awesome nonprofit un banking for the unbanked. They bank the poor. They have a table in the expo hall. It's awesome. They're always looking for people uh to help out or just spread the word interest. Um if you've heard of micro finance, Muhammad Ununis, Nobel Peace Winner and early as um this is building on that technology and it's incredible. So, Meifos Initiative, check it out. They're awesome. Any wrap-up qu Are we over time?
>> Oh, no. We have I mean the it ends the session ends at 55 and we can take another 15ish minutes.
>> Well, if you want to leave, please leave quietly. Otherwise, I'd love to entertain questions.
>> So, do you see any um foreseeing business use for this?
>> Oh, yeah. And so, the question is about business use. Is there business use for this? Absolutely. Endless. So, one thing that comes to mind often in my head is like, oh, you got a law firm, they have, you know, 28,000 documents from,400 cases, and really none of it should go online. And you have two main choices these days. One, you can get a no train, no gnome retain contract with a big AI uh firm. And I think that's probably most people's go-to. I think there's a strong use case for uh flip it on with some on-prem server. Uh, but it's it's quite a gap between those two things.
On-prem is so much work. The in cloud thing that you just pay somebody for is so tempting.
>> I guess that's where I'm going because if we have people that need to be trained to do this, like should we be um tooling up in the colleges and whatnot to be training our next workforce to be able to support this?
>> Yeah, it's another question. Should we be training our next workforce to support this tool this up for if it's Yeah, I mean 100%. I think we are. I mean, if you look at the course curriculums, you'll see lots of AI creeping in AI itself. It's just another computer tool. I don't know. I mean, you know, >> this is a different angle.
>> Yeah, I got to learn object orientation still. Now, I got to learn AI still, you know, it's it's fine. I think um but absolutely and this is a great way to to learn it. And I think a lot of the a lot of the stuff you can get for home even newer laptops are uh AI compute capable.
Um I mean if you could get out like the hardware in an Xbox you could do a ton of stuff. So So yeah, I think absolutely please tinker with it. Let me know how it goes and uh uh tell me what you learn.
Yeah, I built up a whole community around a book I wrote about self-hosting and um folks come in and AI was one of the first things that came. It was like just few years ago, can you run it at home too? Oh yeah, somebody released an open model llama and now it's it seems there's as much uh as much development in the open as there is in proprietary big tech. I mean there's some new foundation model. It makes images slightly better. I mean, I don't know.
It feels some of it's kind of plateaued.
And now to me, the really interesting thing is glue. The glue stuff to glue. I want to glue my next cloud server to my jellyfin server. And I mean, I do that in an hour and then I still got to review the code and understand it enough. Um, but it's it's okay. Yeah.
Vibe I can't do much vibe coding with 8 gigs of VRAMm.
>> Um, but I've played around with the the large proprietary stuff to do vibe coding. Yeah. Yeah. Please. What do you think is the maximum use on a personal home desktop with a local offline?
>> Uh, well, it depends on your VRAMm.
>> So, the more the better. I mean, I I Yeah, I >> mean, what do you think you can do and what definitely can't do? Oh well uh I haven't experimented much with this is going to be third hand secondhand but uh 48 gigs of VRAM you can do quite a bit even Vive coding uh MCP servers can I see some nodding so some validation around that but yeah it's going to really depend on how much RAM your AI has access to at compute time and a lot of the newer chips like metal M2 they they combine GPU specific it's inte integrated or something. So, you have access to quite a bit more. Um, I'm curious about Offley's hardware. Maybe some of that is starting to pop up on eBay as well or wherever um secondhand marketplace where you can get. I don't video cards for gamers are hard to find like properly cooled and ones that fit.
I had to actually cut open my case, my one UU case to fit in a GPU and it was a slim. So, I don't know. I don't know much about hardware and I'd love to do a talk or see a talk all about hardware because that's a entirely huge world.
But the the takehome for me about hardware is like it's out there. Start with at least let's say 48 gigs of VRAM.
If you can you can get something with that much. You could do probably just about anything you'd want to do at home.
>> Yeah.
>> If your home lab is like just mainly PCs with no GPUs, is this just out of the question? Not completely, but it limits you quite a bit. So, the question is about what if your home lab is just CPU.
Um, I would say try uh whisper is a model I use a ton only on CPU and it just it's slow, but I don't care. I mean, I'm transcribing took voice memos in the car or something and then I'm like transcribing it later and I don't care how long it takes. Um so so yes there are things you can do even with CPU and also if you can afford to wait like if you're running something overnight I think there's some interesting uh efficient and conservative compute methods that even work on CPU with low RAM. I I don't know. I haven't tried that stuff myself, but >> yeah.
>> Voice box.
>> Okay. Sorry.
>> Yeah.
>> Last question.
>> Let's have a question more of a comment to your your your first question. Um uh my my company we work with a lot of sensitive data. We're in uh uh health health tech. So it's a lot of private patient data that um we do not want to be sending to uh to to the big AI providers. So we've been we're on probably our fourth generation of deploying AI models locally in our own sandboxed environments for it's not quite you know it's it's it's halfway between the own personal uh PC thing and uh someone else's uh data sovereignty um so that we can you know control the boundaries of where that data is going and and uh that's been very useful for us and I learned some some new things about it today, but I think there's a, you know, a million variants of how and why you you'd want to do that.
>> Yeah. Well, let's keep it going. I think there are a million use cases. There's gentleman just uh giving us a use case in healthcare where they're prevented from taking from using the cloud with cloud AI.
>> We we wouldn't want to I mean the whole it's we prefer not to chase down that first legal route and just prefer to just know that this this data is >> Yeah.
>> safe. Yeah.
>> Yeah. Absolutely. Well, thank you again everybody. appreciate you coming today.
>> You've done this before.
>> No, you you just maintained really good control of like the room and everything.
>> Oh, thank you.
>> Yeah.
>> Hey, appreciate it. Thanks for coming to mention a project called mesh.
>> It's a good idea.
Yeah, but at least you like literally have to push the side of >> when it comes to like kind of like is that kind of a similar context like similar protocol they use proprietary using >> both. So I think with co-pilot uh is the online one or is there a local one?
>> I haven't used it's all >> Yeah. Well, I mean MCP itself is obviously open. So if something speaks MCP, you can talk to it. Okay?
>> You know, on on either end, the the the model framework doesn't talk to MCP. A client talks to MCP and then you it it affects what you hand the model.
>> Yeah.
>> So yeah, but I I don't know the answer.
Okay.
>> Yeah. Thanks for coming.
>> Have you tried uh the open web UI for the RN generics?
>> No, I have not. No, I've heard of it.
Other other folks are talking about Seattle Matrix. This gets a lot of >> I've been I've been using it for the last two weeks. I just built out my my house.
>> It looks like chat GPT, right?
>> Yeah, it looks like chat GPT. Uh you can do model toggling throughout the whole thing. Okay.
>> Uh it has obviously rag.
>> Yeah.
>> Um they're they're calling it knowledge, >> but um >> it's it's super easy for uh when you're first learning.
>> I've been learning a lot about that.
uh things like switching your your models and when they when they age out and release the VRAM, there's actually a timer on there for our llama and it defaults to 5 minutes where you ask it a query >> that that model will stay in memory for 5 minutes and then it clears the memory and that model is unloaded.
>> That's feature or or >> it's it's a open web.
>> This is a llama feature that you can you can it's one of the settings you can set.
>> You can manipulate it from open web.
>> Yeah. So your different your different models you can have them live for different lifetimes.
>> Oh okay.
>> But uh I wanted to ask you about uh MCP server.
>> Yeah.
>> Are you running any locally?
>> Well honestly just this like fetch example.
>> I mean I've tried you can run any of them. They don't take up much RAM themselves.
>> Excuse me. We're closing this room.
>> So if we could start moving things out.
Thank you.
I started to look at >> you can run them easily. It's just that you need a bigger model with more things in your MC. So >> like the next one tools should have one tool, >> right?
>> Like what use is it to just to only be able to read or something?
>> Yeah. So this one was about MCP >> local AI he was getting into that MCP servers a little bit.
>> Yeah.
>> So the like nothing too indepth but it's a little more in depth than what anything I know.
>> That's that's my jam.
Just go back to the uh area.
>> Good work. Thank you.
do this for >> I'll be right back.
>> Okay.
Well, actually, All right.
Yeah.
Front left.
Front left.
Front, back.
Heat. Heat.
Heat. Hey, heat. Hey, heat.
Heat. Heat. N.
Heat.
Heat.
front left.
Got it.
because it's Yeah.
Heat.
Heat. Heat.
Hey, hey, hey.
Heat.
Heat.
N you know Heat. Heat.
Heat. Heat. N.
Right.
I don't know how to make that happen. I think we could probably have That's the input.
Yeah, that one.
Cool. Okay, do that.
Uh, technically we're not supposed to let tuck these in until the room happens.
>> Okay, >> cool.
Um, >> I just set that down there. That's great. Thank you very much. I really appreciate that.
>> Oh, yeah. My pleasure.
>> Thank you.
>> Um, cool. I'll uh I can set this door so it'll just lock automatically if >> That's fine.
>> Uh um yeah, and then uh I should be back around here in like 25 minutes or so.
>> Get your room captain in.
>> Great.
>> Um >> yeah, >> I don't suffer. Uh uh uh performance anxiety you quietly >> it it's nice to have uh some peaceful time.
>> It is nice to have some peaceful time right.
>> Thank you very much. I really appreciate it. What's your name by the way?
>> Justin.
>> Justin.
>> Yeah. Good to meet you.
>> Good to meet you.
>> Thank you very much.
>> Yeah. Have a good talk.
Stop.
Good job.
Sorry.
Joe, sorry.
>> I'm Adam. I I think I might have left my name badge in there. Do you mind if I check?
>> Come in.
>> Okay. Thank you, sir.
>> I think you did.
>> Ah, good.
>> I think it is.
>> There it is.
>> Right here. Yeah.
>> Excellent. Thank you.
>> You're welcome.
>> When's your Q&A? uh in about uh 39 minutes. 29 minutes. In fact, I want to ask of you a favor.
>> Yes.
>> Can you stay here for a moment? You bet.
>> I want to go to the little boy's room.
>> Okay.
>> And thank you.
>> Yeah, you got it.
Thank you. Of course.
>> Thank you.
See?
May I?
>> By all means.
>> May I?
>> Are you the room captain?
>> The board. No. Do you know how do you know how to plug in the um my computer to the overhead projector?
>> Oh, yes. Yes. Yes. The uh >> set up with the HDMI.
>> Yeah.
>> All right. Uh so we're So you're trying to get the laptop connected?
>> Yep. So here's an output.
>> So uh this is your own cable. This is my own cable.
>> Uh I think we have we have our own cable set up. So >> Oh, okay. Great. In that case, pull a plug.
Need to figure out where we put it. It should be connected.
>> Uh I was looking for it and did not find it.
>> Uh okay. Uh so I do not know what this blue thing is.
It goes to an orange wire which I thought was Ethernet but no it's not connected anything. Uh, looks like it connects to this thing here, which Hold on. This is Yeah, they they've done some modifications to a setup. I was not >> Okay.
>> I was not not familiar with. Hold on.
Let me figure out where. So So you said that orange cable to where?
>> It goes nowhere.
>> It goes nowhere.
>> It goes It's just laying on the floor.
>> Well, uh uh can I see uh your HDMI cable?
>> Yeah, I'll be here. You go.
>> All right.
Let me just pop this off. Uh if something goes wrong, I can just message the port and then figure out why that's there. But yeah, uh let's plug that in. And then here's the other end.
Oh, there's Oh, that's where our cable went. Yeah, here's our official cable, but we can use your cable.
All right. Well, something happened and >> uh let's see. Uh okay. Yep. There the mic's working. Yep.
Mic and camera's working. I see you right there.
>> All right.
>> And uh where will you be stand?
>> I'll be standing right here.
>> Okay.
>> Okay. So now it should be the same on both monitors >> and it's not.
>> I got no signal.
>> Now wait, wait a second. I got a 50/50 chance of plugging it into the right jack.
>> Look something.
>> Okay, try it now.
>> Nothing changed.
Uh, is it plugged in?
Yeah, it is plugged in.
Huh?
Let me uh get a support.
>> All right.
And meanwhile, I did not capture that.
>> All right.
>> Calendar event.
>> Tell me about it.
I first went in connected to Oh. Oh. Oh. Oh. It's kind of going in and out.
>> Yeah, that's the splitter.
>> Okay. So, we should have trained you on this, but go out of here and directly into the capture card up here. And what that will do is this will bypass the splitter. And what that should let you do is when you Yep. Like that. Oh, there you go. See?
Perfect.
Uh, can you stand where you're going to stand for the session really fast? Is that where you're going to stand for the session? Okay, let me move this camera for you. Tilt it.
No, we don't want to get the table on me.
>> Would you like a >> me? I was uh offering a >> Oh, him. Yeah.
>> No, I I think I'll I'll stand. Thank you. Okay, good.
>> Start recording.
>> Oh, click record button on that, please.
>> Uh, it's already going.
>> Good.
>> All right. Now, um, I don't see anything on this screen or on the screen of the screen.
>> That is also an issue.
Uh, let's see here.
>> This projector was overheating this morning. I don't know if it's on again or off right now.
>> Oh, yeah. Okay. Yeah, looks like it's still open some of the windows during one of the morning talks kind of freeze cool down and stay on the talking.
So, I was sitting at the opposite corner of the room and I watched them open up the windows, but I didn't understand the reason why.
>> Yeah, at least we're getting something on the projector now.
>> Yeah. Okay. So, great. That's the same thing that I see here, >> but it's not the same thing that I see here.
>> Yes.
>> Uh uh maybe joint meeting. Oh, wait. Oh, yeah.
That's a different thing. That's Is it this? No. I think I think it may be this. Yeah. Uh, where's the mouse? Oh, yep. It's over here.
>> Okay.
>> See if that'll work.
Uh, give me a second.
>> That out.
Uh, there it is.
support again.
>> This is why we get here a bit early.
>> Yep.
Um, it's okay. I don't I don't know how to There's Yeah, I don't know how to get the string going up onto there cuz Okay, so I believe this is an OBS thing.
>> Oh, okay.
>> There we go.
>> Okay. Yeah. Yeah. I just knew that I was not allowed to touch OBS. So, >> no, that's the only part where you want to cast their protectors.
>> Okay.
>> All right. Thank you.
>> What's your name? Jeff Silverman.
>> Nice to meet you, Jeff. I'm Hunter.
>> Hi, Hunter.
>> Shirt.
>> Well, it was the shirt from uh last year.
>> Last year, so >> well, the uh talk that I gave last year is was better than the talk I'm going to give this year.
What?
>> I'm not going to say anything. I'm going to answer questions.
>> If nobody has any questions, we're just going to stare at one another for 30 minutes.
>> Are you going to hold the envelopes up like John Carson?
>> I beg you pardon.
>> Hold up the envelopes like Johnny Carson did.
>> No, I'm not. Um, are you um old enough to remember a television program called To Tell the Truth?
That's going to be more like that.
>> Okay.
>> Okay. So, I am going to do an imitation of someone with schizophrenia and I'm going to have four different personalities and you have to figure out which one is the real one.
>> That should be easy.
>> I beg you pardon?
>> That should be easy.
>> Oh, I hope so. Uh, my psychiatrist has been working on it for years and he hasn't succeeded yet.
to the west.
I am not interested.
Thank you very much.
Okay.
calendar event.
Yes. Let's go.
There's something Yeah. Yeah.
>> Tomorrow.
>> Oh, yeah. Hopefully.
>> Yeah. I need to get some tickets.
>> Yeah.
All right, it is uh 2 o'clock and um welcome to uh questions and answers. Um so if you've got questions, I've got answers. Maybe they'll be right. Uh if I don't know, I will uh admit that I don't know. But uh basically the reason why I'm doing this is because uh when I was at um Seagull last fall, some several people asked me if I was coming to Linuxfest and I said no because I couldn't think of anything to say and they both told me the same thing. Jeff, you do a really good job of answering questions. So why don't we do a question and answer session and get people who've got questions and you can give answers.
And I said, nobody's going to go to that, but here you guys are. And uh so you know come on in come on in I won't bite hard. Um so you know having said that um I have been uh using Linux since um uh 1997 and uh I uh is like a little bit more than half of my professional career and uh I am uh allegedly retired but I've uh found a new role as a uh emergency manager and one of the things that I've done is is over here in the table I've got a uh hub server. So, our emergency hub has got a whole bunch of educational material, uh, couple hundred pages of educational material on on what to do in case of a disaster, uh, earthquake, flood, fire, power outage, cold spell, um, Starbucks going out of business, all those contingencies are are covered.
And, um, that's a lot of paper. And, um, my assumption is that there's not going to be any electricity, so nothing to go run a printer with. So, this emergency hub server has got all that educational material stored inside of it, and people can just come up with their cell phone, connect up to it through the Wi-Fi, and um uh it is uh if if you go look at the um um um um SSIDs that are floating around, if you see one that says HLE, that is the Hower Lake Emergency Hub.
And this is this machine over here. At the moment, it's drawing about one and a half watts. Uh, and with the little battery that it's got, it it should be good for about 10 hours of continuous operation.
So, um, and then, uh, what I have on on the display right here right now is a, uh, list of, uh, all of the, um, uh, uh, SSIDs that are currently visible to my, uh, laptop. And you can see that, uh, HLEH is is there at the top. and and that's because it is the closest one. Um and unless other people have got questions about disasters and emergencies and things like that, that's pretty much all I'm going to say about it. So, um um how many people here are from the are from live in the city of Seattle? Can I see a show of hands here?
Okay, so uh you guys should all go to a website called seattle emergencyubs.org or and um uh learn what to do in case of a disaster. Um there are 815,000 people in Seattle and if there is a major earthquake uh unless you know where your emergency hub is, you're going to be uncomfortable. It's not the earthquake that's going to kill you. It's going to be the aftermath when all the infrastructure we depend on is missing.
Um, what are you going to do if all the bridges over the ship canal fall into the water, for example? We don't think it's going to happen, but that's what the people in San Francisco thought um uh before the Nimttz Freeway collapsed. So, um uh that's Yes.
>> What is the password for >> Oh, it is um big uh with a capital B.
Bang with a capital B. 2025 exclamation mark.
Okay.
So, uh let's see one of the terminal windows I've got. Uh I was uh keeping an eye on it. Uh yeah. So this is the access uh log. So when you get to the web server uh we should see something. So anyway, so uh having uh gone through that introduction uh I'm going to open the floor to questions. Don't everybody raise your hand at once.
Okay. One of you can raise your hands once.
>> Okay. Yes.
>> So since we have infrastructure as code >> Yes.
>> and we have AI that writes code.
>> Yes. How long until we have AI managed infrastructure?
>> I have no idea.
>> It's already happening in some markets.
I was at S3 a few weeks ago and it's the next Jaser thing.
Be terrified now. Yes.
>> Well, so you know, I I have two two views about that sort of thing and they're both informed by my favorite science fiction movies. Uh, one of them is The Forbidden Planet. And, uh, if you'll uh, recall uh, if you have not seen the movie, you should definitely go see it.
If ever someone talks to you about the monsters of the id, that's from Forbidden Planet. At one point, the robot is given a pistol and is ordered to shoot the commander of the spacecraft and he can't do it because uh, it's u he's got a rule that says he he can't kill people. And then my other favorite science fiction movie is 2001 of space odyssey where Hal um and Hal's actually kind of a tragic character uh he kills the the the crew of the ship uh Discovery with the exception of uh the astronaut uh Dave Bowman uh who then goes on the fantastic uh uh voyage. Um so those are two opposite views of artificial intelligence and uh reverence for life.
Um and that's science fiction. They don't.
Yes.
>> Can you all go down the rabbit hole a bit of the big giant concern of AI becoming the infrastructure?
>> There there is >> a job system admin.
>> Okay, that would make sense.
>> All right. So, there's a a science fiction movie on the subject if you wish. Um, it is called Colossus, the Forbidden Project. And uh essentially this uh gigantic computer uh is given uh launch capability and uh basically takes over the planet. And um it's hard to know if the movie has a happy ending or or a sad sad one. Uh I don't like the ending, but it's an example of what happens when you turn management of the infrastructure over to computers.
>> Okay. Yes. FYI, there's an older and better movie. Um, it's got Captain Heburn and Spence Tracy and it's called The Desk Set. It's >> called The What?
>> The Desk Set.
>> The Desk Set.
>> Yeah. It was um put out in the late 1950s and it's about people working for a te television broadcast research department um and management bringing in a mainframe computer to replace them >> um but without telling any of them. So all of a sudden a guy comes into their workspace and he's measuring the floor.
>> Mhm.
>> And so it's I I would say probably the oldest movie that depicts the situation where people are using their jaws because of computer.
>> All right.
>> All right. So I have a sneaky suspicion, you know, based on the questions we've had so far, we're going to spend most of our time talking about science fiction movies and not much time. And I'm perfectly happy with doing that. If you guys are happy with doing that, just understand that's not what you showed up here for. Yes. Go ahead.
>> Um, there's been a I've seen recently a lot of uh a lot of new vehicles. Uh, you can't even access the onboard computers anymore. Um, which is pretty concerning because if I can't there's already technology where you can stop the vehicle from moving and you can do that remotely.
>> So, >> like >> how what do we like how do you prevent this? Do we just not buy new technology?
>> I don't have a good answer for that.
>> Oh, you that's that's why I asked >> now. You know, uh a a car that you can't get out the guts is not necessarily a bad thing. Um there was a European automobile manufacturer that assembled a team of female mechanical engineers and asked them to design a car for women and they came up with something really clever.
You can't open the hood to get to the engine compartment. However, you can check the oil, check the um windshield wiper fluid, check the brake fluid um without opening up the the hood. So, those are the kinds of routine maintenance things. And if you can do those things without opening the hood, you don't need a hood. And so, the mechanism to get the engine compartment is is gone because you don't have to do it on a routine basis. And that actually solves a whole bunch of problems. So just because you can't get into the guts of something doesn't necessarily mean that's a bad thing.
>> Doesn't doesn't the uh exclusiveness of who can and can't get to the guts of things set up incentives that are not aligned with necessarily what's the public interest?
>> That is a good question and I don't know how to address it. All I can tell you is that for for people who just want to use their car to go drive from one place to another and not have to go worry about maintaining it or in the the guts of it.
You know, for those people, that kind of a car might be a good thing. There are people out there that want to use Apple computers and they want to use uh Google Chrome Chromebooks because they don't want to have to worry about the guts of the thing. They just want to do something. And for those people, there should be a product. You and I probably wouldn't buy something like that, but there are people out there that would.
And I I think we have to acknowledge that people like that exist and and I feel sorry for them. But of course, you know, that's legitimate. I I think just to clarify my question, I don't think the issue is whether people should feel obliged to or or feel the need that to be involved in any of the guts of the stuff. Mh.
>> But it's a question of if anybody who they choose to trust accessing it versus the I think the simplest way I found to put it happy to hear your thoughts on this is Cory Dro gave us this term and we now have can say that initification is a process that we see in the world and because of that we can't say oh yes you're happy with Apple computers so the product is great and forever and you're fine because you can say oh they they suck you in, but because there's an a power imbalance there, this is the process. This is the path of inchification.
>> A quick gloss.
>> Yeah, I I you know, I can I can certainly see your your point of view about that. Um, but I just, you know, we have to acknowledge that there are people out there that they don't care about this. They just want to use their computer to go do something and the computer does it and they're happy with it >> for now.
>> For now. Yes.
>> They're happy until they're not. They're happy until they're not.
>> Yes.
>> We have an example of John Deere.
If something goes wrong, you can't fix it. You have to use them.
>> Yes.
>> And some of the professional critics coming up with laws that they want to change that. So, anyone qualifying, >> not just the company.
>> Yeah. That's the the right to repair laws. Yeah. I think those are a marvelous idea. So, uh, most farmers want to be able to work on their own equipment and that's because that's just the reality of farm life. Um, so, um, I mean, my suggestion is that what we should do is we should all go to places like Iowa and North Dakota and South Dakota and teach people how to use Linux. And I think that would fit right in with with their mindset. They just don't know it yet.
>> Yeah. Yeah.
>> I'm from Iowa.
>> I beg your pardon?
>> I'm from Iowa.
>> Yes.
>> Oh, okay. Are you interested in going back?
>> Yes.
>> Oh, okay. So, I am uh writing a science fiction novel and my uh heroine has has got this nightmare that she's actually in a in a mental health hospital in De Moines, Iowa. And uh uh her artificial intelligence asks her, "Is there a mental health hospital in De Mo, Iowa?"
And she says, "I don't know. I've never been there, but that's my nightmare that I'm actually in a mental health hospital in De Moine, Iowa.
So, um, the artificial intelligence also asks her why there's a top level domain called TV. And she says, "Oh, there's this island in the middle of Pacific Ocean called Tuvalu." Uh, and AI says, "Oh, guess what? It's it's uh 32 degrees Celsius, relative humidity, 80, visibility 20 miles. Sounds like an idyllic place. I'm never going to be able to go there, am I?" and she says, "No, you're you're trapped in Earth orbit forever. I'm sorry."
>> Okay, next. Yes, go ahead.
>> Mine's more practical question. So, >> Oh, no.
>> I live in Black Diamond. So, and I see that, you know, there's all these hubs in Seattle. Is there anything in, you know, King County has in general or Pierce County?
>> Um, King County has an office of emergency management.
>> Yes. and they would know.
>> No.
>> Okay. I I do not know. Now, having said that, if you want to be a hub captain for Black Diamond, if Black Diamond does not have an emergency hub and you've got the time and the energy to to go do it, and it's not that big a deal. I can do it. You know, I'm retired. You're probably still a working woman, but no, you're retired. Okay, great. Then, um you know, go for it. I uh the Seattle Hubs will be happy to provide you with technical assistance. Okay? We we we will we will answer any questions you've got. We've got the you we'll do the training. Um uh we'll help you uh get materials. We'll help you write a grant.
Um uh my hub I uh um got it set up for about $2,800, but I had kind of a head start. But but for $5,000, you can get a nice steel box, which which is what you want because you want this thing to be outdoors so that if there's an earthquake and the building that it's in collapses, it'll be out outside. So, and and it's got to be chained down because people steal the the boxes. Uh, and >> you can't have anything in Seattle outside.
>> Yes, you Yes, you can, but you have to It has to be armored. It really does. So my my hub is held down with four uh stakes that will withstand 400 pounds each. So and they're in chained in such a way that they they can't rotate.
>> Yes.
>> So it seems like you you have a lot of knowledge on preparedness, emergency preparedness. Yes.
>> Um >> what in your opinion are you most prepared for?
>> What I'm most >> What do you think you're or what do you think is the most relevant to today?
>> What's our existential crisis?
You're not going to like this answer.
It's >> okay.
>> Global thermonuclear war. Uh I'm I'm I'm I'm going to go out on a limb here with with this crowd. Okay, I'm going to go out on a limb. I shouldn't do this, but we have too many idiots who have got access to too many powerful weapons. Um, if you have little boys with squirt bun guns, you know, going around squirting water at one another, that's no big deal. If you've got little boys firing thermonuclear weapons on top of ICBMs, that that is a big deal. And I think that is the world's situation uh right now. And I am terrified of it. And um I am not very happy with the response of the city, the county, the state or the federal government about it because um our planning is basically recycled materials from the 1950s.
So the idea is that if you see this big flash, uh get down on the ground and cover yourself because the shock wave is going to arrive in in a few seconds. And what they don't tell you is that if you can see the flash, you're probably already dead from the radiation from it.
Um, it's terrible advice, but >> I am I am not I wanted this to be a happy discussion, guys.
>> You just seem to have a lot of >> Yes.
Can you demonstrate how to use the emergency hub because I haven't connected to it but I don't know what to do now.
>> Oh, okay. So, uh point your browser at http um colon slh.h and uh it's got a little uh name server in it and it should tell you what the IPv4 address is and uh then from there all you have to do is just follow the links.
Yes.
>> Uh what desktop environment and distribution is that running up there?
>> This Oh, my laptop is uh running um uh yubuntu with uh the um um cinnamon um uh environment, but the the hub itself is is not it's it's running um Raspberryian uh it's running a really shrunken down uh Linux uh distribution. It doesn't have a graphical user interface at all.
>> Yes.
>> Uh the title of the talk says that you look in a crystal ball. Do you have the crystal ball with you?
>> No, I'm speaking. That's metaphorical.
>> You're talking about your computer.
>> I beg you pardon.
>> Are are you saying that your computer is a crystal ball?
>> No. Well, yes, I am.
>> I am. It's it's made it's it's made out of silicon and uh so is crystal. Okay.
Yes.
>> Uh, touching back on the like I guess for lack of a better term end of days resilient devices.
Um, has there been much effort made to have this stack like this data stack in its own format that can then be ported to say I need to run it on an Alpine emulator on my iPhone because it's all I have to be a web server. Um, has it been much effort like for the data source itself for the >> you you you can you can do it. You can run uh wget um minus minus mirror and it should copy all the material over uh like a champ. I haven't actually tried that, but I see no reason why that wouldn't work. It's it's it's a stock EngineX uh web server and um the the um site that I uh p perloined it off of is running WordPress. Uh, I didn't do that.
I just you uh swiped the content by uh downloading it uh through by mirroring >> the URL to get to the >> because I tried and couldn't get it. Um, >> so if you punch in um 10.122.0.1, does that work?
Did that work?
>> All right, next question.
Can you go into a little bit about your hub? Like what you're running on it, like how you how you how does it work?
>> It it is a uh straight uh engine X. Um I um um stripped out uh a whole bunch of unnecessary modules just to save space.
And it turns out that I uh saved uh a couple of megabytes on a um 250 uh gigabyte um uh drive. So, >> didn't didn't really need to do that, but but basically it it is a a stock Linux machine. Uh it's um running on a Raspberry Pi uh model zero and uh it's currently draws in including the the Wi-Fi transceiver about one and a half watts.
So, it's really it's designed to be low powered and the whole thing cost less than $100. So, so it was low cost, low power, uh, high durability.
Um, yeah, I'm nowhere near the the top 500 supercomputers with this one. Next question.
>> Yes.
>> Can you describe the battery that you have in >> the battery?
>> Yeah.
>> Standard USB battery.
>> Uh, this particular one is 10,400 milliamp hours.
>> Okay. Um, and it's got a type A socket, uh, a, um, mini and a USBC connector on it. And, uh, my experience is that this battery will fully charge, and keep this thing running for about eight hours.
>> I was imagining something huge when you said battery. So, I don't know why, but no, it is it's Yeah, it's tiny. So, the the idea was was to be lowowered.
Yes.
>> Uh what are your thoughts on uh Nyx Nyxos and like declarative uh system distributions? Is it something that you would use for like a high reliability setting like emergency preparedness? Well, um, high high reliability for emergency preparedness means something different than what most people think about when they talk about high reliability. Um, for for most people, uh, they're thinking in terms of the operating system will crash and they want something that will keep going or the application will crash, they want something will keep going or the the the rack will go down or the data center will go down. And so those are the kinds of things that they're thinking about when they're talking about high high reliability. So in this particular case, I'm thinking about something that is is very very different. It has minimalist infrastructure and extremely minimal cost. So one of the things that I am very very concerned about is what is called the digital divide. So, I'm really looking for something that is lowcost, very simple, very turnkey. I I want to be able to give this thing to people who are not wealthy, not well trained, and they want just want something that just works.
So, I'm I'm doing something different than than what you're thinking about.
Okay. Now, if you're interested in high reliability solutions, okay, I used to work for F5 Networks, and F5 Networks made a gizmo called a uh uh local traffic manager, and that was an absolutely staggeringly powerful uh parallel processing system. So you could have a bunch of physical servers and the LTM would not only do reliability. it would, you know, test to see if the server was working, but it would also, if you had a bunch of servers, it would divvy out the work to the different servers so that uh you could have a server, you could have a website that would be handling a million transactions per second if you wanted to, using off-the-shelf servers, PCs, none of which could go anywhere near that fast. because you're spreading the work out in parallel. Um um it gives you the ability to have this really fast website. And when you have applications that are split up like that, you start running into problems about how do you manage state. The local traffic manager has got several different solutions for how to handle state. either it can share the state uh in the LTM or the LTM can make a connection and and sticky so that all the transactions once it goes to a physical server it stays going to that physical server um very powerful very very expensive machine um and um to my astonishment F5 has not gone out of business because Amazon webs all the cloud providers now have got something similar it's it's baked into their infrastructure but F5 is selling these things and and they're selling like hot cakes and F5 is making a lot of money.
>> Yes.
>> Where if if we were magically five years from now describe artificial intelligence words >> God who knows. So, um, >> you're crystal ball.
>> Well, all crystal ball is giving me is an IPv4 address. Um, but uh um there are there are a couple of possibilities.
we may run into a limit as to what artificial intelligence can do. I don't know what that limit might be. I don't think anyone knows what that limit might be, but we might bump into a limit about how powerful you can make an artificial intelligence. Um, we may decide that we're just not going to use artificial intelligence. Uh just like we made the decision we're not going to use DDT and we're not going to use Freon. Uh we just came to the conclusion that these are are technologies that are too dangerous to be used. Um we may decide that artificial intelligence can do a better job of running the planet than we can and we will turn over management chores to the artificial intelligence and it will be programmed to think about the the greater good of humanity. we may enter a utopian age where people don't have to work. Um that we can sit around and and be philosophers and poets and science fiction writers and gardeners, um long-distance runners, whatever it is we want, and all of our needs will be taken care of by machines. And we'll have machines to go repair the machines.
And and if someone wants to go build a better machine, they'll have the free time to go do it. But if someone just wants to sit around and work on their suntan, they they can. Um, or there could be a dystopian future where the machines uh um are taken over by a bunch of able of uh evil people and uh it they will be a uh instrument for enslaving humanity.
Um so but that's really the realm of highly speculative science fiction. Not not really. I'm a science fiction fan, but but I and so I can see utopian futures and I can see dystopian futures.
>> Yes, Rowan.
>> Rowan, you have to speak a little louder, please. I'm sorry.
>> According to that clock, the uh time is up.
>> That's not a question.
>> We still got another two minutes.
>> You phrase that as a question.
What time do we have?
>> Okay, so Rowan, as a matter of fact, we according to this clock, which is synced using NTP, we still have a minute and 25 seconds.
>> Minute 20,000.
>> Yeah.
>> For an alternative uh view of possible futures and science, um we saw last night the Project Hail Mary.
>> Everybody should see that. Yeah, >> it's wonderful. It it really has accorded with every talk that I've heard so far today >> um as far as people getting together and helping one another and hopeful outcomes. I mean there is more than the insane clown posy that seems to be running the world right now. But >> right >> um okay >> we don't have to allow that. Well, so look, um, you know, th this this meeting is is being transcribed by, uh, artificial intelligence, and I would like to, uh, end this meeting by enumerating all the science fiction movies that we talked about in the past half an hour. So, they were uh, Forbidden Planet, 2001, a space odyssey, Colossus, The Forbidden Project, Hail Mary, um, what are some other movies that I we mentioned?
>> I beg your pardon.
>> Desk set. desk set with some other science fiction movies that people think is appropriate that we should put in our summary of this meeting.
>> 2001 of the space.
>> 2001 and the space odyssey.
>> What?
>> Dr. Strange Love.
>> Oh, Dr. Strange Love. Yes, absolutely.
>> Yes.
>> What are your thoughts on the Geneva conventions?
>> A tiny nice we brought them back.
Well, so um I'm I'm not sure the Geneva Convention applies. Donald Trump isn't a P yet, is he?
>> Where?
>> Actually, actually um the um um head of state from Venezuela that we kidnapped, uh that's actually a pretty flagrant violation of uh the Geneva Convention. the the United States is basically guilty of of uh kidnapping. If um um they want are wanted for trial here in the United States, they should have been extradited uh through Interpol. And um that's you you just can't you can't do what we did. That's >> we did it.
>> I know we did it. I'm not happy.
>> You can't legally >> You can't legally do that. Yes.
>> To end on a happy note, >> please. Uh, if you could bring one technology back from the technology graveyard, what would it be?
>> Oh, downstairs in the in the uh G building, they've got a Strager switch.
Uh, and um uh definitely go see it. This this thing is almost but not quite as old as I am. Uh, and uh it's got a um he's got four Strager switches uh arranged together. And um so the stronger switches is how telephones worked back in the days when there were rotary telephones. And so you've got one part that goes around like this 10 positions and then it goes up and down like this 10 positions. So each stragger switch can connect a 100 lines together.
And he's got four stragger switches connected together. And uh one of the reasons why they went out of Vogue is well for one thing we've got transistors now which can do the same job faster and more reliably. And also he's got problems with this his equipment and um he's he's about the same age as my grandson and I I kind of >> feel but uh that that is definitely a technology that I would I would love to bring back. Um let's see what else is has come and gone.
>> Beg your pardon?
>> Eight tracks.
>> Eight track cassettes. Well, >> medium.
>> Okay.
>> So, um, in the late 1940s and early 1950s, uh, the electrootive division of General Motors built these really beautiful diesel electric locomotives, and I would love to see them, uh, make a a return. Of course, they were dirty as hell, uh, but they were in just incredibly reliable and was a major renaissance of the railroad industry in the late 1940s and early 1950s.
Um, I would love to see those come back.
They're beautiful machines. Uh, I'd love to see passenger rail come back.
>> Yeah. You got a question?
>> A quick, how well do hot cakes sell?
>> Beg your pardon?
>> How well do hot cakes sell?
>> Well, I had three for breakfast this morning >> for $10.
>> Three for $10.
>> Three for 10. Well, you know, within with inflation.
Yeah. Ran, do you have a question?
>> Um, >> just a question.
>> Remember the time I It's already been two minutes since you said I was >> Did you remember >> Aaron, get get Rowan an analog watch.
>> He had he >> he loves his analog clock that he's got.
>> Okay. So, Rowan, do you know if you were lost in the woods in the daytime and you had an analog watch, how would you use your analog watch to tell you which way is self?
>> You may assume standard time.
>> I've definitely learned that sort of thing.
>> Think it through.
>> Rowan, think it through for a second.
Okay. Here is a clock.
>> Well, if if I could go an extremely long way, then I could tell.
>> Well, okay. So, if it is 10:00, our hands pointing this way. Where's the sun going to be?
>> Oh, >> 10:00 in the morning.
>> That would be at 10:00 in the morning.
Yeah, the sun would be about there.
Is it >> Which direction?
>> Which direction? North, south, east, west.
>> Uh, it would be in >> the east.
>> Yes.
>> Can you do better than that?
>> It would be um less than uh it would be toward the east 90 less than 90°. Uh >> less than 45°.
>> Rowan, you are lost in the forest. You don't have any food or water. Your life depends on your ability to navigate using only the sun and your watch. What do you do?
>> Teachable moment.
>> Hey, Rowan. The sun always bisects the angle between the hourhand and noon or the hourhand and 1 daylight time. So if the sun is here at 10 o'clock in the morning, the sun is going to be here halfway between 10 and 12. So if you orient your analog watch so that the sun is at 11:00, 12:00 will be pointing due south or 1:00 if you're daylight time. All right. So, um, Aaron, if he gets lost in the forest and can't be found, okay, I'm going to be intensely annoyed.
Didn't have his landlord watch.
>> Rowan, I don't mean to pick on you. I'm sorry.
>> Well, he was picking on you.
>> Well, yeah, but I'm bigger than he is.
>> Yes.
>> I just have a technology that I'd love to see brought back.
>> Oh, please.
>> Nobody uh knows anything about uh but as soon as they do they like it too. Uh it's a service called Teleaharmonium service >> called what?
>> Cella hararmonium.
>> Kella harmonium.
>> Like it means you get to have a harmonium at a distance.
What it is is um before uh amplification was invented um there were services in uh large cities uh that were able to provide music um to stores and restaurants so that they didn't have to employ musicians anymore.
>> Teleaharmonium >> like that idea.
>> Yes.
>> Uhhuh. So, it's like elevator music without uh >> It's elevator music and it's the explanation as to why elevator music historically didn't include voices because the ability to transmit music predated the ability to amplify voices.
>> And um essentially what they had was keyboards connected to giant generators >> that had a wheel on them for each note in a scale.
And then because they also didn't have recording yet, um operators would sit at these keyboards and play songs. Um and the way they're able to get it out into the city was by using really high voltages instead of amplification. And so then in your store you would have a essentially a speaker uh no no amplifier, no other electronics, just a speaker and a knob that allows you to select from one of six channels. Um, and then you would hear essentially it sounds like organ music.
>> Um, and so if you ever heard of like a Hammond organ, a vintage Hammond or it was essentially that except much much higher voltages so the service could extend throughout the city.
>> Yeah.
>> And and no voice because they hadn't invented a way of doing that yet.
>> Yeah.
>> Uh so it even it predates uh telephone service.
>> Huh.
>> And um and so they there was no recording. So 24 hours a day they would have operators sitting there playing live and you hear it in your store or restaurant or wherever you want the service from.
>> Well that's that's very funny. People uh sometimes ask me um if I can give an example of a machine to to do a uh power spectrum estimate. And the example that I come up with is you take a piano which has 88 keys and you tie a little magnet to each string and you put a little electromagnet next to each magnet and then have some music. The strings on the piano will actually resonate vibrate at their resonant frequencies and the voltages that are generated by the little coils next to little magnets is the power spectrum estimate for that instant.
That is a Rube Goldberg solution using physics and not mechanical engineering, but it's still a real world example of how to do a power spectrum estimate.
>> Guitar players actually use that um that characteristic of strings. If you ever see in a um you know like a stadium type performance, the guitar players will turn their back on the audience and stand close to the speaker. um they're allowing the speaker echoing back their strings. Um and if they keep the um the guitar um ringing that same note, vibrating that same note, and then they can change the pitch with their hand without having to strum.
>> There's a there's a add-on you can add to guitars that actually does the feedback internally between the pickup on the guitar and then feeds it back into the string that you can get a couple of these called a sustainer. Uh >> you're a music teacher. Yeah.
Aren't memory ser? Isn't that right?
>> Yes.
>> Humans are so creative.
>> I think that that's I think that that there's going to be a future without artificial intelligence for people coming up with creative solutions like that. I I don't think an LLM would would think about that. In the back, you've got a question.
>> How far over time are we on this?
>> Uh >> don't ask me, ask Rowan. officially >> about 15 minutes.
>> Yeah, >> we have less than uh yeah, we have about less than like uh we have about four minutes till the hard cut off. But yeah, >> the hard cut off when the talk is over, but people are talking. So this is the so this is the Q&A session. Now we can ask now we time to ask the quest ask any questions about talk.
>> Yeah. So, I was hoping that someone would ask me like, you know, something about operating system internals or what's virtual memory or, you know, how what does it mean to go service an interrupt, but obviously you guys aren't interested in that sort of thing.
>> Oh, yes.
>> Why aren't we interested in that sort of thing?
>> Because it's not important. Um, >> it's a solved problem.
>> VIM or EMAC?
>> Yes. Oh, yeah. No.
>> Yes. Rob, >> are there any initiatives going on right now to continue to ideas and bring them into Linux because there's so many great things Multics can do that you still can't do with any other operating system.
>> I'm I don't know enough about Multix. Uh what's an example of something Multix can do that Linux can't do.
>> Uh well, one of the cool things they had was builtin database.
>> What's a what?
>> It had a built-in database.
>> Yeah. Um, and what that meant is that every time you bought a product, regardless of whether it was compatible with the rest of your software, you knew that at least it was writing to the same database format.
>> Um, and so that gave them the ability to do things like you could set up security rules on a zone of a file that was accessible to a person with a with a given security class. So you could say if you know the people with this class can't see these fields or they can't access these records which I think you can do in most um SQL databases today.
>> MySQL will do that. Does that does that really belong in the operating system?
>> Well the great thing about it is that it was something that was universal that you could rely on. Um when you wrote a program you knew that it was going to be able to do those sort of things.
>> Yeah.
you looked into the Erling system because Erlang has that uh they have a database called Amnesia >> and uh it's essentially what you're describing.
>> Yeah. I mean co Cobalt has that but who wants to go write software in Cobalt?
>> Well, more people want to write code in Irving than Cobalt obviously.
>> Yeah. Um but I mean so um this comes down to a to you know I I would rather debate this question than vi versus emacs is what how much functionality belongs in the operating system and how much uh functionality belongs in um uh applications which are u uh might be optional. So for example I would not want a database in my hub server because my hub server doesn't need it. I I want a really small operating system. I don't I don't want a database in the operating system. I can think of other pl cases where you might want to put a database in the operating system that would save you doing a bunch of context swaps and uh a context swap is expensive. So I can see a use case for putting a a database in the operating system. I'm just not sure if that's a good solution or not.
But then if you look at operating systems, virtually all IBM mid-frame and mainframe operating systems have built in database capabilities. And every operating system that IBM ever produced in the mainframe era, which is many, many operating systems, all of them added together take up less business space than one copy of Windows.
>> That's an interesting indictment of Windows, don't you think? operating systems, but all those databases, it's not really going to be a problem still.
>> So, I mean, you know, digital digital equip Whoops, I didn't want to do that.
Digital Equipment Corporation had um um record management services inside of uh VMS and that would do things like record locking and you could do protection at the record level. Um VMS was a failure in the marketplace after the early 1990s. uh they just uh couldn't um uh it just didn't work anymore. Um so um uh RMS is is gone but but but not forgotten. Um but I'm still not sure that's the right thing for for an for an operating system. But I I you know it could be done. There's no reason why it can't be done. It's just nobody's done it. Um, if you really feel strongly about it, go put in a feature request >> to to kernel.org, you know, or or, you know, write a kernel mode database management system.
>> It also had a lot more hot swap capabilities than we have >> right now. You could hot swap CPUs and instruct the operating system not to use a certain CPU. So, you could really take it out.
>> Yeah. any harm. You can hot spot hot swap discs these days, but not as far as annual CPUs.
>> Mhm. Well, >> so this room for telecom.
>> Well, thank you for coming everybody.
Hunter, thank you for putting up with me.
>> You're welcome. I mean, you're it was it was very interesting.
>> Thanks, John.
>> Thank you.
Oh yeah, somebody must have not.
Jeff, when did you work at five?
>> What?
>> When did you work at five?
>> I think for a second. I was there for four and a half years.
>> Um, >> I uh senior moment. I I uh I was there for five and a half years. for for as a customer service representative here as a performance about >> um and um I'm trying to remember when they laid me off my resume >> worked who was your nephew >> your captain.
>> So is this your running?
>> This is this is the hub. This is a USB at the moment.
>> Okay.
>> Basically holding up this antenna.
>> I think this is their laptop still >> but then Yes.
>> Uh yeah, I I should do that.
>> It is a Raspberry Pi.
It's a uh I believe it's a model zero.
Okay. Now, I think I've got all my cables.
>> It looks that way to me.
>> Okay.
Nice.
But, um, anyway, this is >> it was designed to be um cheap, simple, and low power.
>> Looks like they were not using the >> and and so, >> you know, everybody says, "Why? Why did you do this? And my response is always >> that's it.
seen this before.
>> I've not seen this before.
>> Interesting.
>> Okay.
>> My name is Andy.
>> Hi.
>> Or for I think I read that online.
>> Cool.
>> Thought I can plug this tower thing in.
>> Yes.
>> It's like on the side there.
filter thing, but that's perfect.
>> I guess that >> works.
>> All right. Uh, there's HDMI cable to plug in.
>> And then we plug it in here. If you need an HDMI, we have one here.
>> Don't I didn't bring a cable with me.
>> No worries. And is that a Mac?
>> It's a MacBook running next to us right now.
>> Oh, yeah.
Well, we should also have a HDMI to something adapter.
>> Well, let's see if this uh shows up on the screen.
>> All right. It's saying that there should be duplicated.
>> Okay. Well, there is >> could bypass the >> I would never try to just relaunch OBS.
>> Yeah, that might be what we So I have your screen on this end now.
>> Yeah. But now it is >> using the wrong monitor, >> which might be the quickest to just unplug it.
Looks like mouse turn on there something >> the same as this.
Okay, let me message >> for the question down to non.
I was doing that for a while.
Thank you.
>> So that monitor is the same as the I should remember that.
>> Yeah.
>> Okay. So, the 30-minut talk, we have 15 minutes afterwards for questions and we should be wrapping up around 5 minutes before then and I will give you a hands up.
>> Oh, sweet.
>> Yeah. Yeah. I'll like give you a hands up when the five marker approaches and I'll introduce you before the talk which we have about five minutes for >> cool question.
We still have five minutes before the talk, so feel free to keep on talking if you'd like.
transcribing.
>> Yeah, it's very sensitive.
Hello. We're going to get started. This is Kaso Aster hope and they'll be talking about nine months of cultivating an online community environment.
Thank you.
>> Hello. Um, you know what? I'm going to take my mask off because it's probably going to be hard for you all to hear me with this on.
Hello. Um, this talk was pitched kind of um, idealistically in that I hoped that I'd have a lot to talk about by the time it came around to writing it. Uh, less has happened than I anticipated, which is fine, but it means that there'll be a good bit of time for questions and answers. So, yeah. Um, I'm gonna get it started. So, first, what is the Octo friends garden? It's little internet community thing that I I've created and administer uh the the infrastructure.
It's a physical machine in my apartment just desktop PC hardware. And then there's about 30 minute 30 members um pretty much all of them uh joined by direct invitation um through like oftentimes just meeting up in physical reality like like this like hello would you are you interested in this? Um so yeah uh so what the garden offers for its members we've got like email calendars contact books through the mailcow uh uh container services we've got file hosting using copy party um there's a pure tube instance for for uh video hosting uh matrix vav synapse deployment and three web client deployments of fluffy chat element insi give people choice why not uh a mastadon deployment for micro blogging um there's a there's a wiki software on there book stack we have forjo for git hosting uh RSS like reader that useful and events calendar and then there's single sign on to tie everything together so it's one account for members and they have access to all the things um which is nice Um, and why I started it, I I was already home labing for like over a year when I started it, but it was just things for myself and I was like, you know, some of this might be useful for friends. So, I was like, why not? As to the naming of it, the Octo friends garden came kind of from I I have to credit my roommate for coming up with the idea. Uh, it was kind of a cross between the name of this plushy that I've given them, Octo friend, with the song title Octopus's Garden by the Beatles. The Octo Friends Garden.
Um, so yeah, uh, started about nine months ago. It's it's been slow and erratic growth because I've been basically doing direct inviting. like members are free to invite other people, but that hasn't really happened yet.
um and software that I that I deployed first was the authentication system of course because that's the key piece and then the micro blogging like the mastadon and the matrix first and those were all things I knew how to deploy had done it before for myself single user instances and was like all right I can bring these to uh yeah um and so um and one thing that I did early on that I think matters a good bit was that I established the rules and values of the garden like early on basically before I even started inviting people. Um the the rules are going to be are pretty straightforward I think. Um the values I have on this slide. Um I decided to include the full uh the full text of them that I that I wrote originally. So I'm going to read them out. Uh people shouldn't have to defend their right to exist. Riskaware consent should be considered in all interaction.
Accommodating people's needs is worthwhile as an ends to itself. The things we say, not just what we do, has consequences.
We deserve a better world and we should make efforts to bring about the world we dream of.
Identity outside of the norm shouldn't be something one has to hide and should be something one can choose to celebrate instead. Omnidirectional kindness should be the default. So these are the values of the garden. Um there's yeah um and that is something that members are expected to um kind of just generally be chill with uh it's kind of part of the agreements of joining the garden is that you say that yeah these are these are cool with me um kind of thing. So, uh, and how it grew, I just put up more services when I felt like it and kept inviting people to it slowly and through like just meeting people and such. And um, some of it was through like interactions in physical reality spaces like the Dev Hack hacker space in Seattle and some some people I uh invited through the Fedverse uh like Masttodon and such and that was cool too. Um, as to some things that I realize I wish I had done differently was it's not as reliable as I hoped for.
Um, and a lot of the unreliability has been resolved now, but um, I think some of the members have been a have perhaps gotten frustrated by its early on unreliability and just haven't really been using it as a result. Um, so that's a that's a little bit of a pain point that Yeah. Um, also, uh, documentation is a thing I wish I had more of because like I can invite people to it, but if they don't necessarily know the software, they'll be a little overwhelmed between the multiple dis between all the different software that includes and just what even those softwares individually do. Um, and then I kind of want to I wish that I had a way to kind of showcase what the community was like in kind of in a more interactive way, like a kind of curated example of like, you know, things people discuss in the community and stuff and and I don't really have that set figured out yet, but it's not too late for me to figure these things out. Um, I am at the end of my slides. This was hastily prepared.
But uh you all are very welcome to ask qu any questions you feel are on topic.
Um uh we already have a hand up. Uh what is your >> this is more of a comment than a question but I really like your talk.
>> All right we'll do uh one the closer to me two further back three and then four.
If we can try and remember that order.
We'll see. Um >> I can remember I can remember you talked about having an RSS meter earlier. What RSS?
>> Oh yeah. Uh it uses I have fresh RSS deployed for that. Yeah.
>> So could you repeat the question to the microphone?
>> Oh yes. The the yeah the question was what RSS reader is uh is deployed on the garden and so my it was uh fresh RSS is what I use for it. Um then >> what would you say is currently what you would expect to be your capacity of users right now?
>> All right. Um so I was asked about capacity like estimated capacity of users. I don't have a hard number on that that I can easily estimate. Um it's for the most part uh like we've had a little bit of hardware limitations.
um specifically that the storage uh is a little the storage used is a little slow and there's room for me to improve on that without getting new hardware. Um there's some ZFS optimizations that I should be doing to it that I haven't yet applied um because of the uh like I'd have to plan around that for the downtime involved. But uh that is on the road map of something I'm going to handle. Um all right, let's see. Uh I'm trying to remember we had over back here I think.
Um, >> uh, I have this question like what are some of the I guess like >> um so far we haven't had any sort of uh re real like um interpersonal incidents involving me uh members of the garden.
So, no, there hasn't been any needing to remove uh access for any uh anyone or anything like that. Um, we do have rules. No one's violated them thus far.
So, >> the first All right. Uh, back here.
>> Uh, I'm curious how you have handled uh moderation. You you mentioned that that people on the server haven't broken the rules, but I'm curious how you handle moderation for people off server. I know there's deferration as an option.
>> Yeah. Um so for the master and matrix uh we do yeah we do um like deferation as needed and so um and so that's something that's just largely handled on like case by case kind of basis of starting with a known we started with like a known reliable source for uh for a starting uh defederation list and then worked from and then as things came up uh blocked on blocked from there and stuff. We'd usually evaluate to see all right are the admins actually doing anything about the problem before we decide whether to defer or not or just suspend the account remotely.
Um uh I don't know uh why not >> uh so one of my friends was trying to uh block certain servers from being fed with uh their server but they couldn't figure that out for matrix. How do you do that? Oh, matri a question about matrix federation. Um, so >> defederate from other matrix.
>> Oh, defederate. Yeah, it's there's not really a direct deferation process. It's more about like moderation tooling that you have to set up that that can then ban users from a server as soon as they join a room and things like that. It's not as preventative essentially of tooling as I would like. So every single time someone joins a room on your server, you check their u >> automated thing check automated bot would check if they're they belong within a um a block server list and they get banned.
>> Yes, that would be >> bit cluy but I can see how that works.
Yeah, decently left.
>> Okay.
>> Uh over here >> you explain the onboarding process. Do they get a link and what happens?
>> Okay. So onboarding uh first it's typically like there's a web form that uh people who want to join are supposed to fill out. Um that form in asks for the things needed to create the account as well as a few screener questions to make sure it's a person filling it out and that you know that they understand kind of the values rules and so and so and are willing to go along with that.
Um, and then if they, you know, fill that out and their application's approved, what happens is then I create the account for them with a unset password. No, no password set. And then send I send them an email containing instructions of like basically go to the authent authentication system on this link. Uh, there should they'll use the reset password flow that sends you an email to set your initial password. Um that's it's a pretty manual process which I'm fine with right now. Um but I do wish I have more doc documentation to help with that.
>> And then does they just enter into like a server with all the rooms in it services?
>> So when yeah uh like once once they've signed in there's a there's a page that has all the different services listed individually that they can then click into. Um something interesting about how it's set up is uh the until a user until a member signs into a particular service for the first time on the garden uh that that service doesn't know of their account until that happens until they sign into it for the first time from the authentication system.
Uh so um you've mentioned that octo octo friends garden uh has mostly spread in physical spaces IRL and that uh it overlaps a lot with physical spaces because of that. Um, but have uh have you ever considered or have you ever done um like an in-person meetup for users of the garden or like any other in-person component of it? And like would that like are are most of its users in the Seattle area? Like what would have you considered what that would look like?
>> I would say a meaningful portion of the user base is in the uh Seattle Puget Sound area. Um, I have not planned any sort of physical reality gathering of members at any point yet. Perhaps I will for the first anniversary or perhaps I won't. Um, uh, yeah, that's Yeah.
>> What kind of feedback have you received from the community?
>> Um, I generally received much feedback in general from members um that like I can you know, they use it or they don't. I don't necessarily hear too much about their experience using it really. Um, I do want to I don't really have formalized methods of for them to send feedback, but I do make it I do try and make it known to members that I'm just they can chat with me. They can just, you know, message me about things. It's fine. Um, yeah.
What are some non-technical uh things that you've learned about growing community?
>> Uh some non-technical things I've learned about go growing community. I think kind of um it's important to have a idea in mind of how you want to grow the community. Um like the methods, but also how like what how do you want it to take shape? How do you want the people inside to be interacting in things? like I I established the values before I started inviting members as a way of shaping kind of how it will grow in a in a way um and how that how the culture of it before it really even meaningfully started. Um that's that's what I have to say on that. I think that comes to mind.
Uh yeah.
>> Uh you mentioned that like at the start you had some issues with reliability and that some people didn't stick around because of that. curious like what what have you done to make the the services more reliable?
>> Uh well, I investigated why it was being unreliable pretty much. Um there were some specific issues. Uh the like I ended up having to switch backup software for the garden because the initial backup software I was using got into got the repositories into a corrupted state which is terrible and also that was causing crashes when it would try to do the backups which was painful. Um so I ended up switching to a different backup software entirely. Uh and that's been working much better. Um uh the the software I use now is uh Borgmatic which is a wrapper around the Borg backup set uh software that also has nice features like being able to back up the database um easily and things.
>> Do you have like any off-site backups?
>> Uh we do have off-site backups. Yes, we have um a backup in the same in the same place like on on an external drive and then we have a cloud backup. Um yeah.
>> Cool.
>> Not quite 321 but yeah.
>> So it sounds like at this time you're kind of the sole person running the show. Supposing this kind of takes off.
Um, how are you looking at maybe bringing in because you probably want to bring in like a team. How are you uh have you given any thought to that?
>> All right. So, I was asked about how like right now it's just me doing the administration and such and like if I had thought about like you know bringing on a team how how I would do that. Uh that's that's something that I've been thinking about a bit here and there. uh haven't really had too much time to sit down and uh and plan it out.
Um there is some technical work that needs to needs to be done for isolating some of the garden services from my personal services that are not part of the garden. Um it was not structured in a way that keeps them too separate on the physical machine. Um so there is some considerations I have to think about with that. And then there's the interpersonal stuff of like how uh figuring out who do I trust enough that also has the appropriate skills to on board for this. And so I would be looking uh presumably probably within gu existing garden membership or you know or down the line existing at that point even not you know people who may have not even signed up to the garden yet but eventually will I don't know but yeah that's One thing I've noticed about communities that stick together is they have a shared purpose um in approaching people to join your community.
How do you pitch them or or is there a purpose behind it other than just the values?
>> Um so this I was asked about like is there is there a purpose included as like when I when I pitch the garden to people? Um, I haven't really been approaching it that way all that much.
Um, I like there's I kind of I've I don't necessarily like how I've been pitching it all that well, but I I I've yet to think of a stronger approach to it. Um it's it's been that I kind of like have are like I have these cool services and there's a bit of community around it and um it's it's not all that uh I I'd say it's not as compelling as I would like it to be and part of that is yeah I don't have a um thing beyond the values thought out that uh that is meant to be super interesting to the right people Is it safe, secure, and not going to track you?
>> We do our best on all those things. It's the the question is, is it safe, secure, or not going to track you? Um, and yeah, my answer is we do our best. Um, in terms of the not going to track you, that that part I can most reliably say it won't track you. um anything that anything that it collects is is things that would be given to it by your by a member's using of the services and and like things that would be effectively unavoidable for the features that the user might be using. Um and there there's nothing that goes to any third parties. the real the like the backups are encrypted and so you know there's no third parties receiving anything in any any way that they can decrypt. Um yeah uh the safe and secure it's pretty good on that I think. Um like there's no public SSH access to the machine.
There's uh um I mean and various other security precautions taken. Um like all of the services are only accessible through the reverse proxy. One can't just you know find a port hunt to access something more directly. Um, yeah, there's various considerations, but >> do you think there's an ideal size for a community group like this?
>> All right. Uh, ideal size for a community like this. Um, >> not too much, not too >> uh I'm not I'm not sure what the kind of Goldilock size would be for this. I mean that would depend on whether I'm like if I just continue running it myself or if I bring other people on board. Um because it's like right now I think the re uh it it's not necessarily constrained too much by the hardware.
It's more kind of what I have bandwidth to keep up with in terms of uh like keeping an eye on things. Um but if I say brought more people on board in in helpful roles and in what I call the gardening roles just at theming um uh then then it might then it can grow to a point where perhaps the hardware then becomes a limiting factor and I don't necessarily know what the uh what a good size for the community would be that I'd want to shoot for. Um yeah right uh any more more questions? Um >> I'll just share I launched a community this uh at the beginning of the year. 50 people joined up and then about eight of them are like active. So like that that just gave me an understanding of like you need a lot of people in there to just have a little bit of action happen.
Yeah, there is a significant ratio for a lot of things of uh of how many people actually sign up versus how many people are active. I don't h I don't keep tabs on the activity numbers too well. Um it's I I don't do that because I I don't know.
I just feel like that might stress me out actually looking at that too much.
Yeah. Um any further questions? Oh >> um I know you're using like open source prepackaged software as far as you like told. Um would there be any like uh interest to like try and integrate the other um more seamlessly?
Uh so talking about like integrating the services more seamlessly that's what you're >> so like on here you have like a link to send to the master instance or whatever.
>> Um yeah I I don't necessarily know how I'd imagine doing that but that is something I would want to do make it even more seamless between the services.
Yeah that would that would be a cool thing. Um, yeah. Uh, was there Yeah.
>> So, you sort of talked about the the possibility of growing it beyond you or what what that might look like, but I guess my question is would that be like do you is it is it does this feel like a a personal project that you would you would you want to grow it? um like I I would want to grow it beyond me. Um at this it does feel like a personal project, but it's one that I do do want it to succeed even if even if I I need to step away, I still want it to succeed kind of thing. So, it's it's both personal and I and that I I care about it in a way that I want it to last longer than the risk of my interest waning. I don't know uh how to best word it, but yeah.
Um I do appreciate that question and all the questions y'all have thrown at me.
Uh any more? Uh yeah.
>> Uh trying to figure out how to word this, but like um what is like the vibe inside of the garden? Because there's like with with fediverse things like you can kind of interact with them as though like I am joining this server so that I can communicate with the people in that server but like not be completely siloed.
>> Yeah.
>> Or you can be like this server is just a means to an end to basically use Twitter without being on Twitter.
>> So uh >> and so like is it >> are people using the services to build community or they just kind of using it as a platform for fiverse stuff? Um, so being asked about like the the mastadon in particular, are people using it more to build community internally or or just as broader fetverse usage? Um, I would say it's more the latter. We do uh like it federates and so um people do use it for that more more so than um like the particular maston fork we're using glitch social does have functionality for local only posts but that tends to be not commonly not used all that much in proportion because yeah the and and our I would say that our federated timeline does not scroll super fast like a lot of instance like larger instances would. So, it's still kind of community vibes even from that somewhat.
Yeah. Um >> maybe maybe related not how do you like communicate with your announcement inform?
>> All right. So, I was asked how I communicate with this with the with the members and like how do I communicate announcements and things like that. Um, it's kind of uh a little disorganized right now as a lot of things with the garden are. Um, I usually do by email and uh for members that h that have signed into uh like the mastadon or the matrix through like local posts on the mastadon and uh and nonfederating chats on the matrix and things like that. Um, I do I do have plans to set up like a uh like a newsletter software for both uh like both the internal focused newsletter and then also one for like people who are interested in the garden but don't necessarily want to be members. Um yeah, any more questions?
I know I put y'all on the spot by having to y'all generate questions for such a long amount of time, but yeah, >> if anyone here wants to sign up.
>> Yeah. Um, if if folks are if people are interested in signing up, um, the domain is literally octofriend.garden.
Um, and I have cards with like my business cards have QR codes that link to it as well as the URL written on it.
Or if you just want to get in touch with me and not necessarily join it, that's also an option, too. I can give you a card if you want that for Yeah. Um, yeah, that's >> awesome.
>> All right, I guess. No. Yeah, you're good.
>> All right.
two hours.
>> Yeah, it's This is the bag. And they're not entirely wrong, but there's a small hole for the bottom of it. And we've got Yeah.
American.
Hallelujah.
I But also Well, I'm not the only The main issue is actually right I would say And I don't Everybody.
Who is it?
Linux terminal.
>> This is where I wanted to go anyway.
>> Is my jacket in here?
>> Your jacket?
>> Yep. Bet. That's mine.
for gratuitous that you were back.
>> Mhm.
>> Hello.
>> Hello.
>> Dude, these are so awesome.
>> Hello.
>> So awesome.
>> By the way, that's laptops.
AI.
>> This was lost and found.
>> It is lost.
>> I think we're offline.
>> Oh, we are. Okay.
>> I can do.
>> Yeah. Yeah, I'm wondering if this is a generated by AI or if it's like some kind of voice recognition.
>> I I I think the ladder.
>> I'm going to try this in a different way.
We've got >> mics are talking with him so that they don't they can feed it through the uh the overhead.
>> They're not uh >> Is this yours? It is not. No, that's mine. Thank you.
>> Excellent.
>> Good.
actually.
>> Okay, let me run over to the other one.
>> Sounds good.
It's always Holy [ __ ] >> Okay.
>> What's up, >> Cameron? Can we hear Can you hear us?
>> Yep.
>> Can you hear me?
>> Perfect. Yes, we can hear you.
>> I I got 20 minutes. I just wanted to come early to test my stuff.
>> Oh, yeah, for sure.
>> See if we can give you some uh the turn off the camera there.
>> Dude, can you see us too?
>> They always end up going wrong.
I I cannot see you guys, but as a workaround, uh my boss is actually at Linuxfest right now and he he's at a table, so if he's able to leave, he's going to come to the talk and he'll FaceTime me so I can see the audience.
>> Oh, cool. Oh, that'd be great. Yeah.
>> Okay. And then we have uh microphones that we can hand out for Q&A afterwards.
They're just little things. So, that would be >> Yeah. I'm also going to be in Discord uh after my talk to if so that people can talk to me about what I talked about.
>> Okay, cool.
>> I put a QR code on my last slide for that.
>> Right on.
>> Okay. And then I the other thing we can do is >> make my bed for the background.
>> Yeah. Uh so we want to turn Oh, I just realized it's I the audio's the audio is fine. I just had it I just realized I had it going out of my speakers and it might have been picking up on my mic, but I switched it to my headphones.
Yeah, good enough. It It's a little staticky, but I'm just turning it down.
Why is my computer tweaking?
Uh, no, no feedback. It's just a little staticky.
Huh. That's bizarre. My My headphones are the on my system. It keeps just lowering my input.
That's weird.
I was sick.
Oh, yep.
Uh, can I give me three minutes? I'm going to refill this up so that because I'm a little sick, so just in case I need water.
Awesome.
All right, I'm back.
That was That was annoying. I'm on I live in a dorm and I get filtered water from the first floor and then the elevator was just going on every floor both ways.
I I'm also on the third floor, but we we have we have 12 floors and uh Huh.
Yeah, it's it's nice because usually I'm able to just like run up the stairs when I get back.
Google SL. Oh, here it is. Presenter view. Sick. Because I have two monitors.
That works perfect.
Cool.
Uh, so do I have do I have a 30 minutes total door or is it 20 + 10?
Oh, perfect. Awesome.
Awesome.
Yeah, my practice run took 28 minutes.
So, but I I was worried it would be 20 minutes with 10 of Q&A and I'd have to rush through it because that's what it was like at Seagull.
Awesome.
Awesome.
>> Okay, >> awesome. Uh, I can I can also record myself too and send the footage if you guys need.
Okay.
How how's been the setup for all of the stuff going? Is it going smoothly?
Because when I was doing AV for seagull, I was running around.
Awesome.
Not not to like crap on Seagull Great Conference. I just mean that I just remember doing that like Yeah. I I also last time on my talk I accidentally played copyrighted music and I realized that it would screw up their live stream like as I was about to play it but since I had experience doing AV from the years before uh I was I just hit up the guy and was like hey is it okay if I mute the audio for this so that it doesn't ruin your stream and he it was the same guy that was I was like uh basically my boss when I was helping the years before. So he was like, "Yeah, cuz he knew what he knew that I knew what I was doing." Except then I completely forgot to turn on the audio after and ruin my own talk on the live Dream.
I really wish I had blackout curtains.
That works.
I'll see if I can fix my lighting real quick.
rip. I was trying to fix my lighting.
Yep, I can see you guys.
It It's funny because you guys are on my second monitor and but I'm gonna have to talk into a camera. So So I'm Cameron Noff. So I'm a YouTuber in the Linux space. I actually just hit 10K the other night. So thanks to the community for that. I'm a 20-year-old entrepreneurship major at Wazu and I'm the founder and lead developer of Still HQ. I also previously created the Linux distro which was a Fedora based drove with a bunch of stuff added and I have a bunch of media experience because I'm I'm working I work for Tux Digital who does or Michael Tanell who does the this weekend Linux show and I help produce that show and I also work for Tech Hut who fun fact is at the Netbird booth.
So, it'd be funny if everyone went over and talked to him afterwards saying I sent him. But, uh, and then I also back when I was 16 used to do the video versions of the Pine 64 community updates. So, with that said, I think it's time for a change because uh, people are fed up with both Windows and Mac OS, but at the same time, just there isn't a better experience out there than those operating systems. And I believe that now is the time it would take off if there was a better experience. That said, I know this is a Linux conference, but unfortunately Linux distros are not yet a better experience for the majority of people compared to Windows and Mac OS. And Linux has the tech to be a better experience. everything is in place but it's not due to usability issues and as well as branding and targeting issues where a lot of Linux developers are trying to build for themselves which isn't a bad thing but at the same time they don't have any business or marketing experience and so they don't know how to target a type of user who isn't the person that they are and so my question is how can we build a new operating system on the Linux stack that can compete against Windows and Mac OS So, I'm a business student at Wazu. Uh, most of my friends don't know or care what an operating system is. And so, uh, oh, the thing cut out on the the laptop screen, but sorry, that aside, most of my friends don't know or care what an operating system is. What most people want is they want to just get their work done and get off their computer and then or maybe stay on their computer and just hop on a video game or watch a show or something. and they're using their PCs as tools and entertainment devices.
Now, people are very, very frustrated right now by Windows for a variety of reasons. You open your start menu and you see ads for random apps, Disney Plus, Spotify, Candy Crush in your start menu and then people are having to throw away their perfectly working fine hardware due to the TPM requirement they added to Windows 11. And so with that, Windows used to be 95% market share back in the day. Now it's back all the way down to 60 mostly with Mac OS taking over. Mac OS is now 25% market share, but uh 8% of this is unknown. That could also be Windows 2, but we have Chrome OS and Linux collectively together giving 8% uh market share too. So there's just tons and tons of random usability problems. Uh I'm not going to read out this entire list. This is just everything I can think of, but I'm not trying to just read slides. So, instead I have a little video I made of the Windows 11 installation experience.
Hopefully, there's audio.
Yeah.
It's just playing funny music right now over the Windows installation with all the here's all the Microsoft ads right here.
It's playing Windows and then you open up Windows and you go straight to another ad. So, so the point I was trying to make with that little I'm trying to stop it.
>> Uh, okay. Removed my slidem. Good one, Google Docs. But the point I was trying to make there is just people are fed up with the Windows 11 experience completely. It just completely nags you.
And frankly, it's just annoying. People are especially when you're trying to work. Now, Mac OS is a lot better, but it still has usability problems, too.
For one, you have to pay for Apple hardware, which I'm not trying to spend that amount of money. Uh, and then when MA when Apple stops updating your Mac, you're out of luck. And this is going to be even more of a problem with the new ARM Max because yes, there is Ashahed Linux, but that isn't fully hardware complete yet. It's not uh 100% finished and you're very limited on the Linux distros that are going to support it. I also just think multitasking in Mac OS is very annoying and it does weird quirks like uh if you close a window it doesn't actually close the app. You have to run hit like control key or force quit it. And then you also have a different keyboard layout on Mac OS which is very annoying to me because when uh Mac OS is my second primary OS when I can't do something on Linux and when I go between a Linux system and Mac I have to re get used to the Mac OS keyboard shortcuts. Even of all that though, I think it's safe to say it's 100% of a better experience than Windows.
Now, Linux has different usability problems that have made it so that uh people aren't willing to give it a shot compared to Windows and Mac OS. So, my question is how can we design a distribution that fixes these for your average consumer? So, let's start with KDE Plasma. Great desktop environment for Linux users. But the two problems that KDE have KDE has, excuse me. One, uh, the customization overload is just too much for most people. There's just too much choice. They don't know what to do when they open up the system settings in KDE. And the other problem KDE has is uh the app ecosystem around KDE just isn't as mature as the Gnome app ecosystem. They do have their own human interface guidelines, but they aren't really that enforced. And there's also all these different libraries. There's QAMI, there's uh just stock QT there.
I'm pretty sure there's another one other than Kaggami 2 that I just can't remember the name of. And so KDE apps just look very consistent across the whole eco sorry inconsistent across the whole ecosystem. Whereas Gnome has the complete opposite problem. So Gnome all of the apps look very coherent and it's fits in all into this one ecosystem. if you know how to use one nom map, you know how to use all of the nom maps. But on the desktop side, and unfortunately the camera for the room's a little broken, so someone might have to shout out, but if has anyone in the room never used the Gnome desktop before?
Uh, so does one c does one of the two hands want to guess how you're supposed to open a AMP app launcher just from the screenshot if you've never used Gnome before.
Um, so what you what you do on a normal Gnome desktop, mine's a little bit different, but you would go to the top left button right here, and then on the dash, you would have to click the show app buttons.
So to me, I think that's very unintuitive to someone who's never used a Gnome desktop before. You have to figure it out.
Now, Ubuntu does fix this. Subuntu has a lot more intuitive of a implementation of a Gnome desktop and so they make it more like what Unity was back in the day. So you have an app button right here and you uh that and you have a dock that's like always showing. Uh other distros too do a pretty good job of this. This is my own Linux DRO. It has out of the box a little bit more of a Windows layout, but we have a little layout switcher with uh 13 other layouts you can pick too to help suit your needs. And this is a copied idea from Zoran OS. So I'm not going to take the credit for this. Zoran OS uh does Gnome in this way very good too. But uh the problem is if you're picking the stock desktops, I don't think either of the stock desktops are ready for mainstream usage. Now first impressions really matter. When people install Linux, they're asking the questions, do their Wi-Fi drivers work? Does their printer work? Can they install apps that they recognize? Can they play a video without codec issues? Does screen sharing work?
Is an update going to break their computer? And a lot of the like big distributions like the Ubuntu's, Fedora, Debian is pretty bad at this at the moment. And a lot of community distributions like Linux Mint or Basite uh do a much better job at this. But if we're going to like the behemoths, just I this is really fresh in my mind because yesterday I made a whole YouTube video about this. But the Ubuntu app center for example is barely functional.
You'll open random pages and it it'll just break. There's really uh all of the featured apps and everything is a snap package which is actually okay for mainstream usage. I don't want to get into this whole package manager debate because a mainstream consumer doesn't care what package manager they're using.
If it installs and it opens, it's good enough. But then on the Ubuntu app center when you go and search for an app it ends up showing like separate apps and de sorry separate uh snaps and separate dev packages and it's not in a very coherent interface and it's just very confusing on which package you're supposed to use. And I think this is a horrible first impression for people going to Ubuntu, which is like the uh DRO to use or was for a long time the DRO to use if you're a beginner. And I don't really think I can call Ubuntu a beginner DRO anymore because of things like this.
And speaking of package management, package managers add a huge failure point to Linux desktops. So they're really, really good if you're a Power user. I don't think we should get rid of Linux distros that use traditional package managers, but over time when you're installing and removing applications, you're going to have uh leftover dependencies. Uh if you install the wrong app on like a KDE desktop, there's literally, and I'll see if I can find it real quick. There's literally a whole subreddit called getting gnomed where install random apps and then they have the entire gnome desktop on their computer. Usually this happens with ProtonVPN but oh this person got xfceeded right here. But yeah, this person's posting about losing their taskbar when in reality they just got gnomed. And this is a huge failure point that commonly happens with package managers unfortunately.
Uh, and then on top of that, if you're on a distribution like a like a rolling release distro like Arch Linux, Open Tumbleweed, there's going to be times where you have to manually intervene on an update to fix something. And if you're doing a big distribution update, especially like going between LTS versions of Ubuntu, like a 2404 to the brand new 2604, good luck with that doing it first try. there's always going to be random issues you have to fix. It disables your PPAs. If you have a package that isn't in the newer version, you have to delete that. Like just good luck. And so, and usually it makes it so that if you're trying to do a big upgrade like that, you just might as well do a reinstall.
Luckily to this though, we already have the solutions to this problem. So, boot C, uh, that's a technology that lets you make a bootable Docker container. And so, uh, people are building Linux distros like this that are super specialized and tailored to people's needs. And as a maintainer, they're able to basically act like it for all of their users essentially and be able to manage it you through containers. And then you use Flatpack for app management, which is fully separate from the rest of your system, and it's able to get around all these dependency uh, bloat problems and disruptions that a traditional package manager uses. So with that, we have the rise of mutable distributions here and I think that immutable distributions are going to play a huge part getting more people involved with Linux and not having to worry with the package manager. So these are distributions where the entire OS is distributed as a complete package in some way. With bootc it's containers with Nyx OS, you have a Nyx config file where everything's in one config file that you can share around and it generates your system based on that. And this is an area of Linux that is growing and exciting. You have Fedora Silverblue. You have the universal blue project which maintains several distros using the booty technology. Still OS is my dro. Uh Endless OS uh is one aimed at the education sector. We have Gnome OS and there's actually KDE uh Linux coming up too. And these are supposed to be atomic reference implementations of the desktop that other distros can refer to.
Uh, I already mentioned Nyx OS and then let's go over to SteamOS because this is a billion-dollar case study of how this technology can be used to actually bring Linux to a consumer oriented audience.
So the Steam Deck is actually built on top of Arch Linux, which you think would be a horrible base to build a consumer product on top of, but thanks to this immutable technology, Valve is able to make a very stable consumerfocused Linux-based OS to power their devices like the Steam Deck and the upcoming Steam Machine. Now, next up, let's talk about the brand image of Linux. Now, I love the Linux community. Like that's why I'm doing talks here and trying to improve Linux. But outside of the Linux community, this is what they think of us. They think of uh people they think of people racing their desktop environments. They think of really old Unix desktops on old computers. They think of Linus TechTips breaking his system with yes, do as I say. And yes, they think of the arch community which is often compared to vegans.
Now the problem that Linux has now the now basically the problem with Linux has is the Linux brand encompasses basically every single Linux distro there is. So there's going to be very userfriendly experiences on Linux like a Linux mint or a basi even things like that. But then at the same time the Linux brand also is going to encompass your Arch Linuxes and your Nyx OSs and your Gen Twos all the dros that a new user should not be using. So the whether you like it or not, there's there are people in the Linux community who recommend Arch to everyone, including brand new Linux users. And someone who gets a bad recommendation from someone who's just out of touch or could be a troll, too, uh is going to install Arch and have the whole brand image of of Linux ruining them because they had such a bad difficult time trying to set up Arch as a new user. And even then, like I was saying, Ubuntu was still known as an Ubunt a brand new user-friendly desktop. And like I said, I don't think Ubuntu is that beginner friendly anymore. And distributions like that are giving uh Linux as a whole a bad brand perception even if the distribution itself is doing exactly its specific goal perfectly. Like I'm not trying to hate on Arch and Gen 2 and Nixos here.
These are all really great distributions. that you have a lot of control over your DRO and they do exactly what they intend to do, but the problem is just the Linux brand has to encompass all of these different things instead of just one specific product or project.
So, how can a DRO work around this?
Well, I think the solution is to push itself as its own platform. So going back to Steam OS, yes, in the technical specs specs they do put that it's based on Arch Linux because even Valve wanted to say I use Arch by the way, but but uh they don't market specifically that they're a Linux project. They market powered by Steam OS on the Steam Deck and things like that. And Chrome OS is another example. And some people may be reluctant to call Chrome OS a Linux distro, but part of why people are reluctant to do that is because they do such a good job at pushing itself as its own platform. Chrome OS is actually based on Gen 2 and you can do a lot of Linuxy things in Chrome OS. Like you can run Linux containers and through that even run like flat packs and do all of that sort of thing. And that's a distribution of Linux that has actually been able to establish itself in the consumer mainstream market.
Now go now uh I'm going to come back to this idea in the next slide but just another huge problem with Linux is the choice overload really hurts this hurts Linux for the mainstream population. And I mentioned this a little bit with uh when I was talking about KDE, but Linux people, they ask which distribution should I use, which desktop should I use, which package format should I use?
Should I be using X11 or Wayin? Should I be using an LTS or a rolling release? Uh what init Linux people are into trying to figure out all of that stuff. That's part of the fun of using Linux, but everyone else is asking what OS just works. So I think another benefit of marketing a Linux distro here as its own platform is a platform like that is able to make difficult tech choices for you and remove this choice overload that Linux currently has. Now what I'm not saying is I do not think a upstream project trying to do something like this should just straight abandon Linux and just sort of take advantage of the stack to build something. Uh, I don't think that should be going on. And I think Steam OS is again another example of treating this correctly. Steam OS has a bunch of developers working on things like KD Plasma, Wine/Proton, uh, a bunch of gaming related stuff, even kernel work. Valve is doing a lot to make Linux as a whole a very good and usable uh, platform. But in terms of the actual marketing, they're able to market Steam OS as its own thing, which then is actually able to get people using Linux for a gaming platform on the Steam Deck.
Now, the next thing I want to talk about is app support on Linux. Now, app support has been getting much better, especially game support thanks to Proton, but at the same time, there's still going to be key apps like Microsoft Office and Photoshop that you cannot run on Linux. And these are killer apps when it comes to Windows and Mac OS. Luckily, a lot of the software industry is moving to the software as a service industry. And while sometime and while there is a lot of drawbacks for the consumer of the software as a service transition, like for example, instead of a one-time purchase, you're usually paying monthly. And because these are all web applications, they can just update things instantly. And it's because it's a website, you can't revert it or have any control over that. So if they make a huge change that breaks your workflow, you're out of luck. But I do think that these web apps can serve as a really good stop gap for people coming to Linux for the first time. So you now have web versions of Microsoft Office and even Photoshop here. And as a university student, most of my friends actually prefer the web version over the desktop version because they have all these collaboration features for things like group work and things like that.
And so, uh, I believe that if a Linux system is able to properly integrate web apps here, then I think that could be a huge advantage to actually help, uh, handhold people in their transition to using a Windows or Mac OS alternative.
Now, unfortunately, the web app integration in Linux is very lackluster at the moment. So, right here in this picture, the screenshot, I have a little Chromium web app right here. And if you like look closely, the title bar is thin. It has a bad margin between buttons like this, things like that. And all this seems really nitpicky, but uh first impressions again matter. And little things like this make Linux look like a unstable and experimental platform. You really need to polish things like this up. The other solution is there's peppermint OS back in the day introduced something called ice which would let you make like sight specific browsers and then uh the modern version of this uh Linux mint would fork this into the Linux mint web app manager which is the same thing and uses the same thing behind the scenes but uh actually has a much better user interface for doing it. The problem with these is it just hijacks a existing web browser and creates a new instance profile of that. And so if you were to open an external link, it would open in like a broken malformed version of that web browser that is being hijacked. And so that doesn't have a very good experience either. And I will say I am part of the problem here because I was actually a maintainer of ICE when I was a teenager and uh I wasn't able to find a solution for the browser hijacking when I was involved with these projects.
So I looked over to Mac OS and Mac OS does this perfectly. It's able to use uh the because Apple also makes Safari, they're able to take Safari's engine and be able to very heavily integrate it into the system and get very good web app experiences with things like the title bars, being able to match the colors of the web apps and things like that. Now, uh I built a technology similar to this for Stowos called SWY. Now, be we aren't using WebKit GDK because that just can't run the majority of web apps. Well, and that is a bad experience. So, we are using Electron to do this, but I basically remade uh libid way header bars. I didn't mean to do that, but I remade libida header header bars right here. And I made it so it actually matches the color scheme of the website.
So, this is photo pe by the way. If you need a good Photoshop replacement on Linux that's a little bit more viable to use for work than [ __ ] is because this has a lot more features that Photoshop has that [ __ ] is missing. This is a great uh app to check out by the way.
But if I change the theme right here, the title bar is actually going to adjust with the theme. And so, and then if I were to open an external link here somewhere, like if I go here, uh, I'm just trying to find something that is going to go outside Twitter right here. It's actually going to open it in your web browser that same web browser I was using to present my slides right here.
So, it's just going to give you a much more polished experience to use these web apps and it really makes these web apps feel at home. Uh, even some of the web apps are actually able to interact with each other. Like I have notion right here and if I were to open notion calendar through this, it actually opens the notion calendar web app right here instead of the same the web app in the same type of window. And so it really makes these web apps feel like a native and desktop experience when you're using them. And I think this could be a perfect stop gap for things like Microsoft Office, Photoshop, things like that. Because now uh if people go to the software center and they type Microsoft Word, instead of people getting frustrated because they get some random program they've never heard of called Libre Office instead, now they can actually open a web app for what they're already used to. And I think it serves as really good handholding that people can use and they're actually ready to switch to something open source like Libra Office or Only Office.
Now, even without the common apps, I still believe that Linux could catch up.
If you look back at 2002, I have I can't remember where I found this article because this slide I made a while ago, but uh back then, Windows had 97.46% 46% market share and Mac OS only had 1.43% market share. But through just improving Mac OS constantly and polishing it up and then integrating it with Apple's other products, like at the time it would be an iPod and then eventually became iPhones, Mac OS was able to create an experience that was worth switching to even if it didn't have the app support at the time. And through that nowadays, Mac OS has 25% market share in the United States right here.
So you could I believe Linux right now we have more market share than Mac OS did in the past. If developers make the right moves in the Linux space I believe we could make a comeback the same way Mac OS did back in the day. Now Chrome OS is another case study here to look at people. Chrome OS proves that people will use a non Windows or Mac desktop if it's simple and easy to use. And web first is an acceptable standard for schools, students, families, and offices alike. So, it's an example of the app gap being reduced just from having good enough web app integration. But also worth mentioning, Chrome OS one because it was on devices that you could go out and buy at like a Best Buy. Now, I'm not saying we should do a full complete uh web first operating system the same way web uh Chrome OS does, but if you have a Linux system that already has great integration with like normal Linux apps and everything, right? Uh and then you add the web app integration from Chrome OS on top of that and like I said, use that as a stop gap. I think that this is the ultimate chance to actually build something that people can actually use for daily work without uh needing to worry about technical support things like that. It's something people can actually figure out to use.
Now, the last point I want to make is like I mentioned of Chrome OS, we need to actually get this in front of people.
So, Valve's doing a very good job with this with Steam OS. Back to them. uh they're building a whole experience around the OS on a handheld where you can play your entire backlog of Steam while you're laying in bed and they're using this to actually build out a whole new platform that has been able to accelerate Linux gaming. So, if we have cool new hardware like this that can be for niche purposes but is using Linux behind the scenes, I think that is a great way to encourage more Linux adoption. Additionally, uh I think if we start putting a stable Linux OS on things like uh refurbished computers and like recycled computers and give those out to people in need and they can do their work on them and use them to better themselves. That is a great opportunity to get more people learning how to use a Linux system as well. And then the last idea I have right now to get Linux in front of people is if you need to maintain like a fleet of Windows computers, the IT for it is very expensive. You have to pay for all these Windows licenses, get your own servers, hire IT staff. And I think if we use some of the atomic like Bootsie technology or something like Nyx OS, we could have these small consulting companies be able to manage whole IT fleets through the atomic Linux technology and use that also as a way to get Linux out to businesses where then the employees have to use the Linux system and then they're going to realize that it's a better experience than Windows and then maybe they might consider Linux for their next computer.
So a summary of all the things I'm going to be trying to with Stowos in order to make this stream possible. One, right now we are very entwined with the Linux ecosystem, but as Stow OS grows, I want to sort of develop a whole new brand that isn't related to the Linux brand alongside Linux. So, I want people to be talking about how good and stable still OS specifically is and not have to worry about talking about Arch getting grouped in with Still OS. And then also, like I said, the heavy web app integration, I think, is a really good stop gap for the app problem that Linux has right now.
And I also think we just need to be focusing more on stability and the overall user experience. So, we really need to get all of the way stuff ironed out as soon as possible. need get need to get all the flat pack problems ironed out. Even the atomic file systems, yes, while it makes things way more stable, there are still a little bit of small amounts of usability problems like how hard it is to overlay things like drivers onto it. And so we need to really be focusing on the things that affect the usability of the Linux experience. And that is something I will be trying to do with Still OS. And then finally, we just need to get something like this in front of people. Like I said, more Linux devices, uh, recycling centers and computer labs, things like that. All of these are great ways to drive Linux adoption.
So, I've been looking for contributors.
I It says I've been developing STO by myself for three years. That's a little inaccurate because I have had new contributors come in in the last two months, and I have been getting uh help developing things, but I'm still looking for new cont uh contributions. I really want to build a community around ST still OS. So you don't have to be a programmer to help. Uh eventually I'm going to be turning the app store in still OS to be something way more interactive with like blog post with article or sorry blog post with app recommendations, things like that. Even like screens where you can find alternatives for common Photoshop apps.
So I'll I'll need people to help curate the app store and do things like that.
even if you just have new ideas on something that could help improve the Linux experience for people that is very helpful to the still west project here and with that I did have to switch this talk to be virtual because I was sick when I would have needed to travel here and I still am a little bit sick but I wasn't able to show up in person because of that but uh after this talk and even after questions I will be hanging out in Discord VC if anyone wants to talk to me about Stowos and there will be a QR code for that so This is my call to the Linux community to design for everyone, not just Linux users. Build for the top 90%, students, families, offices, creators.
Prioritize stability and ease of use.
Treat the web as a for a first class citizen and meet people where they are, stores, labs, refurbs channels.
With that, that's the end of my talk.
Feel free to uh send any thoughts you have to to my personal email. And then this QR code right here is for Discord.
Yeah, I can hear you guys, but I can't see people anymore.
There we go. I can see it now. Hello.
Uh so Android like pretty much hard forks the Linux kernel and even like device support ends up getting put into different random various forks of the kernel. And that is also why it's so hard to build custom ROMs. you have to build a custom ROM for every single device tree. And so, uh, I don't really think Android does that much to help the Linux ecosystem because they're barely doing anything upstream at all. It I don't even think it uses GNU core utils or anything like that. So, uh, I do think Android is Android is a good example of a separate platform, but I also don't think it really helps Linux adoption at all. Whereas Chrome OS on the other hand is close enough to Linux that I think that does help with Linux adoption. Uh Google is using Gen 2 to build on top of Chrome OS and I believe they have contributed upstream to Gen 2.
And on top of that, there's all these Linux container features built into uh Chrome Chrome OS and things like that that help introduce people to Linux.
>> Hey Cameron. Um, you mentioned that you tested out the desktop environment on college students. Could you talk a little bit more about that? How many students, what the general feedback was?
>> Yes, >> it was only a few, but basically I took a bunch of uh people that were in my classes who are not tech-savvy at all.
And I was trying out variations of uh the custom gnome layout people that I have. I was basically just handing people my laptop on a fresh user account and giving them certain tasks like uh can you open an app easily like what would you do to open an app and uh can you install an app through the software center and I was just sort of observing them do certain like very common tasks to see if they could figure it out without help and that's that was how I came to the design that I ended up shipping for still west but even then I'm still iterating and taking feedback like uh one of the defaults. This isn't the default layout, by the way. This is my preferred layout, but it's I made it I didn't put my preferred layout as the default because this is confusing for some people. This is just how I like using Gnome. But even with that, the default layout I did ship out auto hid the taskbar. And while the college students I tested it on were able to figure out uh how to open an app and everything, uh I have been getting a lot of feedback to remove the autohiding as a default because it's confusing people.
And that is going to be something I will be doing in the next update. And so you just have to learn how to talk to the type of people you're targeting and just be able to take their feedback well, take criticism and iterate.
question.
>> Yeah, sorry.
I guess I'm a little worried that uh this push towards accessibility is kind of risks moving away from the spirit of Linux. Um because I think a lot of the reason that proprietary systems are so locked down is to make them dummy proof.
And I'm wondering if um you're at all concerned about that with this effort.
Uh, so I I agree with you a little bit that some of the design decisions I'm making are against sort of the spirit of open source and Linux. I actually do agree with you on that. But at the same time, the people the current like market of Linux just isn't able to target people who don't care about that, which is to be honest, most people most of the mainstream doesn't care about things like that. And so if we want them to be using an open platform at all, you do have to make some sacrifices in some places. That said, that is another argument for making marketing still OS as its own platform is uh you can sort of separate it to where uh if you don't want a system like that, you can go use a traditional system. Whereas if this is something you're looking for and you're just looking to get your work done on a system that isn't going to nag you, then still OS can exist in that ecosystem too.
>> Does that does that uh answer your question?
>> Uh hello.
>> It's it's just about more options too.
just even if this isn't a thing that everyone would want, it's an extra option for the people who uh could use something like this. Sorry.
>> When talking about getting the interest out, will you be utilizing droatch?
That's what I've been referring anybody to. Just like go watch, you know, check out the top 15, see what kind of fits.
Um, do you plan to have still less or is it already listed on Discwatch?
Yeah, we are listed on DRW watch already, but I'm also uh I just had the first stable release very recently in the last month and I did get a lot of feedback, but uh there's still some things I want to change before I'm ready for like the entire Linux space to be covering it. So, I've intentionally at the moment not emailed press yet, but uh I I'm actually in my finals week right now for university and as soon as my finals are done, I'm going to be putting all my prioritization into this update that I'm going to do. Uh that will should be out about two weeks after my finals. And then once I get that update out and I'm it's polished enough for like mainstream attention, that's when I'm going to be going on my press run.
>> Um I was curious, the web app sandboxing looks really cool. Um sorry, is it are the are the when you when you put a web ba web app in its own window, is it also kind of sandboxed like >> containerized? You're using secret?
Could you just tell me more about uh the sandboxing technology for the web apps?
>> Yeah, so uh there's basically two types of web apps. So some of the web apps are just oneoff web apps. And I'm trying to think of an example. Apple Music, for example, this is the only web app I have in the Apple ecosystem at the moment.
And so by itself, this web app uh isolates itself completely. But when there's multiple web apps installed that may for some reason need to be able to share cookies and things like I used notion apps talking to each other as an example earlier. So the notion ecosystem these these are all isolated specifically to the notion apps basically and then none apps outside of the notion ecosystem aren't able to like share cookies or anything but the apps that we set up to work together are able to share things like that. So it's kind of isolated on like a web app vendor level.
Yeah.
Yeah. I can open a file real quick to show you what uh one of these web apps looks like. So nano. Yes. I'm not using Vim. You make fun of me.
uh or it's user local share apps and I'll do I'll do notion.
Oh, I think it's so some of these apps like I built while I was making them. But as you can see right here, these are the allowed URLs.
So if it either hits a hits a URL that isn't in the allowed URLs or there is an explicit notallowed URL which right here I have the other ecosystem apps for that as an example so it knows to actually open the other apps in the correct app. Uh that using the uh these two URLs will open in the regular notion app. These URLs will open in either the notion calendar app or the notion mail app. and then every other URL will open in your browser and not share cookies or anything.
So, it's it's supposed to just feel a lot more native without all the weird like clunk that the other web app technology is using. The only problem is I don't expect beginners to be able to make a file like this to build one of these web apps. So the way we're doing this is uh still HQ and then also just contributors make these files and then users just grab them from the app store.
Yep.
And with cosmic desktop um oh nice >> the cosmic store is a rust reimplementation that does use appstream data. So if you do want to look at something that is >> quite fast um >> uh >> it is >> uh flatp pack enabled first uses package kit in the back end. So if you have >> package kit integration, it is a pretty nice brand new Rust codebase to be able to take a look at things.
>> Awesome. I also I built this app store too. It's still a little bit MVP. Uh, I'm gonna be working on adding more things to it, but uh I was having what I've I've noticed that just a lot of app stores on Linux tend to be very buggy, especially on uh Debbie and an Ubuntu based dros like Gnome Software Center and KDE Discover. So, sorry I know Pop OS is based on Ubuntu too. I haven't tried Cosmix, but one of the nice things about uh building a stack like this for my own OS is I don't have to worry about other Linux distros. So I was able to design this around only the text still OS uses and not have to worry about uh having to uh support every single package manager which is the problem package kit is trying to solve. So the apps through this are able to like far more consistently uh install on the first try because it's bypassing package kit entirely. And I also want to do some experimental things where in the future uh some of the apps that have flatp pack issues like steam and web browsers for example I want to maybe move those to maybe instead install Nyx package and be able to pick the one that works the best for the user so that they don't have to think about it.
Okay, got it. Thank you.
>> And then we can do >> anyone else need the QR before we stop.
>> Can I say QRL?
Related Videos
Agentforce NOW AMA: Build with React and Salesforce Multi-Framework
SalesforceDevs
490 views•2026-05-28
How agent o11y differs from traditional o11y — Phil Hetzel, Braintrust
aiDotEngineer
450 views•2026-05-28
WEB TECHNOLOGIES UNIT-2 | Degree 4th sem BCOM Computers web technologies unit-2 full explanation💯✅
LearnwithSahera
1K views•2026-05-29
More tests are always better? How to use AI to identify tests that bring little value
Alliance4Qualification
335 views•2026-05-29
Search Algorithms Explained in 60 Seconds! 🤖💨
samarthtuliofficial
218 views•2026-06-01
People of Game of Thrones using JavaScript DOM
AltCampus
296 views•2026-05-30
Introduction to Problem Solving Part - 1 | Lecture 1 | Intermediate DSA
ascensionix
107 views•2026-05-29
🚀 BCS613C Compiler Design | Module 1 to 5 Schema Evaluation 🔥 | VTU 6th Sem 💯 #VTU #bcs613c #exam
Pranavaa-y4y
104 views•2026-06-02
