High Performance Compute jobs and training. Outlining a course. (part 2)

Añadido: 2026-05-31

[00:01:28]All right, let's see what shenanigans we can get into here.

[00:01:37]Let's go.

[00:01:40]Let's see what we can get into here.

[00:01:44]Transition this over. All right. So, what are we doing here? Um, I am kind of sitting around and working on this course a little bit. Uh, I I etched out some more stuff today on the high performance compute stuff that we were talking about yesterday. So, I'll probably bring some more of that up. And then uh I just need to keep cleaning it up to to kind of have something ready to kick over. I don't think we really have like a kickoff meeting or anything, but I'll probably I'll throw something on uh the the calendar later. So, this is going to be different because this is going to be probably modified in some way. And it'll probably be um this will probably be red line when it goes that way for my company. Uh and when we spell red line, we do it like that.

[00:02:27]um access to the HPC, right? Environment.

[00:02:37]Uh probably need to modify this.

[00:02:40]Probably need to update the the way that we access the environment. So, it's a little bit different model for that. Um we tend to do something that looks more like I wonder if I have this somewhere.

[00:02:53]Um, let's go over to Excaladraw real quick.

[00:02:59]I don't think I have this loaded up on here.

[00:03:04]I'm pretty sure I don't.

[00:03:07]Yeah, these are all just examples and stuff. Uh, if I had it, it probably would be in here. Let's go see.

[00:03:18]Um, it'll look it'll look something like this.

[00:03:23]I think somebody else was drawing that.

[00:03:26]But let's see what I was doing.

[00:03:32]So, my different Yeah, these are different things that I was drawing for people. We might just draw something new. Um, yeah, cuz we don't need It's not really It's not really in line with any of these things what we're going to be doing. So, it's going to look more like this. It's going to be um instead of jumping into an environment, we tend to do a something a little different.

[00:03:52]So, it'll be more of a um let me go grab this internet here.

[00:04:07]Go draw this over here. If you guys got any questions, you know, let me know.

[00:04:10]I'm just kind of messing around. Um this will be internet and then um won't be any of this anymore.

[00:04:20]It won't be that. So it just saves me drawing that I guess. And the connection in will be created via what's that tool we end up using? It's uh let me go look. Yeah, some kind of OpenVPN.

[00:04:42]So, you're going to openVPN certificate into our uh what do we use? I think it's um PFSense, right? So, we've got a pfSense device.

[00:04:55]So, PFSense VPN uh and for lack of better terms, we'll call it concentrator, right? So, there will be a PFSense VPN concentrator sitting right there. And then that'll drop you into the environment which is something like this and probably gonna have some dotted thing like that and then you know lab environment and I'll update this later probably as I as I add more and more to it right um as I know kind of the the structure that we're going through from there. Hey, three code warrior. How's it going? Um, we are just working on updating some of this. So, I'm kind of etching it out right now. None none of it's perfect yet, but it's it's kind of like um you know, you just get it get it close to what you need first and then you can modify it as you go. So, I'm going to get rid of this image and I'm going to drop that one in there and then that'll red line access to HPC environment and then we'll have to have some kind of setup with the users where they they get access in there.

[00:06:06]Uh what's the name of the site with the sandbox learning? Uh MZ Fitness, you might have to you might have to tell me a little more about the question. And I say that only because uh I've shown a few things in the last couple days for sand like that I would call sandboxes.

[00:06:20]Um so you know um in our in our home lab, if you're talking about the prologue lab, the lab that I run, that's over here.

[00:06:29]So, if you go over here and then you go down in our discord to the uh prologue lab environment and go to the pinned messages, it shows you how to get access. You give your public key, never private key, right? Never never let the private key go anywhere and it'll give you access and then you can connect in to the system here. So, that's that's the lab set of servers that I maintain and those ones get updated. Uh we get little update reports on those pretty regularly. There's 18 of those systems that you guys can get into and do whatever you need. Then when you're done, you just reboot them and they come up clean every time. So it's kind of a nice little If that's what you're wondering, that that's the the prologue lab environment. If you are talking about one of the other things that I showed yesterday, uh most everything that we were talking about dropped down into uh we put them in different places.

[00:07:16]So there was in the DevOps there was a link for a couple of good little git branching if you wanted to get better at git branching. And I actually really like this one, this Git game that uh you don't even have to log into. You can just go do all their stuff. It won't save it for you unless you log in. But hey, you know, if you want to go power through, I think there's like 25 of them or something. Um you can just start playing. I was doing it earlier. So continue without saving and it'll just keep track of all the things that you've done and it just walks you through how to use Git if you're not super familiar.

[00:07:45]So I'm gonna keep working on it too because I know Git, you know, I know I know the basics of it, but I don't claim to be some, you know, expert in it. I just kind of so I'm going to keep playing with that a little bit. Um yeah.

[00:07:56]Anyway, so I don't know I don't know if there's anything else you're wondering about that but that's that's the things that I know uh how to get into. All right. So instruction methods. So this course is designed to promote learner centered activities to support development of fundamental HPC skills uh around uh in a Linux environment.

[00:08:28]Course utilize individual and group learning activities, performance-driven assignments, problem based cases, projects and discussions. Uh we might not do projects for this one. I I don't think if if if you give So college is a lot of times when you when you have control of a situation kind of like it's a um um yeah you can ask any question you want. Caparazzi. In fact, I'll move this up. I I had this down because my other chat was kind of old, but I'll yeah, I'll move the chat on. If you if you got questions, feel free to ask them. I'll try to answer in the best way I can. Um, so let's look at this. Uh, the course utilizes individual and group learning activities, performance-driven assignments. Um, yeah, that's fine.

[00:09:11]Problem based cases, that's true. And discussions, yep, those are all true.

[00:09:15]Class size, the class will effectively engage 4 to six learners. That's that's just a general thing that I had from before. And then, so I need to add there's there's a thing that I need to add into this because I definitely pulled this from one of my older ones because this is the original. So, let me go pull another one of these that I have which is newer.

[00:09:34]You know, good writers borrow, great writer steal. So, I'm going to go steal from myself. Uh, where's my automation course syllabus? So, here's the automation course syllabus. And the main difference on this one, do sub.

[00:09:53]Yeah, I don't normally do sub bullets like that. Ah, this this. So, uh, this is a thing that I also like to do. So, we're going to throw this in there.

[00:10:07]So, in the instructional methods here, we're going to put that explore practice. So, this is a a acquisition of knowledge that I I like. This comes from an old college that I used to work for that doesn't exist anymore. And um the the idea is you explore things, you you learn about the individual component pieces of things and then as you kind of get into them uh then you start practicing with those things and maybe putting pieces together and then you apply it to like real world situations.

[00:10:36]So that's that. Uh let's see. You're working in Office 365 in tune in at work but for the most part T1 tech support some AD stuff mostly Windows. Should I pursue a Microsoft AWS? There's so many.

[00:10:49]Um, what I did, so I mean it depends. I mean, one, I would ask, does your organization use any of those cloud vendors or are they planning to? And that would kind of guide my decision of of where I wanted to start with that. Also, are you planning on staying with them, right? Uh, that's something I would ask, too. Um, the only Azure I have is the A900, which is like the general cloud one that I think most people could probably study for like 16 hours and just go past that thing pretty easy. Um, that's the Azure 900. Then there's the AWS uh like the the CCP, whatever they call it, the uh certified cloud practitioner, which is those are both kind of the the same just basic little I think they're both $100 shirts.

[00:11:31]They're pretty pretty cheap as far as that goes. Um, but I mean it depends like do you want to stay doing Microsoft? Are you planning on eventually moving over to Linux? That kind of stuff. Uh, for me, when I talk to people about this stuff, it's like, well, what do you kind of want to do?

[00:11:45]like do you want to get more into cloud?

[00:11:46]Then I would start focusing on your cloud search. If you say hey you know I don't really want to do that Scott. I want to stay on the tech world. Well I would you know I would look at you know what your team needs and what you know what your I have a friend who's stayed a Windows guy his whole career. I mean it's I' he's probably been in it as long as me and he just has never changed off being just a base Windows guy. And there's nothing wrong with that, right?

[00:12:08]Like for you guys that want to do that stuff. um most of us and the focus of of my group and and the stuff that I do is Linux, but um you know if that's like I can't dictate what other people want to do obviously. So um like Microsoft Entra and Tuned. Yep. I want to do more with cloud tech and I'm planning to stick around. Okay. Well then I would definitely start moving up there. Um I have a buddy that goes and speaks on Intra Intune. He's a he's a cloud architect. Um and he goes and speaks at all the Microsoft events. His name's Chris Turner. Um, and I've known him I've known him well before. Um, he was an architect, but yeah, he goes and speaks at all the Microsoft events on Entra and stuff. Uh, but yeah, I just, you know, I I stay I stay out of like the Windows space. They've been they they've made me do Windows over the years, but mostly I won't be um I won't be caught doing that too often. Um, let's go.

[00:13:07]Let's keep looking. All right. So, now we got to start turning in from what we have here. So, so this is the easiest way I know to do this, right? And and because I'm mostly because I'm lazy. I'm going to grab all these major instructional areas. I'm going to control C them. I'm going to throw them in my uh probably in a notepad. It's probably the easiest thing here.

[00:13:28]All right. So now we've got to align the course objectives that we have here with the number of lessons or sessions or meetings or whatever we want to call them. Right? So basically as the course objectives have been fleshed out from the instructional areas, then we got to think about what we're going to do. Man, I'm getting like pinged on email. Hold on a second. What's going on here?

[00:13:51]Oh man, I just got like five emails in a row about some AI learning assistant thing. Oh, it's going to tell me every time it emails one of my students.

[00:14:00]That's That's insane. That That's a bad model. I just got a bunch of pings if you guys didn't hear that. Uh that was um Man, so my college like so it used to be and I still kind of do it. We we do these things called um nudges or whatever you want to call it to get students back in class and working on stuff. And I just I just uh just this quarter we switched over to this AI that kind of does it for us. And I think that's kind of impersonal. So, I still send the little nudges every now and then, too. Um, you know, just touching base with students, seeing what they're doing. It's it's not as personal as when we used to get in the classroom and talk directly to students. So, anyways, it just sent me a message right now for all my students at Nudged. I'm like, I don't want to I mean, that could be that could potentially be like a hundred. Well, maybe this quarter only like 70 students because I only have two classes, but it's uh that could be a lot of students eventually, you know, if it's just nudging all of them. So, all right. Um anyway, so let's let's go here. Let's let's take the dates off of here. None none of these dates apply anymore. So we can get rid of all that. And then the sessions. Um if we're looking at a 16week thing, then then we could start to go from here. Um in this course, we'll be studying individually as well within a group of your peers. I don't know if this one's going to have a group component to it. So I'll leave that, but I'm going to I'm going to potentially pull any mentions of group and that kind of stuff out. Uh let's go then and look through this is this is from the basic Linux course right here. Right? These are the the components of the basic Linux one here but we're going to be obviously modifying it. So the first topic will be um obviously HPC architectures and components. A lot of these they will be they will not be one for one but they will you know obviously pretty close to one to one in some of these some of these instances. So, um, oh, oh, oh, MZ Fitness, now that I think about it, you might have been talking about Killer Kota. I I just realized that. I apologize. I I just realized right now that you might be talking about Killer Kota. Is that the Is that what you were talking about? Sorry.

[00:16:04]Where where it's the actual sandboxes and you can drop it and do Linux. I just realized when I saw sandbox, I'm like, maybe they were talking about this. So, I apologize if I've led you astray or if you've already left and and don't get to hear it. But yeah, this is this is where you just dropped into an environment.

[00:16:19]You just start doing right and just figure out what's going on in there. I think that might have been what you were talking about is Killer Kod. So this is a free site that just drops you in a Linux shell. You can do whatever you want in here. Play around with it.

[00:16:38]See what's going on in the environment you've got.

[00:16:43]Right. You've got a Oh, this one doesn't have. So, if you're in a one node environment, you don't have it, but you do have Docker and Podman. So, you can do Docker and Podman stuff in the individual nodes. If you go to a two node environment, which is the other sandbox, then you've got all your Kubernetes stuff. So, if you want to practice Kubernetes, I mean, that's I used killer cod mainly when I went and passed my CKA and my CKAD. So, that's that's what I used to prep for those. Um, but anyways, yeah, that's uh I think that might I just realized what you were probably asking, so I apologize for that. Um, let's say this one is going to be.

[00:17:33]All right. All right. So, components of NIST as they apply to or components of HPC as they apply to NIST 223. That could be the first topic. Um, the next one would be let's see see I think two I think two needs to move down because I think you need to understand the the networking and and uh I think I built I add one later.

[00:17:55]uh common benchmark compilers, deploy monitoring, troubleshooting, storage.

[00:18:05]There's also a network. There needs to be a networking component in here, too.

[00:18:12]Yeah, network designs up there. Okay.

[00:18:14]So, I might I might have to add some of these. Uh what's serious about? We are we're just sitting here working on um high performance compute like getting into uh this is a training that I'm going to be putting together. This is me kind of etching the outline of a syllabus for high performance compute for my company. So I might be doing this. This one will probably be something that I professionally end up kind of turning into something that this one won't be like the live one that that I do, you know, for for the HBC group.

[00:18:45]This will be something different. I'm just showing you guys like the component pieces of it in case anybody wants to study it or or work on it. So, yeah, that's that's what I'm working on now.

[00:18:53]Um, and I've moved my stuff over here so that I have them. And I will probably also be updating this section here because I definitely need to mention networking in here. So, update server. I need to also probably do um these are not these are not properly numbered and I don't remember why I did that. Normally I use the numbering but maybe it was indenting too much and it was making me mad. So I might have to manually do these numbers which is annoying but that's fine. And let's say um uh types HPC.

[00:19:44]So, that's going to be there. These all have to change all the way down.

[00:19:50]I know it's better to do it manually, but or automatically, but we'll we got what we got.

[00:19:59]Yeah, I like that more because networking is mentioned up there. And then I'll just regrab these for my course objectives here.

[00:20:09]There we go. One, two, three. Looks good. Okay. Those are just my my references down over there. So, as we do this, um, network uh network environment network say network devices and configuration HPC. So I've got a bunch of stuff that we so I have a bunch of docs that I can't show outside of the company but you know the last probably 10 deploys that I've been on. I I' I'm the guy that trains like I'm I'm the guy who trains um well both our internal guys with this and then also I train the customer often. So, I have multiple pages of like network devices and how we lay them out and and the different types of networks.

[00:21:04]They look a lot like this, which I showed you guys yesterday. Um, they look a lot like uh this here. Where's it at? Right here.

[00:21:15]This is the this is the most common model that we all use right here for high performance compute. Uh, is this for beginner or juniors? Uh, I would say this is probably what we're working on right here is is probably a little more advanced, but I think anybody could get it. Um, I'm also not strictly speaking giving actual training right now on this. I'm just kind of showing you guys the the training that I'm setting up.

[00:21:37]Um, so I would I would call this one more um yeah, this this could be junior, but it would be junior HPC stuff because uh it's not junior for like the Linux side of things. If we pull up at some Linux and look at some Linux, it's going to be it's going to be, you know, in the context of high performance comput. So, it's going to be a little more advanced um as we're as we're kind of looking at it.

[00:21:57]So, anyways, yeah, there's that. Uh so, let's keep let's keep going with this.

[00:22:02]Um class will meet over a weekend. No, so we're not doing over the weekend. So, uh during the week, uh during the week, brown bag sessions one time per week for 16 weeks. there would be a total of 16 sessions. All right. Why is this doing this? There we go. I don't know why that was scrolled in so far like that, but yeah, that's that's how we plan on doing it. Um, this will likely be when I do this professionally, it'll probably be two times times per week for 16 weeks.

[00:22:34]There'll be a total of 32 sessions. So, and and the way this will work is we'll do probably and I'll I'll work this out with the with the boss when we start getting because we're going to do this internally for some people and then I think we may actually offer this training for other companies because uh uh we we kind of specialize in this HPC stuff my my company. So when when we do that, I would like to meet with the people earlier in the week, get them going, and then in the end of the week, verify that they're on track before, you know, they they get to, you know, so maybe like a a Tuesday, Thursday or Tuesday, Friday.

[00:23:13]Um, where it's the same, it's the same concepts both days, but the first day I show them everything and probably walk them through the stuff that they're doing. Thursday or Friday, I make sure that they understand it and and kind of go into some deeper topics around it and then probably move forward. So, yeah, we'll probably do we'll probably do it like that. Um, and then this will this will modify my my my sessions over there. So, it'll be like session one, two, three, four. Um, and then the dates will be within the same week. Uh, how long does each session?

[00:23:46]So, when I do this with um with them, it'll probably be it'll probably be like a 1-hour like we we kind of call them like brown bags. Like I I was told a few years ago that brown bag's offensive and I'm like I don't I don't know who that could be possibly offensive to. Um you know, like like I brought a sack lunch to school. Like what I mean, so I guess I'm in the group so I I understand what a brown bag is, but apparently like that term is offensive, but I'm still going to use it. Uh but anyways, how long does the session take? I'd say like an hour cuz it's like a lunchtime. like you bring your lunch or as you're you know kind of doing whatever for for lunch you kind of sit through the meeting you know it's not going to be this won't be like an afterwork type thing um and it won't be an intensive however if so if we do it for our company it'll be probably you know spread spread out but if it ends up being you know if somebody said hey we need to um or we want to pay for like a weekl long train like if my company was going to do that it would probably just be in it would probably 80 uh sorry uh 40 hours then. So it would be basically these 32 sessions as 1 hour blocks and then probably we'd probably do it in like um if you guys have ever heard like a boot camp session. The way boot camps tend to work is it's a 10-hour day all the days of the week but you work over the lunch on laps. So we would do and then also you go home at night and work on laps too typically in those environments. So that would be like a 40hour intensive if we were to do that. This would be like 32 hours spread out over the over the weeks, right? Um, that's kind of the way I'm thinking of it. So, this would be one, two, uh, three, four, five, 7, 9, 11, 13, 15, 17, 19. Whoops, where's my hand at? 21, 23, 25, 27, 29, and 31. Right. Oops. There we go.

[00:25:45]And then, you know, I can fill them all out. It doesn't matter. But I can do that later.

[00:25:51]But, um, yeah. Yeah. Yeah. This is So, and I'm going to do a lighter version of this externally for the prologue group as well. We're going to have like some intensives down in the Discord if anybody's ever wondering about that.

[00:26:05]Right. The the Discord's here, right?

[00:26:07]And then you go up to your events and here's all the events. Every event's free. We don't charge anybody for anything. We've got a bunch of events coming up. Bunch of bunch of speakers are coming in on uh August 8th and 9th.

[00:26:19]We're going to have people coming in that are going to talk about different things. And then we have our normal sessions that go throughout the different weeks as well. So, there's kind of rolling things that happen over the weekend that you guys can go play with, talk about. All right.

[00:26:33]So, where am I at here? Um, so deploying, man. See, and here's the thing, man. This is this is one of the hard things right here. So, uh, this this right here, this this next part is is honestly I could talk a whole week on this next one. So, it's like I can I can only do so much, but you know, we're just going to work through it. Um, deploy and uh deploy werewolf and configure serverless or uh it's it's um it's stateless but what we would call typically um yeah stateless and config and stateless node.

[00:27:19]So deploy werewolf and stateless node configuration. Now there's man like that that would be hard to talk to for only an hour and and really only even two hours because you got to deploy werewolf. That's its own thing. That's that's got labs to it. Um then we get down and we say well uh what about um building the images out? Okay, so now we learn how to build the images. Okay, now we secure the images. So those are like those are all their own component pieces, right? I could I could greatly expand this u for sure. And then YouTuber, you asked about sandbox learning. This is um yeah, this will be some of this will be out on Killer Cod.

[00:27:59]Some of this will be internal to the company, I think, because we have a pretty goodiz lab environment for a lot of this. I'll have to spin up and build the lab out. Um but that's not that's not too terrible. That's just man, that's so much stuff. So much stuff. If you guys have ever seen my talk on werewolf, it takes me an hour just to talk through the werewolf part of it, not even to get into all the labs and stuff. So, it's it's man, this one might almost have to be too um deploy uh werewolf and stateless. Then this one will also be um let's go st uh stand so secure technical uh stig is secure technical implementation guide and hardening uh systems right so that's secure technical implementation guides I I may shorten that because I know what it means and I guess they'll eventually know what it means. Stig uh Stig uh baselines and hardening system. So that's going to be one as well.

[00:29:10]And then the next part of it would be deploy and configure a job scheduler, right? and then uh operate and troubleshoot a job scheduler, which actually uh needs to probably be move I think both of those need to be moved down because we need to get the whole I need to have them build the whole system and have it up before they can do that. Um well, this this right here will actually this will be paid right here eventually.

[00:29:45]um with my company like they're going to have I was talking to the boss one of the bosses the other day and he was like yeah we'll we'll figure out some kind of bonus structure and like some of you guys might go I I know people that would be like ah I'm not going to trust a company to do that. Well my company is pretty good about bonuses. I I think some of you guys I I I've told you guys that I'll just be sitting around working some days I'll just I'll just go in and you know on payday and I'll just get like three checks and I'm like what the hell are all these for? And like I'm not complaining but it's like what the heck's going on? and they just give me bonuses. So, you know, I'm I'm pretty okay with I'm I'm pretty I guess uh I've worked with them long enough that I I trust that, you know, my time is going to be in some way rewarded. But, um what resource would you suggest someone prior to do to the boot camp? Well, I would tell you so question is uh what what might you do? Let's get this out of the way. Let's get this out of the way. If you guys don't know, in the professional Linux users group, we have a set of um courses course books that are already out there. And oops. And these are Whoops. Come on. Course books. And uh the one that I would start with if if you were like, "Hey, Scott, where where would you start with all this?" I would start right here in the Linux admin course. So here's the Linux admin course. This stuff's all free. Uh this stuff can all be connected to and gotten into. Look, this is the same type of thing like what are the major instructional areas? What are the course objectives? How do you connect into our bastions? What are you learning? And then you just go through these, right?

[00:31:14]This is all online. This stuff's all free for everybody if you need the the server. So then the server connection from these boxes is all down in our Discord. You come down here, you go to our prologue lab environment, and the pinned message tells you how to load a key in.

[00:31:30]We got a little bot that takes your public key, puts it on our server, and then you can log into our lab environment. And the lab environment, if you guys have never seen it, is just a bastion node like this. If we go right, bang SSH, he he connects to himself in this case, but you guys would be connecting remotely. It looks like this, right? You guys will be using a key. And then I can pick whichever course I want to go to. And let's say I want to go to the automation course. I'll go in there.

[00:31:55]It'll send me over to one of the auto servers that nobody else is using.

[00:32:00]Right?

[00:32:02]Then you can see the environment. You can do whatever you want to, which is what all of that other stuff does. So that's that's what I would focus on if I was if I was like, well, you know, I don't I don't do much Linux or I want to do more Linux. That's that's what I would go focus on is that kind of stuff.

[00:32:15]Um Yeah. Yeah. And um for people that are doing this, um you've been cut out short with uh caught out short with program.

[00:32:25]Yeah. I wonder like I I would imagine like you'd want to know a good amount because they're going to move fast, right? And honestly, man, that the amount of information I have to try to stuff into somebody's head even if this was a week, right? If this was like a week, 50 and we did 50 hours, let's say, I still don't think I could talk about all this stuff that we do in the high performance compute world, right? Like, but but I don't know, man. That's one of those things like um if you guys have ever heard there's an old saying by General George Patton who was uh one of our generals over in Africa in World War II and he was you know very famous for saying well he was very famous for a lot of things. He was pretty good general.

[00:33:02]Uh but he he very famously said um a good plan executed violently now is better than a perfect plan next week.

[00:33:09]And I think that that's kind of that's a lot of times how I like outline things.

[00:33:13]Like I'll do the best I can and then I Uh, in fact, I think what I'm going to need to do here, um, network devices, stateless. I I think I'm going to do storage first. So, where's my where's my doc at? Where's where's my notes here?

[00:33:30]Um, storage is down here.

[00:33:37]Yeah. Yeah. So, storage might have to get Well, storage for sure needs to be before this. So, let's go storage.

[00:33:50]Um, actually, storage kind of needs to happen before the nodes, too. Um, just hold on.

[00:34:03]Get out of here. Get out of here. Um, let's talk storage next. So, where's my storage section there? Um, storage types.

[00:34:21]Let's see. How do I say it over there?

[00:34:25]Deploy and uh run. Let's not say it like that. Um, let's say storage types, deployment, and um I say troubleshooting and operations. Yeah, that's fine. We'll say we can modify that later if we need. Uh now I want to drop my deploy job scheduler.

[00:35:06]Boy slurm as a job scheduler and then operate slurm operate and troubleshoot slurm. If you guys have never used Slurm, it's just a open- source little tool. If you want to have a really quick run through of how slurm works, I've built a very basic uh lab on it. I'll give you guys a link to lab here, but over here, me uh Linux Labs. There's a bunch of labs out there for you guys to do if you want to get better at this stuff. Uh where is my slurm. So, this is installing slurm on Linux. So, if you guys need this one, I'm not going to do the whole thing right now, but uh basically the lab just starts you off. Says, "Your team wants you to set up a proof of concept around scheduling jobs across multiple nodes.

[00:36:04]The end goal is running a compute cluster that functions like high performance compute cluster, right? So, you want to build out a scheduler and then set jobs off to the environment.

[00:36:13]Stall slur and have the appropriate dams running. So, come down here. We would, this is still setting up in the background. We'll set up slurm, install it, make sure that the munge key is the same because slurm talks over munchge. I wasn't supposed to do that yet. Um, we can click on all these and just update everything if we wanted to. Um, the munge key is how slurm talks, right?

[00:36:34]That's a a project that does um keybased data movement.

[00:36:42]You guys can play with it if you want.

[00:36:43]Anyways, you come in here, you update everything, you you fix your uh slurm config to work for the lab environment that we have. And then when we're done, this is going to fail, but I'm just going to skip it real quick. Yeah. Yeah, that's fine. Um, and then we'll come over here and then we would start to once we had slurm up on both boxes, then we could submit jobs and test and play with it and stuff. So, that'll be part of what would eventually come into here as far as far as the configuration.

[00:37:07]There's two other things that need to happen though um in there and that is um I need to set up the so as part of that will be the deployment of the DRBD. So slurm has a DRBD a database backend and then also um later on is the slurm monitoring. So in the monitoring section I got to hit on slurm and show how to pull information out of there as well. Um Dillinger said dropping by looking for months. Not sure if it's nice. Nice.

[00:37:37]hopefully appear over in Discord. You're studying for the CCNA. So, yeah. Yeah.

[00:37:41]Uh and and if you don't know my backstory, I actually got my CCNA back in 2002. So, that's how I got into networking and and then eventually moved over into being a Linux guy to last, you know, the better part of 25 years now or so. Um I've been doing Linux, but mainly because I didn't love networking.

[00:37:59]Networking is fine, right? I still teach it. I still teach people how to get in and do all their Cisco commands and stuff. um prepping for their CCNA if we're being honest. But um but I don't like I don't do that professionally much anymore. Inside of high performance compute environments, I do still set up VLANs. I do still set up some routing if needed. But for the most part, um uh this stuff is just I I configure it up front and then I never have to touch it again because my my system's air gapped off and there's not that much changing of of things going on. So Oh, the wifey's home. Also, apparently my my thing's busted. Can you guys hear me again? Let's see. I gota I got to pull everything back up now. Uh, we can get rid of that.

[00:38:44]We can pull this back up.

[00:38:54]That up there.

[00:38:56]This back up. It's just because I was uh my thing busted. All right.

[00:39:03]So, let's go back here and look at what I was working on. So, 10 11 12 13 14 16 18 20 3 24 5 26 27 28 29 30 31 32. So that would be that would be 32 total hours worth of time. However we ended up doing this. Um but like I like I was saying is if um if I do this as like you know if my company does sell this as like 40hour blocks or 50our blocks even I would just expand these out and then I would just add more uh individual component pieces. Right? So, we're just etching it out for now. And we're saying as though we were doing it in in a 16w week, you know, meet twice a week type thing. Um, during two times.

[00:40:11]Oh, it says it already. Okay. me during the week.

[00:40:28]That is super annoying.

[00:40:32]What is this? What is this formatting going on here?

[00:40:35]Super annoying. What the What the f is going on here? None. Get out of there.

[00:40:41]There we go.

[00:40:44]There. That looks That looks a little more readable.

[00:40:48]All right. So, what do we got next? So, we'll do all that storage types. Good.

[00:40:53]Good. Um, slurm.

[00:40:57]Okay. Uh, user access user access and provisioning.

[00:41:09]get users into the system, right? Allow users, use some of the tools that deal with giving users access. Um, that's man, that's a long one, too. But that's fine. That's fine. We're going to we're going to expand some of these out and then we're going to have uh this won't be all one for one, obviously.

[00:41:27]um monitoring uh stack and we'll be doing probably alloy and um Prometheus and then Graphana, right?

[00:41:49]Let's say monitoring those be capitalized.

[00:41:57]I'll fix I'll fix all that other stuff later. Um, alloy Prometheus graphana. So, that's going to be all of our logging and that kind of stuff. Are brown bags even available anymore? Do youngsters even know why brown bag is a term? I was mentioning earlier how I've I've been told that brown bags are offensive. I was told this a few years ago and I'm like, that's stupid. I I carried brown bags to school pretty much my whole young life in the 80s and 90s. So, I don't know. I don't know how that's offensive to anybody. Um, but then again, there's a whole bunch of things that I I don't understand why people find offensive uh that we used to, you know, say and call each other all the time. So, uh, you know, maybe I'm just And it's funny, too, cuz I'm the oldest of the millennials, but I just I don't think I fit in with millennials very well. Uh, yeah, I'm definitely older older than most of them in in my uh thinking. So, deploy a monitoring stack alloy Prometheus Graphana. Then we'll probably do um man even before that.

[00:43:07]Well, that's fine. Um let's go.

[00:43:15]Let's say then um I want to say so here's here's my thought process. Um I I got to get in I got to get in RSYNC and the different types of logging. Now Alloy does logging for us sure but typically we still do some type of like arsync logging and we get centralized logging off the systems.

[00:43:37]Um oh man. And then there's also network monitoring.

[00:43:44]Oh, so there's a there's a lot of um alloy promethus graphana. A lot of the stuff is going to be anible based as well. So there's a lot of that. Um what kind of advice would you give a 20-year-old junior DBA who wants to become more employable and transition into an S sur DevOps field? You know, we got a really good um somebody somebody always puts a really good there's like a road map. Let me go see if I can find it real quick. I think it's in the DevOps section of our of our Discord, but there's a really good road map to the things that you have to learn to get into DevOps. I think somebody I think somebody posted in the last month or so, there's not a lot of stuff that goes in this channel. That's why I'm that's why I'm mentioning it. Um, somebody was buying stuff on CodeCloud. Um, I'm pretty sure it's not too far up here.

[00:44:37]If it's not here, then I might have it over.

[00:44:42]Let's see. Hold on. Um, what's that term? Um, I think it's road map. I think that's what they call it.

[00:44:54]Is this Is this the one?

[00:44:56]It's a DevOps road map. I'm pretty sure this is the one. I think Conno shared it. Yeah, this is the one I saw the other day that I really like. So, I'll link this to you, but it's roadmaps.sh DevOps. And this this is where, you know, this is where you spend your time, right? If you want to get into DevOps, maybe learn a programming language, you know, pick up one and start playing with it. There's all kinds of stuff online that that you can do. I think they even have examples here like if you want to learn Python, go, you know, do these.

[00:45:23]There's resources you can pay for.

[00:45:25]There's free stuff. Um, man, there's uh there's one what's what's that one?

[00:45:29]Codingbat.

[00:45:32]I think it's coding.bat. Is that it?

[00:45:34]Yeah. Coding bat does Java and also uh Python. So, you just come in here and you're like, I just want to do some warm-ups and learn how to do some stuff with Python, right? You can go. That's free. You can just get in there and just mess with their stuff, right? Go figure out, you know, whatever whatever you want. That's called Coding Bat.

[00:45:52]I'll link it to you guys right here.

[00:45:53]Python, right?

[00:45:57]Oops.

[00:46:00]There we go. There you go.

[00:46:04]So, if you want that. But anyways, uh for me, for my side of the world, right, and I'm I'm down doing the Linux stuff all day. Uh I I'm I'm mostly like a Linux guy. And then you got to understand from the terminal, you have to know bash, right? That's I think that that's always going to be true. You got to know your bash. And then from there, you have to understand maybe different types of monitoring, different stuff that's going on in there. or what's going on with processes. Uh looks like they've got some different stuff that'll help you out. There's also uh the best thing I can tell you about monitoring servers. Probably the best guy on the web for it is Brendan Greg. So I link this guy all the time, but his stuff's the best, right? His stuff is the the standard for pretty much everything that I I think that you need to know about what's going on inside of a Linux box.

[00:46:49]Um I can link that as well if you guys want it. You can do leak code for Python. I'm sure that's good, too. And I can't I can't give you guys great, you know, uh, advice on the programming side of it and the software engineering side of it because that's not what I do. But what I can do is I can show you this is a pretty good road map for me being a lower level Linux guy to say, yeah, these are all things that are are important, right? Important to know. So that's a DevOps one. All right. Uh, where was I at? I had my my notes up.

[00:47:17]Uh, deploy stack. Um, let's let's edge out or let's uh edetge out, not edge out. Edging is a totally different word nowadays than it used to be. Um, you can't edge anymore, just like you can't goon anymore. Those those two terms are off limits for us adults now because the kids have completely turned those into different things. Um, and uh, man, it looks like we can still say edge and can we also say goon?

[00:47:45]But those don't mean what they used to when we were younger anymore. You can't can't say that around around the younger the younger folks. Um, all right. What's What else is going on here? Where was I at? Uh, allow user access and provisioning. Um, so then I feel like user access stuff might we might need to move this down because while we do need to do it, we need to do it near the end. Um so let's let's put up in here before we deploy any monitoring let's finish the building of stuff which is compiling compilers and software environments with spec. So spec is a tool that will help you to standardize everything across your environment so it always comes up the same. Yeah. Yeah. Yeah. I have a one of my friends has a uh what does Jared have? I think like an 11 and a 13year-old and he's also got some older kids. But the 11 and 13year-old, man, they just they can't stop giggling about gooning. I don't even think they know what it completely means. But they they just call each other it. They make fun of each other for it. And I'm just like sitting there going, "Oh my god, you kids." They know what it means. I mean, from like a from from like an internet perspective, but uh I I think they just kind of use the term just a lot because they think it's funny. All right, I'm going to move this one down too because we need to move in compilers software environments with stack and then so we'll focus on then uh deploying creating and deploying a backup strategy with Bula. So Bula or any other tool, there's a bunch of different tools out there, but I know Bula the best of all the systems that I know. So then let's go down here. We're going to move this down again because monitoring really does come later and user provisioning can come after you've done those things.

[00:49:44]Okay, so that would be user provisioning and uh actually we might have to move this down even further because let's put it let's put it back where it was. There it is. Uh we want to um let's do uh run and run common.

[00:50:08]Let's not call them common. Let's say run benchmark uh and acceptance testing against the system. Now, I do quite a bit of acceptance tests where I am at work and uh the way that I run things and build things has to be accepted by the government. So, we write formal acceptance testing plans and I I will put some of those in there. Um I may even pull a couple of those and give examples of those for just in the prologue group anyways because um benchmark and accepting acceptance testing is so huge with uh the government for sure because it's a formalized process. But also this this is the thing I ran into because some of you guys know I ran my own business from like 2006 to 2008 around the market crash. I went back to work for for big big companies again. Started working at Loheed Martin and doing engineering there. But when I ran my own business, there was there's a major problem when you do contracting. And if you guys haven't done contracting, you won't know this, but you you'll probably understand it right away if you have done contracting. when when does the project finish and you get paid versus when it starts to become kind of like the support. So when when is the discrete project that you're doing over and and this is a huge one like you need to have right you need to have upfront with any project you do with any um either fixed or cost plus type contract a set of things that you look at at the beginning and everybody agrees on that we will check these five boxes or 10 boxes or 100 boxes and when this is done the system is accepted and when that part when when that happens you end the deployment phase of whatever you're doing right so that's kind of like a life cycle um part of this that's big because once you run that and get into acceptance you you flip over into like more of the operations and monitoring of the system which then I guess will make more sense with some of this deploying a monitoring stack and that kind of stuff.

[00:51:59]All right. Um let's say and then and then after that we'll we'll allow users in and then the last thing will be um let's say users can come in here and we're going to say down here start up the system from bare metal and uh this will be runbooks. So runbooks are things that we tend to give the customer that are around all of the uh things that they need to be able to do in the environment, right?

[00:52:31]Like like it's kind of like a big manual that they pull out and look through all of the things. Uh I have a bunch of examples of those internally. I don't have anything external that I can show you guys, but probably the closest thing to runbooks that I could show you uh outside of like a company setting would be kind of like our labs. So, if I were to show you here in our lab environment, if I went to one of the labs here, table of contents. Let's go. Let's just go to firewalls, right? We'll go look at the firewall labs now. Let's give this a second because it's about to roll over.

[00:53:10]Maybe not. Um, and then here's here's stuff that's going on with the labs or or whatever is going on with this also.

[00:53:19]Yeah. Yeah. Yeah. So, so you know what are you doing in an environment? How are you troubleshooting things? This starts to become uh the runbooks, a lot of procedures come from those and get fed up into different, you know, whatever tools you have out there in the world, that kind of stuff. But a runbook tends to be how do you run the system? How do you operate it from like scratch? So, there's all that stuff out there. You guys can play with that all you want.

[00:53:42]All right. Um, let me pull that back up.

[00:53:46]We'll leave that up spinning.

[00:53:49]What am I doing then? I'm pulling up uh start bare metal and then I would say this last part will be troubleshooting user issues or or system issues, right? So we may do Yeah, that's fine. All right. So now if we have everything, let's see if we have everything. We have the components as they apply to NIST. We have the network devices and configurations, right? Uh update servers, deploy werewolf and stateless node configurations. Then stig baselines and hardening systems. Yes. Storage types.

[00:54:30]Deploy slurm as a job scheduler. Operate and troubleshoot slurm or your job scheduler in general. Compiler software environments. Uh do I have any?

[00:54:43]Ah so then we have deploying a monitoring stack um creating um alerting and um let's say system threshold or monitor let's say monitoring thresholds and alerting. That's going to be a big a big one because you need to to monitor everything and then also set up your alerting. Um, other major thing would I really want in here? Let's go look at I've got other other things that we were talking about when we were doing this yesterday. So, my my outline from our my my talk on this from before is this. So, We'll probably do let's do user provisioning and then we'll focus on thinking secure. So user access and provisioning environments. Okay.

[00:55:58]Put this one here. And then we're going to say securing the connection uh the air gap.

[00:56:11]And this will be one of my favorite topics. And some of you probably already are like, "Oh god, Scott's going to talk about bastions again, right? Bastion nodes and how do we build them, right?

[00:56:20]Because that's something I kind of specialize in um for government projects and that kind of stuff. So, this is this is a pretty good pretty good hashed out kind of uh 32 um sessions if we were just doing like talking sessions. And then if we were going to apply the lab time to this as well, uh lab time would be most likely done then it would be like lunchtime.

[00:56:47]people would be like working through lunch and then also like take-home tasks because uh you you know you've got to fit in I mean there's going to be a lot of a lot of lab time because most of this time here would be spent lecturing right like talking through the problems and kind of showing them what's going on inside of the system.

[00:57:06]So that's not bad.

[00:57:17]And then I would say individually but not necessarily in a group sessions.

[00:57:29]I would say courses also there'll be 32 sessions.

[00:57:42]this uh modality. So modality is like the way you you teach a thing, right?

[00:57:46]This modality will require uh ex uh uh most of the lab uh you know we won't put that here.

[00:58:00]We we need to put it somewhere. But basically this modality will require the user to do because when you look at this right if this is 32 sessions but you come back here to how we had originally how I had originally planned it from hours it's you know going to be uh 50 plus theory hours between me lecturing and people doing stuff uh you know to study and then also 70 plus ab uh lab hours. So you're going to have to be spending a lot of time doing labs um to get to a total of 120 contact hours.

[00:58:32]This will probably be Yeah. I mean, it's just it's it's gonna be a lot for for anybody to try to chew this up in a in a week. I don't think anybody could prep this and and get ready for this in a week. I don't think it would be possible. There there's just it's too big an elephant to try to eat in that amount of time. So, um yeah, that's not bad though. That's not that's not bad for etching out the the starting pieces of it. And then I can start building to each one of these from uh my my lesson plans, right? So, if you guys have ever seen how I build out my little lesson plans, they they look a little bit like this, but they're more focused on what each thing I'm talking about is going to be. And uh they have the references, they have the other stuff. If you wanted to see one of those, they they typically I call them worksheets often. Um just do like a unit one worksheet.

[00:59:28]looks something like this typically, right? You've got some instructions, some uh discussion questions, some definitions, notes and links and stuff that I think are important, and then the labs and then some other stuff. This one will be a little different because um I will probably have trying to think as as I'm doing this.

[00:59:52]Some of these will have Oh, there'll be definitely more labs. It won't just be one lab per there will be multiple labs. Um yeah, I'll have to figure out how we're going to do that. So because it's going to it's going to follow this methodology right like explore. So it's all the stuff up front that you need to know and and go read about and study. Then the practice will be in the labs and then the applying will be kind of taking those labs and turning them into like real world scenarios, right? Like that kind of stuff. So all right. Um so there's that. We got that that kind of going. I think that looks good. But I might kick this over as like my my initial setup.

[01:00:46]I might say might say this class.

[01:01:03]Okay.

[01:01:22]to to this modality will require the user to work in the lab environment.

[01:01:30]Um almost 50our boot camp. Um 50 hour boot camp uh 50 hour style training over one week labs uh sessions will be broken down by morning afternoon.

[01:02:38]I may have to move, you know what? I might have to move this here and then put this here.

[01:02:47]Continue my bullets.

[01:02:50]Set numbering value at two. Right. And then I might have to do this, but modify this. Well, oh man, that's going to be a hassle. Um session will be broken down by morning afternoon labs will be labs will be um door. Sorry, sometimes I got to yell for my wife because she doesn't hear the door. I heard the doorbell. Um I'm upstairs but I I'm right above where the door is so I can hear it. Well, right up the stairs from it anyways. I'm not right above it. Labs will be uh completed during the lunch sessions as well as uh at night after the lecture at night as well as labs will be completed during the lunch session as well as uh after hours for the learners to all tasks. Again, I I don't I don't even think I could fit this into 50 hours if I wanted to, but we're going to we're going to give it the whole college try.

[01:04:14]Um and uh yeah, I think I think that's I think that's getting there. So, now I got to take this and come down here.

[01:04:27]Deploy that. And then now I got to modify this to be this. This will have to collapse down into only uh uh basically if it's morning and night, five days of the week, then I get 10 I get 10 of these to do. Um but but obviously more things will be in each one. So um so let's do like this. So, it'll be one, two, three, four. Let's Can I pull these and put them here? Nope.

[01:05:07]Nope. Might not be able to do that.

[01:05:12]Let's try this.

[01:05:28]Let's say day one morning.

[01:05:43]Yeah. I don't know if I like that.

[01:05:47]We we'll play with that later, right, four, five. So this will be day two, day two, day three, three, day four, day five, day five. All right. And then we'll because some of these I just I just know are going to have large amounts of uh time involved with them.

[01:06:29]Uh storage can be day two afternoon.

[01:06:36]Slurm will probably be day three.

[01:06:44]Let's do that. And then compilers and creating backup.

[01:06:54]This will probably be in the same. Oops.

[01:07:02]It's weird. It's like if I only get the text, then I can do that. But if not, there we go. Um, run benchmark acceptance testing.

[01:07:18]Yep.

[01:07:19]This one's going to need its own time.

[01:07:26]And then did I Oh, I was going to say I'm definitely miss I'm definitely missing some. I'm like, where the hell did all these other ones go? Oh, yeah. I need I need to compress some of these down then. They're not all going to fit.

[01:07:39]Yeah, I was like the last one should be troubleshooting.

[01:07:42]Okay. Well, that's uh All right. You got to move them all up. Um you join later. Is it a commercial or free course? Um I'm going to do a free version of it on um for everybody in the Discord, right? So, in our professional links user group. Um and honestly, I do that often to like test things and practice things and make sure they're good. But this will also be when this is officially done, it'll probably be a professional course being offered probably by my company. So, I'm doing a lot of this stuff where I'm etching stuff out on the outside. Uh, but that'll be um when it when it ends up being like an offering as we're like training people in our actual environment and it'll be in an actual HPC environment that we have that we use as our lab. Um, it'll probably be a fair amount of um work. Also, another thing I've been So, I've been planning on doing this for a while, but a lot of this stuff we can set up and do in the cloud. Not the werewolf stuff necessarily, but um showing people how to do this out in the cloud is also of some value because um you know, Noah is mostly moving up into the cloud. Um other government stuff is mostly moving out into the cloud. So, that's kind of uh it's kind of going all right. So, I gota I got to compress these down real quick. Um, let's say operate and troubleshoot with slurm. It's probably going to move up into here and then these two can probably be day three afternoon then, right?

[01:09:14]Uh, and I'm I'm just looking at this going I I think some of these like stigs are going to take a long time. Deployment won't take as much time, but maybe people can also use that time in the afternoon to catch up.

[01:09:28]So BCula is going to take a long time.

[01:09:30]Geez, all these are going to take a long time. This is this is not like a one hour thing. This is like a lot of stuff.

[01:09:37]Um run benchmark testing against system that can also have this.

[01:09:45]And then we would go yeah down into monitoring and alerting the start of the next day.

[01:09:51]Start getting users in there that same morning.

[01:09:58]air gap nodes and bastions and catching up with other labs that need to be done and then in the afternoon.

[01:10:08]Maybe this one could also have that in the morning and then the end of the day can just be troubleshooting and and playing with anything else we need.

[01:10:16]Yeah. Yeah, it looks that's good enough for government work, right?

[01:10:21]often say that, but I think that's also shift cells all the way up because we don't need them. All right, so that's also This is This is a nightmare. Move this over a little bit. I want those I want those a little more cleaned up looking.

[01:10:40]Um, we might do this just so things look nicer.

[01:10:53]cuz then the then the page will look a little nicer.

[01:11:08]How's that? That's not the worst thing I've ever seen. Um that's at least this is at least enough to kind of get this proposal going, right? Like um I got asked to do this yesterday by one of the managers at work and um I told him I I had like a basic little idea for it, but I didn't have paragraph line break before page break before. Yeah. Boom. There we go. All right. Then then everything mostly fits.

[01:11:43]I need to I need to update this, right?

[01:11:45]Um and then it'll be it'll be company whatever company we're doing HPC administration engineering for the enterprise. Um engineering comes before typically and administration you build before you run things, right?

[01:12:15]Um, yeah, I think that's a pretty good start. I think it's got all the pieces that we were kind of talking about here, right? This is this is my my other talk that I gave many years ago and then I re-updated it last year for high performance compute, right? Like like what is high performance compute? How do we do things? Um how does this stuff kind of work? Um different uh networking that's kind of in there.

[01:12:43]shared storage and different type of tools that are out there.

[01:12:49]Oh, I need to put in all kinds of stuff to this.

[01:12:53]Talk about spec.

[01:13:03]Yeah, I gota I got to update this talk too again. I got more stuff to add in there, too. Uh, that one's on on YouTube if anybody needs that one. I've talked through that one for hours before. So, if anybody needs that to kind of see if they want to get in there. Um, also I had uh one of the things that we were doing the other day.

[01:13:23]I thought it was right here, but I don't know. I don't know where it is.

[01:13:30]Uh, yeah, this is part of the the user training stuff that I had written up before. So, this is I need this training outline for some of my stuff.

[01:13:44]This came this came from another actually, you know, I'm going to leave that there. I need this um I need this for another for another training that I'm doing, but I also kind of need it for this one as well. What else I'm going to say?

[01:14:04]All right. General training ideas.

[01:14:08]Now over here as well in this where I talk about instructional methods design.

[01:14:16]I might put this.

[01:14:19]He might get his own little page break too. Page break before. Boom. Course plan. Yep. Um, that should be like that.

[01:14:34]Designed to promote learner centered activities and support development of fundamental HPC skills in a Linux environment.

[01:14:41]The uh the wording of that might need to work. Of course, utilize individual group learning activity. We're not going to really have group learning activities in this um I'm just going to say individual learning activities performance-driven assignments, problem based cases and discussions. Uh we won't have discussions. Um not not in the way that I normally do discussions, right? Like if you guys have ever been in the the Discord, um I constantly uh it's funny. I make you guys do discussion posts and it's not cuz I want to, you know, you know, laugh at what people are saying or whatever, but it's because the we we talk about things in the concept of like the marketplace of ideas and and and if your ideas around a thing prove out and um and and uh let me think, let's back up a second.

[01:15:38]argumentation is a very important part of being a um academic or or learning something right and what the idea is that like you have an idea so so I give you some task right like let's just look at 91 discussion post you get a ticket about a problem with containers one of the engineers trying to move his container up uh to a dev environment shared server right it works on his system but now he's trying to move it up and he's having a problem and he gives you this this this thing to look at right some some BS that's going on and actually I really should have pulled that without all the red lines, but whatever. And then people had to come in here and they had to figure out what they would do and what they would troubleshoot. And then people would not necessarily correct them, but say, "Well, I thought about it this way and I thought about it that way." And then if you disagree with somebody who talks, you know, to you about your your problem or your assessment, then maybe they align you with something that you hadn't thought about. Or maybe you kind of like run into each other and you're like, "No, I don't agree with what you And then now again this is like what we call marketplace of ideas. You you've got to be able to describe especially as an engineer why you came to the the places that you are why you think the way that you do about the thing right and um and then again it's it's not uh in academic world argumentation it can be violent and angry but it doesn't have to be it can be done with a a correct way. So, a lot of the guys came in here and they did it, you know, uh much more like, hey, I agree with you and I work on this and this. Um, but this is how we do the courses, right? If you ever go back and look through all the old times that we've run the course, we do one of these each time, but like automation unit 11, what did that look like? Yeah. Oh, yeah.

[01:17:15]This was Oh, bringing up an old old wound here. Like, nobody did them, right? So, like then then I just kind of stopped doing the course because nobody was doing them and I'm like, it. I mean, I can't can't make people do stuff if they don't want to do it, right? Yeah. Look at this.

[01:17:30]There's like one one guy was still doing it by the end. Um but but for the most part, people weren't doing it. So, what can you do? What can you do? This one This one will be people who want to do it because they're paying for it.

[01:17:43]Well, not they're not going to be paying me. They're going to be paying my company, but the company I work for, which is my company, you know. I I consider them uh a good company.

[01:17:56]All right. Of course, you be studying individually and within a group of your peers. That's true. You will probably be working with a group of your peers primarily in a lab environment. As you work on course deliverables, you're encouraged to share ideas with your peers and instructor. Work collaboratively on project and team.

[01:18:10]There's no team assignments. Uh there's no projects either. Um labs raise critical questions and provide constructive feedback. Uh yeah, I think that's all I think that's all pretty good. So, let's I got to read through.

[01:18:27]You guys know some of you guys that have been around know that I'm I'm kind of insane. I got to do things like 30 times to make like this is one of the reasons I'm a good engineer and and I'm I'm often used to build servers for environments because that's that's like a main thing that I focus on. Um is because I'm very uh I don't like using the term neurotic with myself, but I mean I'm definitely more control freakish than than uh than the average engineer I'd even say. So I have to do things multiple times to make sure they're perfect so that in in a normal environment they they'll behave the way that I expect. So um HPC engineering okay good course descriptive uh will address how to design, deploy and administer HPC for modern compute applications. Okay the course will explore HPC architecture from a NIST 800-223 standpoint. That's true. Learner will complete theory and hands-on exercises to gain mastery of the subject matter. Yeah. Uh, yep. Theory and hands-on exercise.

[01:19:23]That's true. Okay. Um, pre prerequisites, knowledge of Linux command line, I would say.

[01:19:33]Oh, but the but typically when we talk about prerequisites, we mean a specific body of knowledge that you know coming into this like like if I wanted you to have CCNA study or or some specific courses previous to this like in a college sense, but I don't really have any of that stuff here. Hey, how's it going Kilobyte? We are just just putting stuff together. I'm just reading through stuff and you know it's it's not it's not super glamorous. It's just something I got to prep for like a larger project that I got asked to do at work. So, there's that. Um, no. So, so the user, so this that's actually a good point. Um, I do training for users of HPC systems as well. Like all of our Noah users. Um, so that's a National Oceanographic Atmospheric Agency, NOAA. I I run the training for the user training on that system as well. Uh well within my within my subprogram of no and not for all of it this though no so this is going to be for our HPC engineers the people who build new high performance compute systems so that's why when you look down here um my user environment section of it is is very small we got to give users access and provision to the environment but we don't need uh but we don't need to give uh like like there's not a lot of time I I have another course that I've that I've built for for users where I I show the users around the environment. I show them like the architecture and that kind of stuff.

[01:21:02]But no, this is for our engineers and this is for uh internally to my company.

[01:21:06]So that's why I'm showing you guys like the outline of it as I put it together.

[01:21:09]Um yeah. Yeah. So let's let's keep going with this. Um it'll be about 120 hours total and that's theory hours which are mostly when we're meeting and then 70 lab hours which you're doing on your own time um or or additional time. Um we'll try to cram that in on the boot camp but again there's no way in 50 hours that we could fit 120 hours of actual you know work time when 70 hours of lab time is probably more but we can at least crash through it and do as as much of it as we can. Um course summary we worked through this yesterday but uh HPC architecture these the major instructional areas are like the highlevel things that we're looking at right and they don't really there's no action plan there's nothing to really do with it it's just what we're talking about right and I do need to monitoring I do need to kind of make this um and uh and let's say monitoring ing and alerting because those are both things that are are important there. And then now course objectives. These are the things like this is like the rubber meets the road.

[01:22:20]Like these are the things that when we're done I want people to be able to do from the course. Hey, how's it going Staint Device? Hopefully this stuff's useful to you. Um course objectives. These are the things that we um you know if if you were to you know go put it on your resume later that you would take in this course. I tell people all the time, if you say, "Oh, I went to Devry and I took a couple of courses on on networking and I feel pretty good about networking." That doesn't play very well on a resume. But if you can say, "I know how to deploy and update servers in a manage provisioning environment using a tool called Werewolf and and you put that line on your resume, right?

[01:22:55]Even if you don't have a lot of other experience on your resume, like it's pretty bare, but you did a course." I would rather say uh instead of saying I took an HPC course with some named Scott that you know you don't know who I am unless you're in the HPC world and then some people there know who I am. But what would be better I think is to have these course objectives. These are actionoriented right. They're they're things that we actually need to know how to do, right?

[01:23:18]Yeah. Yeah. Um Yeah. Like I mean my name's only going to get you some places on people that like know who know who I am or have worked with me before. That's the only that's the only place my my name carries a whole lot of weight. And I would say in in high performance compute circles and security circles, it's probably a lot more than other places. But, you know, it's it's a big pond. So, there's a lot of people that, you know, are more popular and that kind of stuff. And I don't really care about the popularity about it. I care about being able to to build stuff and show other people how to build stuff. Let's add this to the dictionary so it stops showing up. And also werewolf has already been added to my dictionary even though it's spelled wrong. That is the way it's spelled professionally. So that's the way it's named the name of the tool is all right. So we have all that. See if there's anything there. I think those are all fine. We could tweak them later.

[01:24:08]Come down. Uh I got to I got to modify this later, but it's internet OpenVPN certificate and then get into our P pfsenseVPN concentrator and then go back into each of your different lab environments. Right. So that'll be red line access to the HPC environment. Uh instructional methods. This course is designed to promote learner centered activities and support development of fundamental HPC skills in a Linux environment. That should be capitalized.

[01:24:31]Um the course utilizes individual learning activities, performance-driven assignments, and problem based cases. So yeah, it'll be individual learning activities, performance-driven assignments, which will be the labs, and then problem based cases, which will also be part of the labs. So that'll be good. These methods focus on building engaging learning experience conducive to developing uh development of critical knowledge and skills that can be effectively applied in professional context. Cool. Uh explore, practice, apply. Acquisition path will be integrated into this course. Explore, practice, apply. This is how you gain competency, right? I have a bunch of other ways like when I talk to you guys about how you learn stuff. I'm not going to worry about that in the course. We're just going to go over this real real quick. Class size is 40 to 60 learners.

[01:25:14]Um, this may be smaller because uh this will be probably in our environment it'll probably be like five to 20 learners and and we're only going to have a certain amount of capacity as well in our in our lab environment because I'm going to have to carve uh a fair number of servers down. I also have a fair amount of uh usage of my own high performance compute. Like my my lab environment can function as a high performance compute. Uh but I'll have to tool a lot of stuff over to get it to work. um that way. So, there's definitely some some stuff for us to think about there. All right. Um class schedule. This is what we just spent all that time on, but class will meet during the week for brown bag sessions. That would be if we do a 16week course. Um, let's call this 16 16week course. And then down here we say uh 50 hour. So, one week boot camp and I'd say that that would probably be the one that would be like if we did this commercially, that's probably the way it would look is just like a one week and then I'll just, you know, come off my other project, which it's funny, the project that I'm on is um it's still very critical. It's a production system for the US government.

[01:26:47]It's very important. However, it is on the down side of its deployment. So, we finish out the the work on that in 2028, right? When when it decommissions, but a lot of times they still get expanded for for a couple of uh couple months uh or or even a couple years. It'll it'll be expanded out. So, I might still be working on that until 2030 or whatever.

[01:27:06]Um and then I don't know, maybe I'm I'm retiring by then. Who knows? Um technically I could retire anytime I want, but I I I like I I enjoy working too much. So, I enjoy doing systems work too much to to ever do to ever actually stop working. Um, but anyways, this is the this is part of it. Let's make sure we didn't lose anything. So, I did all of these. Let's um we back this out a little bit so we can see the whole page. But we have components of HPC network devices. We've got deploy werewolf. Then we've got our stig baseline storage types. Deploy slurm as a job scheduler. Operate and troubleshoot slurm compilers and software environments with spack. Creating and deploying backup strategy with bula. So that's good. Run benchmark deploy monitoring stack alloy prometheus graphana. Um monitor thresholds and alerting user access and provisioning environments. Um we need to add probably mining stack with holy promethus graphana.

[01:28:17]Um there's another part of this that I would want to include which is our sys log.

[01:28:33]Although our sys log doesn't fit into this specifically. So, I'm going to say let's go RIS log like that. And then we'll move this over just a little bit.

[01:28:46]And then right here, we'll say deploy monitoring stack or log.

[01:29:00]I'd like to fit that on one line if I could. Um, I can do this. I'll cheat a little bit.

[01:29:13]I'll make it one little size smaller, but I'll also do that. Hold on. What is that? Uh, nine. Make these all nine as well.

[01:29:24]Right. And then we'll do it up here as well for the whole table. Uh, nine.

[01:29:32]I don't want to do that.

[01:29:34]Make these smaller.

[01:29:37]Nine.

[01:29:40]Make these all nine as well. There we go. Yeah. All right. They're already nine. Good. Good. Good. Good. Hooked up WRF systems. I think that's all high comput. What is a WRF? I don't know what that is. Um, put this up front.

[01:29:57]I don't know what that is.

[01:30:00]Weather research of Oh, yeah. Yeah.

[01:30:02]Yeah, the back end all the stuff I work for NSEP program within Noah and Yeah.

[01:30:06]Yeah.

[01:30:08]Uh if you've never seen like critical weather days and stuff, this is like what is critical weather day in the US right now. This is this is something I I directly support all of uh all of the backend systems for all this stuff. It's actually funny. They say there's no critical or enhanced car. So there's two types of days that they can be ECEs or CWDs. So that's the type of days. Um, I thought we had them for Oh, I thought we had them for a while. I thought we were in critical.

[01:30:40]Huh. Maybe I didn't hear that. I have a standup morning uh standup meeting every morning where I hear all that stuff and uh I thought we were in that right now.

[01:30:48]But but again, I'm not telling you anything that's not reported out to the world. Everything's on the web page, right? So you can know exactly like if you wanted to know if there was any critical weather days like if you were a shipping company you would absolutely want to know this. You would want to know like where my stuff is, you know, going to be rerouted because of hail or something like that or thunderstorms or whatever like all that kind of stuff. So hey uh D Pirate Roberts, how's it going?

[01:31:11]If that's a reference to the Dread Pirate Roberts, um that's awesome. I actually just made one of my friends watch that movie the other day. He had never seen it. Uh he had never seen it because he's only 26 and that movie is older than him. Um, but that's actually funny. I used to, uh, I used to terrify my college students, you know, almost 20 years ago now in college classes where I can do, you know, I would just start talking through the lines of that movie because I pretty much know the whole damn movie.

[01:31:38]If you're talking about if if D private Roberts comes from that. Uh, if not, then just ignore me. I don't know what I'm talking about. Um, all right. Do we have Okay, so that's 50 hour boot camp type thing. There we go. Dark web hopper. Okay. Okay. then that's probably a different uh or hacker. It's probably a different uh thing than I was thinking. So uh The Dread Pirate Roberts comes from the Princess Bride if you guys don't know. That's kind of a funny little uh funny little uh movie about uh true love as it were. It's kind of a kind of funny sword fights and all that good stuff. Let's move this up here because this can be here. And then what I'm going to do is I'm just going to set this to always start on its own line. I don't always do this because it's kind of annoying to troubleshoot later, but I think once we kind of have this etched out pretty well, I think that uh I think we can do that. Uh where would I publish this course? This course probably won't be published externally. This will probably be internal to the company. Um, but the the there will be a version of this done for the professional Linux users group internal to our discord and that'll probably be done um that'll be in here, right? That'll be in our discord and then that'll also probably mostly be worked out through um Killer Cod Labs.

[01:32:58]So you could do Killer Kota and do all that kind of stuff. So I'll I'll probably take a lot of this as like my outline and then I'll do things out on Killer Kota that people can go and practice. And I was mentioning how I have a lot of these already made in Killer Kota for people to uh play with.

[01:33:14]Now, let me see. Why is this so kind of I need to fix this? Cuz this kind of looks like crap. Uh does it need to be I need to align it better.

[01:33:34]Don't mind the centering part of it as much.

[01:33:39]these can be centered maybe.

[01:33:44]Yeah, that's probably fine. Okay. Um, Killer Code Labs are awesome. Yeah.

[01:33:49]Yeah. I think I think Killer Code is great. I I did most all of my Kubernetes stuff on there. I did a little bit with CodeCloud because back then uh CodeCloud wasn't as big and Mumshad didn't have it like nearly as well laid out and monetized as it is. when you used to buy his Udemy course for like 15 bucks, you would get access to all of CodeCloud with it. So then I I did all the CodeCloud labs very early on, probably 22 or so. Uh I forget I forget when I got my CKA, but it was a while ago, maybe 2023. I think it's I think it just lapsed, so probably 2023 when I got my CKA and CKAD.

[01:34:25]I got both of those in the same month.

[01:34:27]Um, but I worked a lot to get the CKA and then uh the CKAD was just, you know, only 15% more stuff if you ask me. It's pretty much the same study. All right, I'm going to take this. Um, I'll call that pretty good for now. I'll probably uh I'll probably look at that again before I I send that over to the bosses to to look at kind of like my my plan on that outline or whatever. Does this have a dark mode?

[01:34:53]I wonder I've never tried to mess with this in dark mode, but you'd think they'd have one. Yeah. Yeah. But this is all this is all the weather prediction stuff that my my program might the stuff that I work on directly feeds all of this. Well, feeds everything. Everything comes from our our predictions. But um yeah, you got to have access for that.

[01:35:15]Um what is the directive? Here's the weather directive that that defines that where that comes from. Um, and what our goals are, what what are the objectives of of the government. So, if you guys have never messed with any of this stuff, this stuff's all out there on the web, right? This is all I'm not on an open web page here showing all this kind of stuff. Uh, yep. Then there's freedom of information. You could ask, you could request for specific information if you wanted, but everything else is out there. There's also uh APIs where you can go pull uh historical weather data as well. Um now there's there's batches for if you go get large amounts of old weather data um you you have to do batch requests and then they they feed it out to you. But there's that kind of stuff.

[01:36:00]Um what am I using as VM as a cloud or home environment? I do a little bit in the cloud. Most of my stuff that you'll see is in Proxmox.

[01:36:09]So where did my environment just go here?

[01:36:12]Um, most of my home lab is in Proxbox here. Uh, and I just I have most of this running here. I also have some other servers that run a tool called XCPNG, but I've been slowly moving them out of my environment, turning them off to to save power and save power space and cooling, but also because they're older hardware.

[01:36:35]So, a couple of these are still good enough for me to reprovision as Proxmox servers and then I'll run them as well.

[01:36:42]But like like this one here, uh, wait, this one here, he is he is a sad old box. He is uh, he is ready to go. So, so I was five servers, now I'm down to three. And then this one here is going to be one of the ones that goes away because he he's just he's only got 32 gigs of RAM and he is an older Intel uh, Xeon chip. So, probably God, how old is that box? Probably at least 2017 or older. So, almost 10 years old. So, yeah, I run a lot of home lab stuff and then um I have drawings out there on how all of this stuff works uh in the in the Discord for people.

[01:37:23]Uh then the stuff that I do out in the cloud, um I have a few different types of deployments that I do in the cloud. Uh, I use Terraform for that and then I use Antsull to configure it, but I haven't used those in a long time. Um, but that you can still find those out in the Discord if you wanted to see how I built all those out and stuff. So, um, let's pop out of here.

[01:37:49]Let's go over here. Let's see what werewolf's doing.

[01:37:56]we have so werewolf is also where so I use a tool called werewolf that does stateless node provisioning and what that means is um I've got these images that I build on the local machine here and we harden them and we configure them with um anible right so we use anible for that so like how did I harden these environments and I didn't do a whole lot of hardening like I didn't do stig hardening levels with them but I wanted to show people how we kind of touch an environment and do that kind of stuff so I had showed this before this is how we added tools and and added stuff in there. And then all of these nodes, like if I go to like Rocky 5, all the nodes are stateless images. So if I were to let's say just touch temp test file, right?

[01:38:44]Right. So that that file is there and it exists on the box, right? But then I reboot the box, right? And before we reboot him, we'll come over here and we'll watch him reboot. Right. So, just kind of showing the environment in case people aren't super familiar with how Werewolf works.

[01:39:01]So, now we'll watch this box of the console as I kick him over right now.

[01:39:04]So, we're going to kick him over. Here's the console.

[01:39:09]Console's going to boot. He's going to come up. And I know it's a little small, but um we can make it a little bigger, but but the point is he doesn't know who he is. and he went and pulled that information and started an ipixie boot process to find that he was Rocky 5. Now he's downloading his init RAM FS in his kernel and he's going to be able to start and boot up and come back up. And actually in a few seconds he'll it takes only about 18 seconds on the system or so. Should be up right about now. Yep.

[01:39:41]If I cd to temp that file is not there anymore because the file that I created was on a node that doesn't really exist.

[01:39:49]It's it's ephemeral. Everything in there is stateless in that uh it just pulled its clean image. Again, if I want something to be persistent, I need to either go bake it into the image or put it into what we call overlays. And overlays get pushed onto the system later. Uh Jones said, "As a network guy, all my LX's VMs are numbered based on VLAN tag. Uh I see as a Linux guy just do them in order. Yeah. Yeah. Yeah. They just kind of get whatever names and and those were deployed automatically too.

[01:40:15]So they just they inherited the numbers that they got. Um if you guys haven't seen that process that's over here this was uh so like like the hammer nodes the hammer nodes here's the lab servers yl. So that's the definition of those nodes. And then the way I deployed them were from the deploy derp deploy servers here. And then I had to give in the deploy file as part of the the command when I deployed them. And then it it called it and it went and built those nodes and put them in the right vlan and all that kind of stuff. So this loops over the nodes that are defined here, right? And then it puts in the the loop variable item.node.name each time, right? It does item.node.name. name.

[01:41:05]It puts that in for that. The Mac and is there another thing? No. If I had other components and things to put in there, I would do that. Yeah. Yeah. Oh, yeah.

[01:41:15]That just it just happens to be the way I deployed them. Um and then look, Vault had vault had come in there in the middle before I did all the rocky nodes and moved them over. And then one of the rocky nodes is out of place because this was my testing node originally when I was testing to see if I can do this. Hey Juwel, how's it going? Um I am just uh kind of walking people through the environment a little bit. We were doing uh high performance compute stuff. I I kind of put that together today. So So it was based on this for people that were missing it. We were talking through these major instructional areas. Uh these are the things I'm going to talk about and then these are the things that I want people to know how to do when we're done with all of this.

[01:41:57]Um, there's going to be so I mean there's going to be so much lab there's going to be so much talking and training people, but then there's also going to be like so much hands-on lab with this that it's going to be going to be a insane amount of uh sane amount of stuff that I'm doing uh for training and and then again my my company's either going to have me run the training or I'm just going to build the training and then we may try to uh record it and then you know uh do it either as uh live lessons or maybe some online stuff for for cuz cuz you know all kinds of companies come to us for running high performance comput environments and seeing kind of like the sauce like kind of how we do it. Um and then my company's been very embedded with Noah for about 30 over 30 years now actually. Um if you take into the time that before they started the company Don and Carolyn they were working at Noah before that. So we're we're pretty embedded with them, but then we also um we we branched out to a lot of other high performance comput stuff for different customers, too. So um I need to validate your systems connecting to the VPN. Connect to the bastions. I need to systems you land on. Verify your environment and home file system.

[01:43:22]Uh let's do V. Users have to be able to um or uh check user space quotas.

[01:43:35]Uh pi check uh project quotas.

[01:43:47]I got a b I got a bunch more that I probably got to add to this and then clean up.

[01:43:52]I got to do that tonight, too, because I gota I got to give the training one more time to my internal team before I go start doing it for users. So, this is a different training project. So, you say, "How many training projects are you on, Scott?" I don't know. I've lost count over the years of how many times I've trained people on different stuff. But, um, high performance computers is what I've been doing professionally for, you know, the last 20 years of being a Linux guy for about 25 years now. So, um, almost 25 years as a Linux guy. So trying to uh trying to get more people into this is the goal, right? That's that's the stated goal of our group, too. So if you guys go to our Discord, this is um you know, the goal of it is to get people doing Linux. We we all do this stuff for fun, right? We're all trying to get better at it and and just you know, play with it, but um some of you want to do it professionally. And if you do want to do it professionally, that's uh that's kind of what our group is trying to do, right? And we're we're almost to 5,000 people. So, I think we're at like 4,970.

[01:44:52]Oh, I can check I can show you guys. We got a little bot that'll tell us. Uh, where are we at here? So, any any new people that popped in?

[01:45:01]Tell you guys exactly where we're at.

[01:45:02]4974. So, we're 26 people away from 5,000. And I know the bot's going to do something crazy when when that fills up.

[01:45:08]And then it's funny because people leave and come back to like keep triggering it. So, I actually really hope that it it goes over while I'm streaming so that it just goes over and stays over. I don't want it to be going back and forth like that. The bot doesn't really have Well, fisherman might have added So, Fisherman Guy Bro runs the fisherman guy bot. I know. Very, very clever naming.

[01:45:27]Um, I think he might have added some throttling in there so it doesn't keep going again and again, but we'll see.

[01:45:33]We'll see what it does. Uh, reminds you of Nick. Yeah. So, New World Andy, that's interesting that you say that. I missed that, but reminds you of Nixos.

[01:45:39]It's It's interesting. Uh, Nyx OS is completely immutable. So with a with an immutable system, it would be like um so let's let's compare them because that is a good observation with with an immutable system. It's kind of like I give you a little pen. So here's my little notebook. I give you I give you a little a little pen here and then I make you and I tell you to go ahead and write on some paper, right? And if you can write on paper and change it, then the paper itself is not immutable. But if I give you this same pen and I give you some granite and I go, "Okay, go carve into that granite, this pen is going to break well before that granite ever changes."

[01:46:16]So with immutable systems like Nyx OS and some of these other there's there's ways that you can make immutable systems. We do it a lot with containers when you and they we we make them uh read only at the root file system, right? That's a common way. And then sometimes we have attached file systems that can still be moved because volumes and uh uh storage and and things like logs have to be written. So you can't have something completely immutable.

[01:46:41]Well, anyways, where I'm going with that is um immutable systems I can't change in any way. But these systems you can you can change. So these are a little different than immutable. With stateless nodes, I can change whatever I want on this box right now. Right? I can I can um mount opt if I want, but it's busy.

[01:46:56]But I can force an unmount, right? And then df-h and it's gone.

[01:47:01]Whereas with like an immutable system, you typically can't make any changes to the system. It's it's it's set. It's it's read only, right? This stuff, this is not readon. However, I think of I think of immutable systems kind of like you're taking a little marker and trying to like carve into granite. You can't do it. You can't really change anything in there. But with a immutable system uh that is immutable in a stateless system, you can change things around. You can move them whatever you want. But I think of it more as like a trap door. It falls and all the configurations fall off it and it comes up clean every time. So I just modified the system.

[01:47:37]We reboot it. We give him a little bit of time. We go back in there. He'll be completely configured again with opt back running. Uh he'll pop back up over here in the monitoring here. Uh Rocky 5 will drop off of here when this refreshes because he's not responding.

[01:47:52]But in a few seconds, about 20, he'll be back up.

[01:47:57]We'll be able to go into him. Let's see.

[01:48:02]Not quite up yet. Probably now though.

[01:48:06]Yeah, there he is.

[01:48:08]And there he's got that opt back, right?

[01:48:10]That directory is back.

[01:48:13]So anyways, yeah, he so so so stateless images you can modify. I could put like I could put a web server on here. dnf-y install httpd right I can run a web server on this box but when he goes down and he comes back up he's clean nothing will be there which is and the reason I did the labs that way because you might say why didn't you you know why didn't you do something immutable and people can do because I need I need people to be in an environment that's like a real like you're a real Linux admin you're you're root in the environment if you guys hop into the lab environment you're root you can do anything you want against that box you can kill that box if you want right you have all the power but you I I can't you can't train somebody without that. Like how how can I tell you, hey, we're going to learn how to be Linux admins if I don't make you a Linux admin in the environment, right? So that's kind of the way that I did it. So now, right, we got HPD running. Oh, we don't have it running yet.

[01:49:09]Right, we got it running. We see port 80. We can curl on port 80, but it's listening for anybody, right? But it's not actually listening because systemcttl status firewall D. We're not uh Oh, no. We're not running firewall on these, but I think I turned firewall off for these boxes.

[01:49:29]Other ones do have firewall on for for other reasons, but um yeah, he can see he can see his own uh web page. His web page is up. Everything's running.

[01:49:42]Uh then we talk about you know we in the admin course we talk about well would this system would Rocky 5 come back up with Apache running and all the information you need is right there in front of you if you haven't ever done it but there's it's there's a little bit of a trick question there because we're in a lab environment where it goes down and comes back up without it. So the answer is no. But if it were just a regular server would this server start Apache when it comes back up? And the answer to that is also no, right? I'm not trying to trick anybody. I'm just telling you guys um it is also no because right here it is disabled. So we could enable it, right? Systemct ctl enable HPD and then in a normal functioning server this would be persistent and it would come up on the next boot. So enabled or disabled is what's the persistence of a service right with through reboots and um whether it's active or not now is whether it's started or stopped right so those are two different settings that you often have to show people you know if you've never used any service type stuff you've never back in the day it was service now it's systemctl um but if I reboot this box will it come back up and the answer is no because the the box is just a a stateless image and it's not going to have that in there anymore if I wanted to bake that into the image I could and then I could have HTTPD up on all these servers all the time and we could run them as like damn vulnerable web clients, right? DWVBs or whatever they are. DAM DAM DV WBS I think. DAM vulnerable web servers.

[01:51:16]DVWS's maybe is what it is. Yeah, but we used to call them damn vulnerable web services and then we would go and hammer on them and and use our tools against them and see what we could find out.

[01:51:25]Right.

[01:51:27]That box should be back up.

[01:51:36]Not going to have anything with HTTPD running. If we just look at SS- NULP, there's no port 80 there, right? There's nothing going on with that box anymore.

[01:51:45]I could have also done it like this, I guess, if if we really care. Put an E there. Do this, right? And say I want anything HTTPD or port 80. Although there may be the number 80 somewhere else in there and then that would that would not work uh perfectly. But it it gets pretty close to show us that there's nothing running with a web server, right? Nothing uh nothing listening on our local machine for the port web. The only thing on here that is running is the node exporter. So I can curl 127.0.0.1 on port 9100 for metrics, right? Maybe throw that through a head and then I can see the first 10 lines of what it looks like to go and and query my metrics that uh Graphfana gets, right? Prometheus scrapes those from my servers. And then also if I was running the web server, I would see um not the web server, if I was running the firewall, I would see that I have to expose the firewall. So where do I have that on? Uh for sure I have it here.

[01:52:51]Excuse me.

[01:52:55]Let's go grab 9100. So we see there that node exporter is running on this box. We got to switch our environment over to alloy. That's one of the things that we need to do here soon.

[01:53:08]But uh firewall cmd firewall cmd get active zones right and then it's uh what is it?

[01:53:29]Oh man.

[01:53:32]Let's go with what's that command should be get get default zone get active zones get services get zone I think just get zones and then um there's the active zones but I want the where is The where you list zone.

[01:54:18]I think we could just see. Oops. Uh, thought we just could list the zone.

[01:54:29]H I could just list that zone.

[01:54:42]Where's that at?

[01:54:48]It just list services.

[01:54:53]There's the services that are active, but we should be able to see where they're at. Um, I forget the command right now. I go I always got to look it up. I I never bother committing all this to memory.

[01:55:05]Um, you get zone of interface public. But we already know that that one's or or if we know the interface, which is uh I think on this one it's the E.

[01:55:35]They got a weird name.

[01:55:38]Probably could just do we try info zone.

[01:55:47]There we go. That at least shows us then that the Prometheus node exporter is open, right? That's that's allowed through. And then if we go and look for that, we can find that in var lib uh firewall.

[01:56:00]Was it is it not uh var?

[01:56:11]There it is. User lib firewall DN services. And then if I more the Prometheus uh Thought it was Prometheus.

[01:56:29]There it is. Yeah. Then you can see how that's configured to allow that to be listened to, right? So that's how we're allowing that through. So that node exporter is able to be shared through the system and then that's being read. If you're like, well, can you prove that that's being read?

[01:56:46]Yeah. Over here you can see the werewolf node right here. The werewolf node's giving his information to the system. or it's or it's being scraped from Prometheus, right? So, we know that.

[01:56:59]All right. Um, what do we got here?

[01:57:03]It's uh been almost two hours I've been talking. So, I'll probably hop off of here in a little bit. By the way, if you say, "Oh, Scott, could you know where would you go find out more stuff about firewall?" We have all that stuff in our labs, right? Like there's labs.

[01:57:15]Actually, this is the firewall stuff here. So, list all zone public. Yeah, this this would have helped me. That's the command I wanted right there. Right.

[01:57:23]That's what I wanted.

[01:57:25]That would have been that would have been the exact command I was looking for right there. But anyways, this this stuff's all out there for you guys if you want it. Um, and if you say, hey, you know, I always get asked about like what does it take to get into the field.

[01:57:37]I say you just got to start learning Linux, right? You got to start learning it. Got to start planning where you want to go.

[01:57:43]Um, we've got we've got stuff like this, the self- assessment matrix that I think can help you to figure out what you want to do, right? All right. And this stuff's all up in the Discord as well.

[01:57:53]Plus, you can always obviously ask questions and stuff like that in Discord as well if you're interested in getting into the space. Um, are many enterprises using uh Umbu? No. No. Umbu is allowed in the government space. So, I've seen I've seen very little Umbu used.

[01:58:09]Canonicle got it approved probably in like 2020, probably during COVID time.

[01:58:14]Um, sou can be used uh in big government projects but I would say probably I mean well let me back up a 100% of applications or or or high performance compute that I've deployed in the last three years I've deployed like 15 systems that have about 15,000 total um compute nodes. And in those systems every single one of them was either RE or Rocky Linux with probably more of it favoring Rocky nowadays. uh for government systems. So, so to answer that question, the last three years I've probably deployed about 15,000 systems total, like nodes, individual nodes, and the lion share of those have been rocky.

[01:58:55]And then there's been some rail in environments that were specifically is because they were well, I probably shouldn't tell you the exact companies, but big big contractor companies that you guys would know um that use us for their high performance compute building.

[01:59:10]Um, they probably, you know, were the ones, well, they for sure were the ones I know because I built them, but uh they were using Red Hat Enterprise Linux instead of the Rocky uh uh alternative.

[01:59:21]So, anyways, um what else do we got? That's that's pretty much all I got for you guys today. Um I appreciate everybody coming by. If you guys want to message me, I'll be down in the Discord hanging out like I normally do. I I do have more work to do tonight, too. Um so, I need to get to that. And then also I got to go have dinner with the wife. So she's downstairs cutting one of her friends hair. So then when she's done with her um I think we're gonna go get something to eat. But all right, I will talk to you guys later. Let me find somebody to raid over on the Twitch side on the YouTube side. I'll just go ahead and and kill you guys off