Async & Parallel R with {mirai} | Charlie Gao | Data Science Lab

Added: 2026-05-30

[00:00:07]Alright. It's time to announce our lab manager for today.

[00:00:11]We are joined by Charlie.

[00:00:12]Charlie Gao, would you like to introduce yourself?

[00:00:17]Sure. Hi, everyone. Nice to see you all.

[00:00:22]I'm Charlie Gao.

[00:00:23]I'm a member of the open source team at POSIT.

[00:00:26]And, you know, sort of icebreaker, I have on this, Conf T shirt from Seattle in twenty twenty four.

[00:00:34]And as you can see, this is actually a participant's T shirt.

[00:00:37]So Isabella has a green one, one of the, staff T shirts.

[00:00:42]And and so at this the conference two conferences ago, I was a member of the ARC community, and it was really after that conference that I joined POSIT.

[00:00:54]First, I was a contractor and and and then full time.

[00:00:58]So what do I work on?

[00:01:00]Well, work on sort of a number of things across open source.

[00:01:08]So what I've been working on most recently has actually been Quarto.

[00:01:16]So I've been actively involved in Quarti two, which is the new Rust rewrite.

[00:01:23]That has been announced, so I'm free to sort of say that.

[00:01:26]But as part of that, you will all be getting a collaborative editor.

[00:01:33]Yes.

[00:01:34]So that's been, an incredibly exciting project to be working on.

[00:01:41]But apart from that, I'm also the creator and maintainer of a few R packages, Mirai, Nanonac, Secretbase, etcetera.

[00:01:50]And these are all packages that are mostly written in c, actually, for high performance computing and networking.

[00:02:00]So myself, I'm very deep into sort of async, apart from working on Mirai, which is the package I'm probably most well known for, I've also implemented modern async into packages that you all use often like Hida two and Elna.

[00:02:21]I also maintain the yeah.

[00:02:25]I also maintain the later package on behalf of Joan Winston.

[00:02:28]Those are packages that they created very early on, and I've sort of taken over the maintainership as they've focused more on AI related applications.

[00:02:40]Later powers the core event loop in Shiny and other sort of key R packages like Plumber.

[00:02:49]And it is basically a way for different packages to cooperate with each other even though they're running at the same time.

[00:03:00]So as you can see, I'm very much into sort of async, into sort of working on our internals, decode, and also communications.

[00:03:11]So this is like HTTP, WebSockets, and hence my work with Shiny and now, on Quarto and the collaborative editor.

[00:03:23]So today, I'm mostly gonna be focusing on Mirai and how to, use that for parallel and async programming.

[00:03:33]Yes.

[00:03:33]I would love to start with what the heck is async programming, and how is it different from parallel?

[00:03:41]Sure.

[00:03:44]So let me just hit that on the head very, like, straight away so we get rid you know, we cover all the sort of tech technical jargon.

[00:03:54]Ace parallel is basically when you're running, two things at the same time.

[00:03:59]And async is just you're not waiting around while that happens.

[00:04:04]That is as simple as I can put it.

[00:04:07]So let me share my screen, and I give you a concrete example of what I mean by that.

[00:04:16]Yes, please.

[00:04:19]Okay.

[00:04:19]I'm gonna be sharing my whole desktop, and you see my Zoom screen, obviously.

[00:04:25]K.

[00:04:26]Everybody see okay with the size, or does anything need to be zoomed?

[00:04:30]This is now is the time to let us know.

[00:04:33]Brian says he's okay.

[00:04:34]So if I'm if I'm typing stuff like this, is everyone not gonna be okay?

[00:04:41]Let's see. Rest says size is okay.

[00:04:46]I've zoomed this in already.

[00:04:47]So this is We have some thumbs up.

[00:04:49]Okay. If anybody needs us to stop, let us know.

[00:04:52]We'll make it bigger.

[00:04:54]Excellent.

[00:04:55]So this is actually, before I even start with any code, if I just switch to the the package downside for Mirai itself, this might be if I just try and zoom this in.

[00:05:11]Yeah. Okay.

[00:05:14]So Mirai is sort of the main package I'm gonna be talking about today.

[00:05:21]This is a package that I developed about three three to four years ago is when I started work on this.

[00:05:27]So this is much more recent than any other sort of parallel code that you'll see in base R.

[00:05:34]That code has been around for twenty to at at I'm not sure if it's thirty, but it's at least twenty years, and that hasn't changed.

[00:05:42]So this is sort of a much more modern take on how to run our code in parallel and and and async.

[00:05:52]Mirai, is Japanese for future for anyone who's wondering.

[00:05:56]So this is an implementation of future.

[00:05:59]So it's basically a computation that's gonna be ready sometime in the future, and Mirai just is future in Japanese.

[00:06:08]So that's why it's called Mirai.

[00:06:12]And maybe while I'm here on the website, just a digression before I get on to code.

[00:06:22]And this is mirai dot arlib dot org.

[00:06:25]So mirai sorry.

[00:06:27]It's getting annoying.

[00:06:28]So this is just mirai dot rlib dot org.

[00:06:37]You'll see there's this ask deep wiki button, and this is on a few POSIT packages.

[00:06:46]We use this on the Quarto site itself because Quarto is, you know, is very complicated.

[00:06:52]It has a lot of components.

[00:06:53]And it's this is just an easy way, for, users to ask questions.

[00:06:59]So you can click in.

[00:07:02]And this is sort of AI LLM powered.

[00:07:06]There's ways to that you can use this programmatically, but, essentially, you can click in and it sort of it gives you, like, in-depth overview of, like, everything that's within Mirai, like its documentation and its functions.

[00:07:25]But the great thing is you you you drop straight into this sort of chat box that you see here.

[00:07:30]Right?

[00:07:31]And you can ask any question that you have about Mirai.

[00:07:35]So even after this lab is complete, you can just go in and you can ask any question.

[00:07:39]And the great thing about this is you can put in your question in any language, and it will give you the answer in your own language of preference.

[00:07:51]So if I say something like Wait a minute.

[00:08:05]Do you French? Speak French, Charlie?

[00:08:09]Sudden word drop.

[00:08:12]Charlie speaks French.

[00:08:16]No. I haven't really used it since, like, school.

[00:08:19]It's just when I learned it.

[00:08:20]But but you can see, you know, this gives you, like, a nice nice answer.

[00:08:24]It gives you examples, runnable code, etcetera, in the language of your choosing, and this is, like it can be more helpful if you're, you know, in, like, working Japanese or Chinese or something like that.

[00:08:36]It's like it's it's really sort of, you know, powerful, the sort of AI LLM power tools that we get nowadays.

[00:08:47]But that's sort of, like, a nice affordance that's just there on the on the home page for me.

[00:08:55]Right?

[00:08:56]So I'm gonna switch back now to my Positron window.

[00:09:00]And if I just spend, like, a couple of minutes again, I'm just, like, delaying the point where I actually, like, write code.

[00:09:08]You're killing me, Tyler.

[00:09:09]You're killing me.

[00:09:11]No. Just building out the suspense.

[00:09:13]So you can see, like, what I have here is basically, like, the two pane, like, view for Positron.

[00:09:21]And I think this basically gives you a clue that I'm more of a software engineer than a data scientist because I don't have the default data science view.

[00:09:31]And, like, if you're curious about how to get there, it's really simple.

[00:09:37]If you click on this layout button, it's the side by side, and you just enable the sidebar, whatever it's called.

[00:09:45]Yeah. The sidebar.

[00:09:47]And the sidebar gives you all the usual tools, say files, search, GET integration, etcetera.

[00:09:54]And you can see I have these buttons on along the top.

[00:09:56]This is also key to sort of the the whole ergonomics of using Positron.

[00:10:02]And this I I you've probably seen this before because this is you know, I'm sure Libby and Isabella has gone through this, but the default is is this activity bar is the default is sort of down the side, and that sort of, like, for me, takes up a lot of sort of screen real estate.

[00:10:17]So the first thing I do and this is great, Piyos.

[00:10:21]This is sort of remembered.

[00:10:22]You only have to do this once, is to set this to top.

[00:10:27]So activity bar position top, and you're good to go.

[00:10:31]And this is sort of my default setting for all the work I'm sort of usually work on.

[00:10:37]So I have my source files here on the left, and then I will have the R console or I have a terminal sorry.

[00:10:47]I have a terminal session with something I was working on with Claude code earlier. It's not disappearing.

[00:10:56]Okay.

[00:10:57]So very simply, and I have some notes here just like this is like this session is totally unscripted, but this is just some notes of, like, what I definitely want to cover so I don't go completely off topic.

[00:11:11]So, okay, act one. What is Mirai?

[00:11:13]So first things first is launch the package.

[00:11:19]If you haven't got it installed, then just install it either from CRAN, or I would actually encourage you to try and install the development version because there's some nice new features in there that if we have time, I will share with you.

[00:11:38]And it will be the first time I've revealed these significant new features that will be released, hopefully, later this month.

[00:11:46]And to install the install the dev version of Mewray, then you you can just do Packpack and, oops, sorry.

[00:12:02]And you can install it from the GitHub repo r lib slash u r I.

[00:12:15]So will you be using dev version today?

[00:12:19]I will be. Okay.

[00:12:21]And there's no difference from from what I'm showing you unless we go into what specifically I've implemented later on if we have time.

[00:12:32]So if you haven't got it installed, feel free to sort of install it.

[00:12:35]It will pull in the development version of Nanonets as well, which is what underlies Orion Powers or the sort of high performance networking.

[00:12:49]So very quickly.

[00:12:52]So now I've installed it, then I load the package.

[00:12:57]And the main function in Mirai is just Mirai.

[00:13:01]So there's nothing for you to sort of learn.

[00:13:04]Then there aren't sort of there are no options.

[00:13:08]There are no options or environment variables that you can set with Mirai.

[00:13:12]Everything is very explicit.

[00:13:14]The main function is Mirai.

[00:13:16]So it's really designed so you don't have to sort of remember anything.

[00:13:21]So to use Mirai, you can just say oh, I'm sorry.

[00:13:26]Let me just start a new session because I just installed it.

[00:13:30]Sometimes it doesn't like me running things.

[00:13:34]So if I just if I just put something in Mirai and Mirai basically just runs whatever expression in another process.

[00:13:46]Okay.

[00:13:49]So you can put anything here.

[00:13:50]So for example and any any expression.

[00:13:54]So anything you can put in these brackets.

[00:13:59]So we can say, like, sleep.

[00:14:02]So start sleep two and then say, you know, have it return done or something like that.

[00:14:09]Right?

[00:14:10]So when you create a Mirai, this object is a Mirai.

[00:14:15]And you can see, like, it's this is the the the result of this is basically available at data, which is done here.

[00:14:26]Okay.

[00:14:27]This this is too simple. But if we should sorry.

[00:14:31]This is too this is over too quickly.

[00:14:33]So if we have this sort of last ten seconds, then if we look at this object this isn't actually done yet.

[00:14:40]You can see this this sort of brackets here.

[00:14:43]This is just the print method. Right?

[00:14:45]And then what we can do is we can say, is it resolved or not?

[00:14:52]And unresolved. So this is now actually resolved.

[00:14:55]So this is now finished. Right?

[00:14:58]This task is now finished, and then it will be available at data.

[00:15:01]K.

[00:15:04]So this sort of shows you that, basically, when you have this on this is happening in another process, then we're free in our own session to do other tasks.

[00:15:19]And that is essentially what async is.

[00:15:21]So This is where I would love to hop in and ask Lauren's question, which is what was the motivation or the need for developing MRI?

[00:15:35]What problem is this built to solve?

[00:15:37]So I'm imagining as a person who in undergrad and grad school had to leave her computer running overnight to run projects, that this would mean if you have something that's a very, very long process that's going to take a very long time, you can do something else while you're waiting for that to finish.

[00:15:57]Am I right?

[00:15:59]Yes. So there are many ways you can use it.

[00:16:03]One is, as you described, so if you have something like and sort of my background, I say I'm not a data scientist sort of now.

[00:16:13]I did do sort of data science y type things, previously.

[00:16:19]I was actually sort of, building these deep neural nets, actually.

[00:16:24]So I was actually training these neural nets, and then I was actually running influence on them in inference on them.

[00:16:31]Sorry.

[00:16:32]And so I was using Mirai for both of those purposes.

[00:16:35]Right?

[00:16:36]So first of all, if you have something that's sort of ongoing for a long time, you can run them in parallel in background processes and that each of them is self contained.

[00:16:47]So if something sort of errors and goes wrong in one of them, all the others will still progress to completion, hopefully, because they don't interact with each other.

[00:17:00]That's one thing.

[00:17:01]And the second part is really what led me to develop ERI, which is I was actually running inference with one of these neural nets, one of these models.

[00:17:14]Right?

[00:17:15]And I was actually also ingesting real time data.

[00:17:20]And so this was actually financial market data that I was ingesting through an API in real time and trying to run this through this model and then save the inference results in basically a kind of database.

[00:17:36]Right?

[00:17:36]So because this all had to happen in real time, then I couldn't afford to just run things in a sort of a normal loop because if something falls over, then that basically, stops the whole thing.

[00:17:54]So what, Mirai can allow you to do is it can allow you to offload different tasks into different processes so that your main loop can stay reactive.

[00:18:08]So, essentially, your main loop, you're not doing, very much at all in it.

[00:18:13]You're just checking to see if other parts of that you need are all done.

[00:18:18]So whether that's ingesting data or that's writing data, those can be done in separate processes, and you can have basically redundancy and sort of fail fail over in your main loop.

[00:18:31]That is a very sort of high level overview.

[00:18:34]But, hopefully, that makes sense to everyone.

[00:18:38]Well, we got, like, seven more questions based off the answer.

[00:18:44]So I would say that it makes a little bit of sense, but we need more context.

[00:18:49]So do you want to give another give another code example, and then we can hop in with more questions?

[00:18:55]Excellent.

[00:18:56]Let me go through some more of what I wanna show you.

[00:18:59]So this so I was on here.

[00:19:04]Right?

[00:19:05]So we here if so one way we can sort of work with async is we have this thing that's running another process.

[00:19:16]And then in Mirai, the if you do these brackets, right, this is the collect method.

[00:19:23]So this is actually equivalent this is basically collect Mirai. Right?

[00:19:26]This this basically says it's already done, but this will actually wait for the Miraille to complete and then return the results.

[00:19:34]So if I do this again, then this will actually wait.

[00:19:39]So this is basically parallel.

[00:19:43]This is not really async while you're waiting, but this is one way that you can sort of get the results.

[00:19:53]But this is where you can pair Mirai with promises to actually have actions happen as soon as a result is complete without you having to wait for it.

[00:20:06]So this works something like this, and this is basically how Mirai with Shiny works.

[00:20:13]So Shiny, you know, everything is async.

[00:20:16]Your Shiny server serves many different users.

[00:20:22]You can't have that server sort of, like, stop and wait for one particular user to finish.

[00:20:29]So if I if I load up the promises package.

[00:20:40]Yep.

[00:20:40]So what I can do here is so, again, if I say something if I say sleep and then that basically if I just return a value.

[00:21:00]And then how promises work is you can say, you know, once that's done, then do something else with the return value.

[00:21:09]And here, like, an example I like using is just to use the b par function, and it will, like, play a sound.

[00:21:17]So here's you know, this Mirai is gonna run for two two seconds, then it's gonna return two, and then it's gonna call beep ah beep two, which is the second beep sound.

[00:21:32]And I realized, like, you don't hear this. Right?

[00:21:36]No. No. No. You'd have to share your system, Sam.

[00:21:38]We'll trust you that your computer is mute, though.

[00:21:40]So, like, okay.

[00:21:42]I you're gonna have to trust me, but this is basically goes ba ling.

[00:21:48]So this is like it's like a sort of Mario sound.

[00:21:53]Like yeah.

[00:21:54]Like you're getting a coin in Mario.

[00:21:56]Like like yeah. Exactly. It's it's the coin sound.

[00:22:00]And so this is sort of like how this works.

[00:22:04]And, you know, that sound plays without me having to sort of interact.

[00:22:10]And to give you a more sort of concrete example of how this sort of works, if I do a a mirai map, so this is this is basically the equivalent of L apply or or per map, but this basically maps each individual element in a different background process.

[00:22:26]Right?

[00:22:27]So if I map sort of, you know, this thing to a function, and then this just sleeps for the specified number, so from one to five seconds and returns that number.

[00:22:49]And this mural map has an argument which is called dot promise and basically works exactly the same as promises then.

[00:22:58]So we can pass a function to to this.

[00:23:02]And, again, if we use beeper beep, then this will sort of play all the first five beeps in in five seconds.

[00:23:15]And let me try and swish the microphone so that you can actually If you just hit the share button again, then you can click share system sound, and it it should let you do it.

[00:23:31]I think I just did it. Anyway sure.

[00:23:34]If I just play this oh, okay.

[00:23:38]So I hit an error.

[00:23:40]So this is basically telling me that I need to set demons.

[00:23:44]So I can just click through here and and set demons.

[00:23:47]So demons, again, is another piece of jargons.

[00:23:50]They are demons are basically just workers.

[00:23:53]They are background processes. Right?

[00:23:57]And the reason it asked me to set daemons for this map is for a map.

[00:24:02]Here, I'm mapping only five elements, but that map could be over, like, a hundred or a thousand elements.

[00:24:08]Right?

[00:24:09]And if daemons weren't set, then you'd basically be launching a thousand different sessions on your own machine, and that will probably crash your machine.

[00:24:19]So for a map, it will ask you to set the daemons beforehand because it's usually sort of an oversight or mistake as it was in this case.

[00:24:29]So I've just gone ahead and set six daemons, which are basically background processes.

[00:24:35]At any time in MRI, there's a function which is just info, and you can see, like, exactly the status of what's happened.

[00:24:44]So we have six connections here. Okay.

[00:24:48]We'll go through what the rest means in a bit.

[00:24:51]So if I just attempt that function again, then Hope he heard that right. Okay.

[00:25:05]Nope. That's okay. Oh.

[00:25:07]Oh, you didn't? Oh, sorry.

[00:25:09]What what do I need to you need to hit the share button and then you might have to hit advanced and then share system sound or there's a little checkbox on the right that says share system sound.

[00:25:26]But it's okay if it doesn't work out, it's all right.

[00:25:29]We have about fifty questions.

[00:25:32]So I say Sure.

[00:25:34]That we get some questions in here because there's a lot of confusion.

[00:25:39]Right?

[00:25:40]Like there's a lot of things that you have talked about, like future promises, things that we don't have any context for.

[00:25:46]So let's dig into some of these really quickly here.

[00:25:50]Definitely.

[00:25:50]One of them is from Nathan.

[00:25:55]Do the contents of Morai inherit definitions from the global environment?

[00:26:01]No.

[00:26:02]And this is is a conscious sort of design decision and it helps you sort of avoid mistakes.

[00:26:13]Okay.

[00:26:15]And like this is sort of like, to my mind, this is how sort of things are meant to be.

[00:26:25]And I know, like, that future, like, does it differently, and that package has been around for a while.

[00:26:34]So people sort of know that behavior.

[00:26:39]But there are various sort of edge cases there.

[00:26:45]And the sort of the the danger in trying to sort of automagically sort of infer what's sort of in your mirai can lead to sort of well, in the best case it can lead to errors, but it's like the worst case like it's subtly wrong and you get a plausible answer and you don't even know you're wrong.

[00:27:12]So that arguably is sort of worse.

[00:27:15]Mirai is sort of very explicit in that sense, and I'll I'll show you exactly what I mean.

[00:27:21]So if I if I define something like if I define something like slow funk, right, So this say just function and this just just again just sleeps and then it returns something like done.

[00:27:44]Okay.

[00:27:45]So if I attempt to sort of run Mirai slow funk, right, you might think that would work, but what you would actually see is there's error and it just says error cannot find function slow funk because Mirai runs your expression in another process, and this is a clean process.

[00:28:15]Every time you call Mirai, that is like a clean invocation.

[00:28:19]There's nothing in that evaluation environment.

[00:28:22]So what you would do here is you can call it, but then you would pass in SLOYFUNK to the MRI.

[00:28:33]So so something something like this. Right?

[00:28:37]So the expression is the same, Slyfunc, but then you would define Slyfunc inside the MRI is the Slyfunc which is this function that's living in my current environment.

[00:28:49]Okay. And then we get the result we accept.

[00:28:55]Okay.

[00:28:56]So you have mentioned feature and promises again, and you were like, everyone knows what that is.

[00:29:02]Charlie, we do not.

[00:29:04]So if you could give us a little primer on the difference between Mirai and Promises and Future.

[00:29:12]One of the questions was like, wait, I'm confused.

[00:29:14]Why would I choose Mirai versus promises or future?

[00:29:24]So promises is a way basically as I showed to, have actions occur when your Mirai or your or your future completes.

[00:29:38]So they're they're designed to be used together, they do different things.

[00:29:44]A Miraiwa feature is just a way to run an expression in another process.

[00:29:53]Right?

[00:29:55]Either in parallel or or async.

[00:30:01]Okay.

[00:30:03]So they things like feature and promises keep things running during these, like, long running tasks where if you have a chain of tasks that need to happen, for example, in a Shiny app, right, that need to happen one after another or reactive to each other, some of those future tasks need to be set aside and stored so that they can happen once something has finished running, and they can all happen all at one time.

[00:30:30]Does that sound right?

[00:30:33]Or, like, they can all happen happen in order.

[00:30:37]Yes.

[00:30:38]So a promise will be basically it's promise uses these verbs like then.

[00:30:44]Right?

[00:30:44]So this is the key this is the key function from sorry.

[00:30:49]Promises.

[00:30:51]Sorry. That's not very helpful.

[00:30:53]So it's basically It's basically, you know, when when you have MRI and that's finished, then, like, then take the return value from that and pipe it into another function.

[00:31:09]Okay.

[00:31:12]So Renato in the chat says, wait. I'm lost.

[00:31:15]Aren't our functions run sequentially anyway?

[00:31:22]Okay.

[00:31:23]So let me maybe just go through, like, the second part of my Will that answer it?

[00:31:28]What I had what I had in sort of what I wanted to demonstrate.

[00:31:34]So, like, sequential is basically so you're right.

[00:31:41]R is single threaded.

[00:31:43]So without a package like Mirai, everything will run sequentially.

[00:31:49]And Mirai is basically designed to overcome that.

[00:31:53]And why this is important?

[00:31:55]Well, I mean, R has been around for, you know, thirty years or so.

[00:31:59]I mean, back then you were lucky if you had sort of you know two or four cores that was sort of the of the norm.

[00:32:10]But you know modern laptop has at least eight cores.

[00:32:15]My MacBook, I have fourteen cores.

[00:32:20]All your computations in R by default only uses one single core.

[00:32:27]So easiest way to sort of demonstrate that is if I define a function again, DOM task, if I just define that as and again, I'm sort of using these generic functions because it's it's easier for me to control sort of how long they take rather than, like an actual, function that takes a long time.

[00:32:52]So this will just leave for two seconds and then return the the number.

[00:32:56]Okay?

[00:32:58]So if I actually sort of time how long this takes, then if I apply this to, say, five, a long task.

[00:33:11]Right?

[00:33:15]Then, you know, five times two, expect this to last ten seconds as you can see.

[00:33:29]And, you know, you're sort of this is just sleeping for two seconds, but, you know, you can imagine that could be doing some kind of, you know, either complex sort of matrix multiplications, you could be doing sort of or you could be like hitting some kind of API and waiting for sort of data to come in.

[00:33:55]Right?

[00:33:56]So that what you what you would want is for those to run-in parallel.

[00:34:05]So or if you we set six daemons, are six workers, then if we do the same thing and time sort of a mirai map, is the the parallel equivalent of l apply.

[00:34:28]Wrong task.

[00:34:30]And here we actually we actually wait for these to complete because otherwise it will return immediately.

[00:34:41]Then we can see that because they're done in parallel, this whole thing only takes two seconds.

[00:34:50]Alright. That made a lot more sense to my brain.

[00:34:54]We have each of these elements assigned to a different process, and they can all happen at the same time.

[00:35:01]So I can say, I can be like, all right, I have these, let's say five modeling tasks, I think in statistical modeling.

[00:35:09]So I have these five modeling tasks and they're going to take a while because they are each going to iterate over a bunch of things themselves.

[00:35:16]And I can run all of them at the same time on different cores of my computer.

[00:35:21]And I don't have to keep my laptop awake overnight in order to run them all and get all of those and wait for all of them sequentially to run on the same core.

[00:35:31]That's right.

[00:35:32]And, you know, these and so they don't have to be sort of, you know, even sort of simple functions.

[00:35:38]They they can be really com they can be full scripts, and you can literally do MURAI, and you can say, you know, you can literally say source whatever, you know, whatever file you have.

[00:35:50]And they can run your entire script, which, you know, may take hours.

[00:35:55]It may take days even. Right?

[00:35:58]And you can have them running in different in different processes.

[00:36:04]And the key about Mirai is with Mirai, you don't need to care, like, whether that's just running in another process on your own machine.

[00:36:13]So you're sort of utilizing all the cores that your laptop has.

[00:36:17]Or if they're running in processes on another machine.

[00:36:20]So if you have, like, a network server. Right?

[00:36:23]If you have a workstation where you can send jobs.

[00:36:26]Or if you're, you know, if you're yeah. Distributed.

[00:36:29]Exactly.

[00:36:30]If you're at a university and you have access to, like, an HPC cluster, Mirai allows you to do all of that.

[00:36:37]And, hopefully, we'll have time where I can just show you something very simple.

[00:36:41]I'm glad that you said that because I've been holding on to Hubert's question because I knew I knew we're gonna get there.

[00:36:46]Hubert had asked, wait. Can you use Morai with the HPC? Yes.

[00:36:49]Absolutely.

[00:36:50]So we're distributing the work between cores right now on our laptop, on Charlie's laptop.

[00:36:56]But we can assign distributed machines as our workers with Mirai as well.

[00:37:01]And we can scale up and down depending on, our needs.

[00:37:05]Right, Charlie?

[00:37:07]Great.

[00:37:07]And, hopefully, I before we run out of time, I want to just go through this code piece again.

[00:37:15]You've hit the nail on the head.

[00:37:17]There are two sort of core reasons why you would want to use Mirai.

[00:37:21]And the first is you can scale up and down the resources you need dynamically.

[00:37:28]So if you decide you need more workers to sort of crunch the tasks that you have on hand, you can simply add more workers.

[00:37:35]So these things aren't fixed.

[00:37:37]And what I want to show you is basically how this happens.

[00:37:42]Right?

[00:37:43]So Mirai basically works as if you think about it as a kind of a a hub model.

[00:37:52]Right? So when you set up daemons, this basically you set up, like, something that listens for incoming connections.

[00:38:03]Right?

[00:38:05]So what you can do is you can set up daemons with a URL.

[00:38:14]So this and you can and we have this helper function called local URL.

[00:38:18]So if you do that, then what happens is that sets up what you can think of as a base station.

[00:38:29]This listens for incoming connections from your daemons, from daemons that have started.

[00:38:34]Right?

[00:38:35]As long as they have access to this URL, then they can connect in and you have then you have a worker that's connected.

[00:38:47]So if I set this up, then this info function is going to come in handy.

[00:38:53]You can see that there are no connections at the moment.

[00:38:58]And what you can do is you can launch local daemons with dysfunction.

[00:39:05]Launch like sorry. Not like a local. Launch local.

[00:39:10]Okay.

[00:39:11]So launch scrambling Charlie's brain, y'all.

[00:39:15]Yeah.

[00:39:16]And you can say something like, you know, launch four local processes.

[00:39:19]And that basically goes ahead and does that.

[00:39:22]So if we look again, we have four processes.

[00:39:27]So we can do things like, you know, if I just evaluate to move right, like nothing.

[00:39:34]Then we can see that, you know, oh, wow.

[00:39:37]We have one task that's completed here. Right?

[00:39:40]Just to check that something's happening.

[00:39:43]And, you know, assuming we set this up and, you know, we can be running lots of things.

[00:39:48]So I'm just gonna copy this thing here.

[00:39:54]So this just runs a bunch of long tasks and okay.

[00:40:00]So the two that's executing, five that's completed, and now all seven tasks have completed.

[00:40:06]But if you have a bunch of things and you you sort of you realize you have a large backlog, then you can basically simply just launch more daemons.

[00:40:15]So you can say, oh, I want to launch another four daemons.

[00:40:21]And now we basically have eight workers online.

[00:40:25]So any tasks that are have been queued will just automatically use the number of available workers.

[00:40:34]Okay. Wait. Wait.

[00:40:35]That was the first two two seconds.

[00:40:38]We have produced daemons.

[00:40:41]We have created daemons automatically.

[00:40:44]Do we need to manually close them?

[00:40:46]Same with these processes that we're launching or these connections that we're launching.

[00:40:51]Do we need to explicitly close both daemons and connections?

[00:40:55]Yes.

[00:40:55]So, I mean, daemons are connections, and they're the same thing.

[00:40:59]Okay. And Good.

[00:41:01]These it's always good practice to to close them, and the way to do that is to set demons to zero.

[00:41:09]And that will close them straight away.

[00:41:13]But if you forget to close them and you just end your R session, they will all disappear as well.

[00:41:19]There's no danger of sort of leaving hanging processes at all.

[00:41:24]They're all as soon as the connection drops, they're all designed to terminate themselves.

[00:41:32]Perfect. Yes.

[00:41:34]But, you know, if you're if you have a script or if you have markdown document, it's always good to pair a daemons with a daemon zero at the end.

[00:41:48]Okay. Great. I interrupted you.

[00:41:51]We have fifteen minutes left.

[00:41:53]We still have so many questions, but I would love for you to be able to finish your side.

[00:42:01]I thought I was gonna have too much time, which is why sort of, like, overdid the introduction and okay.

[00:42:09]So the second part apart from so apart from sort of the scaling and sort of ability to add and subtract daemons at any time is you can run them anywhere.

[00:42:22]So if you just do daemons and a number, this will just launch processes on your own machine.

[00:42:31]Okay? But those processes can also be another machine.

[00:42:36]And there are different ways you can launch them.

[00:42:42]One is over SSH.

[00:42:43]One is via clusters manager.

[00:42:46]That's if you have access to a cluster.

[00:42:48]The other is, via POSIT workbench.

[00:42:51]So if you're lucky enough to be in an enterprise sort of environment that actually has Workbench, you can easily launch workers as Workbench jobs.

[00:43:03]But I won't cover that.

[00:43:04]I will try and cover SSH as an example.

[00:43:09]Okay.

[00:43:10]So the way you do that is you call daemons and you create a URL which is using this helper, which is host URL.

[00:43:24]Okay.

[00:43:25]So this creates a network socket which is available to network.

[00:43:36]So other computers will be able to connect to this address, this URL.

[00:43:40]Right?

[00:43:43]And if I do that again, like, nothing's connected yet.

[00:43:48]And what I can do is I can I can create a remote configuration?

[00:43:57]And in this case, this will be a an SSH configuration.

[00:44:01]And again, we have these helpers that really sort of, like, minimize what needs to be done.

[00:44:09]So here, I create an SSH configuration.

[00:44:12]And the own sorry.

[00:44:14]There there there are arguments, but the only the only required argument is this remote's argument.

[00:44:22]And that is simply the URL of the computer that you have SSH access to.

[00:44:29]So to give you an example, I switched to my terminal here.

[00:44:37]If I have sorry.

[00:44:39]This is just so I I actually I'm actually connected to a VPN.

[00:44:44]So this is I have access.

[00:44:46]So if I can have SSH, right, over the so slash p, this is the port that you connect to.

[00:44:55]And this is this is the address of the machine that you can connect to.

[00:45:01]Sorry. Here.

[00:45:03]And if I have access to this machine over SSH, and so this is like I'm actually running on this machine.

[00:45:10]So I have a MacBook, but this is connected to a Linux machine now.

[00:45:17]I'm gonna just exit out of that.

[00:45:19]But essentially, that URL there, which was one nine two dot one six eight dot zero dot one zero one.

[00:45:30]That's the only information I need to set up this configuration in Mirai.

[00:45:36]Okay?

[00:45:37]So once I have that, then I can actually launch use launch remotes and not launch local.

[00:45:44]Launch remote.

[00:45:45]So again, I can launch maybe two daemons, and I can pass that configuration in.

[00:45:51]Okay?

[00:46:04]See what happened.

[00:46:05]Oh, ah, so we have a time.

[00:46:09]So again, this is this is this is useful because, again, as I was explaining how Mirai works is you spin up this basically like this base station, right, which listens for incoming connections.

[00:46:27]That means the computer that you have access SSH access to has to be able to dial in to your to the machine that you're running Mirai on.

[00:46:39]And in this case, that's not possible because I don't have these ports that opened on my, like, called laptop.

[00:46:50]Fortunately, Mirai allows you to connect using SSH tunneling.

[00:46:58]So SSH tunneling, again, like, people tend to get confused, so I will try and explain it in as simple terms as possible.

[00:47:09]SSH tunneling is basically when you connect to a computer using SSH, that creates a tunnel immediately.

[00:47:19]So imagine there's a tunnel between the two sides.

[00:47:23]Now instead of a connection being made between the two sides, each side connects to a local port.

[00:47:33]So if I'm as Mirai, sort of if I set daemons, I'm listening to a local port, and then on the daemon side, that computer's dialing to a local port, and then the tunnel basically bridges the two sides.

[00:47:53]So there's no longer any connection from that computer to my computer.

[00:48:00]Connected to an intermediary.

[00:48:02]Yes.

[00:48:03]And, like, very concretely, if you follow this, you will get it.

[00:48:07]So if I just if I if I just shut down the previous daemons, And in this case, what I'm gonna do is I'm gonna create a local URL.

[00:48:22]Right?

[00:48:22]So this is not is not the URL that's open to other computers on the network.

[00:48:29]This is a local URL.

[00:48:31]I'm gonna set TCP equals true because this is this is a TCP connection.

[00:48:41]If I use status, this just gives me a little bit more information than info.

[00:48:47]But you can see we're actually listening to this one two seven this address, which is the local host address.

[00:48:54]So I'm now listening to a port on my own computer, and this is the port that the tunnel is gonna be opened on.

[00:49:06]I can then create this SSH configuration.

[00:49:13]And, again, I can connect to this address is one nine two dot one six eight dot zero dot one zero one.

[00:49:23]And I actually need to use this port because that's how I've configured that.

[00:49:29]And I just need to specify this this argument, which is tunnel equals true.

[00:49:36]K?

[00:49:38]I think I now.

[00:49:40]If I if I try and if if I try and launch now using this this configuration, this should now hopefully work.

[00:49:54]So yay. Now I have two it.

[00:49:56]I have two workers, and they are on another machine that I have that's in London.

[00:50:03]It's actually a very old sort of converted Mac mini, actually, but it's running Linux now.

[00:50:10]And to prove to you that it's running Linux, I'm gonna run like Mirai, and I'm just gonna ask for the sys.

[00:50:18]Info.

[00:50:19]Okay?

[00:50:20]From the worker, and you can see that, hey.

[00:50:25]This is running Linux.

[00:50:27]And just to prove that I'm not running Linux, this is my machine.

[00:50:34]It's definitely some form of Mac.

[00:50:39]Okay.

[00:50:40]That was a whirlwind, but if you got that, you understand how sort of SSH timing.

[00:50:46]But as you can see, all all you had to do was create this configuration, which is just this URL, which is if you can access that machine over SSH, you have all the information you need to to run to run Mirai over SSH.

[00:51:04]Okay.

[00:51:05]Charlie, this was so much information all at one time.

[00:51:09]We have so many questions that we will not be able to get to all of them, but I want in our last five, four or five minutes here, to see if we can get to some of them.

[00:51:19]So one of them from Brent was, I believe NumPy distributes calculations over multiple cores.

[00:51:24]So does Mirai allow R to perform the same thing as well?

[00:51:29]Yes. It does.

[00:51:30]And the NumPy is very specific, so it will run only numerical work that NumPy handles.

[00:51:41]Mirai allows you to run arbitrary expressions.

[00:51:46]So anything you can do in R, can you can put in a Mirai.

[00:51:49]That's that's the difference. Yeah.

[00:51:51]Great. Any R processes. Okay.

[00:51:53]Dan had also asked, does Mirai ship with some kind of task viewing dashboard, like Python's dash visualization so that you don't have to type info over and over?

[00:52:06]Not built in.

[00:52:07]Not built in. Okay. Great. Let's see.

[00:52:12]We had another question that was, can we use parallel detect cores to see how many daemons we can use?

[00:52:20]I really like this question because I don't know how many daemons I would have access to.

[00:52:24]And that's great question. And Yeah.

[00:52:27]Like, Mirage by design forces you to actually specify the number you want.

[00:52:32]So that can be, you know, that can be fourteen, which is sort of number, of course, I have on my laptop.

[00:52:39]So you can use something like parallel, sorry, parallel detect cores or whatever.

[00:52:46]Bam.

[00:52:46]But, you know, you can easily be on a server with, you know, two hundred and fifty six cores or something.

[00:52:51]And, like, sometimes you don't want to use all your calls because you might be running something else.

[00:52:56]Right? So, like, it never has that context.

[00:52:58]You running a a automatic function.

[00:53:03]The other reason is this like, you might want to be using your calls if what you're doing is compute intensive.

[00:53:15]Right? You want to sort of utilize all your calls.

[00:53:18]Fine.

[00:53:19]But instead, if what you're doing is actually waiting on IO, so this is like if you're requesting, like, a download.

[00:53:29]Right?

[00:53:30]That is actually not taking all your computation power.

[00:53:33]So for those kind of tasks, you might actually want to use more than the number of calls you have.

[00:53:38]So you might wanna use, like, twenty eight, like, if all you're doing is, like, querying some remote API.

[00:53:44]So that's also another reason why, like, I don't sort of try and assume, like, the number that people want.

[00:53:52]And yeah.

[00:53:54]Okay.

[00:53:55]So with our last two minutes, Zach said he had to leave, but he had a great question.

[00:53:59]He said, in typical data science work, would parallel processing be more useful than async processing, do you think?

[00:54:08]Yes.

[00:54:10]If and and Mirai actually powers a lot of things sort of under the hood.

[00:54:15]Right? So if you're using parallel per, that uses Mirai.

[00:54:21]So you set daemons, you use the null per syntax, and that is just parallel.

[00:54:27]It's not async.

[00:54:28]Where async really sort of comes into its own is if you're, using Mirai with Shiny or using Mirai with something like Plummer two.

[00:54:39]So, again, Mirai powers Plummer two under the hood.

[00:54:42]So if you use an async function in Plummer two, that implicitly uses Mirai.

[00:54:49]Awesome. Okay.

[00:54:50]I bet we can get in one more question.

[00:54:54]One more. One more.

[00:54:58]So Rob had asked, can you print progress slash percentages that only get shown on call, like, while things are running?

[00:55:06]I think that's what that's asking.

[00:55:08]Yes. Right.

[00:55:10]So so very quickly, if you do, like, a Mirai Map, one of the collection options so instead of just collecting is you can do, like, progress, and and that gives you a progress indicator.

[00:55:25]Okay. Cool.

[00:55:27]Please carry on while I do something.

[00:55:29]I'm going through and just looking at all these questions.

[00:55:35]I think that I will get with Charlie since we are out of time, We'll see if we can answer some of these.

[00:55:41]There is one from Nadebeco. There is one from David Diaz.

[00:55:45]And they look like they could be answered pretty quickly.

[00:55:48]So we'll answer them in the chat after the session.

[00:55:50]Charlie will help.

[00:55:53]Excellent.

[00:55:56]All right.

[00:55:56]The last thing I want to mention.

[00:55:58]Oh no, I'm out of time.

[00:56:00]Is a MiRai skill.

[00:56:05]If you folks out have access to Claude code or OpenCode or some great AI agent, install the rlib skill, Mirai, and you can just invoke that.

[00:56:19]You can trans like, you can transform all your sort of normal scripts into parallel and async code.

[00:56:29]Amazing.

[00:56:30]Oh, I just spelled sorcery wrong in the chat. Don't me.

[00:56:34]Hold on. Let's fix it.

[00:56:37]All right. Thank you so much, Charlie. That was amazing.

[00:56:40]You're getting thank yous in the chat.

[00:56:42]Thank you for being patient with us while we all try to muddle through what async is, what Parallel is, what Mirai does, what future and promises are.

[00:56:51]I think we all learned a lot and have a lot to like go Google and think about.

[00:56:55]I wish that I knew about this in undergrad and grad school.

[00:56:59]I definitely didn't and it would have helped my life, quality of life with coding so much.

[00:57:05]So thank you so much.

[00:57:06]We will get with Charlie to answer some of the more simple questions afterwards in the chat.

[00:57:12]We will see you all on Thursday at the Data Science Hangout if you would like.

[00:57:16]Thank you, everybody. Have a fantastic week.

[00:57:20]I'll see you when you see you. Thanks, Pauline.

[00:57:24]Thanks, everybody.

[00:57:26]Bye. Thank you, Charlie.