Training a Tiny VLM to Play Kirby with a New Long-Horizon RL Recipe

Added: 2026-05-20

[00:00:02]So, in this video, I'm going to be talking a little bit about VLMs, vision language models, and I'm going to be showing how I was able to train one myself, a small 50 million parameter model, uh to play this Kirby game that you're watching right now.

[00:00:19]And you'll see how it goes from not being able to play at all to being able to do decently, got over that.

[00:00:27]Uh so, watch the end of the video to see how I do this and what the full results are. I saw this paper the other day, and it recently came out from Princeton, and I thought it was pretty interesting in the way that they approached AI agent training.

[00:00:40]So, it's called Odysseus, scaling VLMs to 100+ turn decision-making in games via reinforcement learning.

[00:00:48]So, it's about basically how to teach vision models to play video games, not by showing it how, but by letting it learn from experience.

[00:00:56]So, this headline itself sounds pretty simple, right? AI just plays Mario.

[00:01:01]But, that's not really what it makes it interesting. What's interesting is that the training recipe that they figured out, because it turns out getting an AI to make 100 decisions in a row where each one matters is really hard.

[00:01:13]So, they actually cracked this with a pretty interesting trick. So, in this video, I'm going to break down exactly what this means, what they did, and what their results were. And then, the latter half, we're actually going to do an experiment ourselves, training this not using Super Mario, but using Kirby Stream Land, one of the classics from the Game Boy era.

[00:01:33]So, let's get started.

[00:01:36]And if you like this video, please subscribe to my free weekly newsletter, where I give my honest thoughts about the week's AI news that I can't share here, as well as interesting research and papers that I found, and what projects I've been building behind the scenes.

[00:01:49]There's a link in the description, or go to onchainaigarage.com.

[00:01:53]Every word is written by me, mistakes and all. So, if you're sick of AI slop articles or all hype with no substance, subscribe and give AI Garage a weekly shot. Now, back to the video.

[00:02:04]So, a quick background cuz I think a lot of people might be thrown by this. Most people are familiar with ChatGPT, Claude, that sort of thing. So, those are LLMs, large language models. Sure you're familiar with the term.

[00:02:16]Um and this is for text, right? Text goes in, text comes out.

[00:02:20]You ask it a question, it answers.

[00:02:22]So, VLMs are a little bit different, right?

[00:02:25]This is the next step. This is a vision language model. Same idea, but now we can also look at images.

[00:02:31]You can give it a screenshot and ask, "What's happening here?"

[00:02:34]and it'll describe what it sees.

[00:02:36]Think of it like this. An LLM is like talking to somebody on the phone.

[00:02:41]You can only describe things with words.

[00:02:42]A VLM is like a video call. They can actually see what they're looking at.

[00:02:49]So, VLMs can see.

[00:02:51]That's great, but can they do things?

[00:02:54]So, if I show a VLM a game screen and ask, "What button should I press?" It can answer. But, here's the problem. A game isn't one decision, it's hundreds.

[00:03:03]And the consequences of each decision might not show up until way later.

[00:03:08]So, imagine you're playing Mario and you skip a power-up on turn three.

[00:03:12]Everything seems fine, but then 80 turns later you hit a boss and you don't have a fire power you need to beat it. You die.

[00:03:20]But, which turn was the mistake?

[00:03:22]Turn three or turn 80? So, that's the hard part of it in all this.

[00:03:30]So, how people tried to handle this, solve this issue before, so the option one was supervised imitation.

[00:03:37]You record an expert playing the game, and then you show the AI those recordings and say, "Copy this."

[00:03:42]So, the problem with this is you need a ton of expert demonstrations. It doesn't scale really well, and the AI becomes brittle. It It knows what to do in situations it's seen before. In other words, it's all getting overfit. So, you throw something new at it and it completely falls apart. This is a common issue with training models. This issue of overfitting. It doesn't generalize.

[00:04:03]So, option two is short horizon RL.

[00:04:07]Reinforcement learning, which is where the AI learns by trial and error.

[00:04:10]But, previous work only did this for maybe 20 or 30 steps. So, it's not really enough to handle long horizon tasks.

[00:04:21]And neither of these can actually prove an AI can handle long complex decision-making.

[00:04:31]So, this is the core difficulty and it has a name. The credit assignment. So, think of it like coaching a basketball team.

[00:04:38]Your team loses by two points. Was it the missed free throw in the third quarter that caused it? The bad pass in the first? The lineup choice at halftime?

[00:04:46]When the outcome is the result of hundreds of decisions stacked together, figuring out which one actually mattered is incredibly hard.

[00:04:53]In a game like Mario, the AI might die on turn 80.

[00:04:57]But, the real mistake might have been turn three or turn 15, turn 42.

[00:05:03]The longer the chain of decisions, the harder it is to assign credit or blame to any individual choice.

[00:05:13]And this is what the Odysseus paper proposes.

[00:05:16]So, instead of trying to solve everything at once, you split the problem into two phases.

[00:05:21]Phase one.

[00:05:23]SFT.

[00:05:24]Supervised fine-tuning.

[00:05:26]You teach the AI to see.

[00:05:29]But, critically, they're not teaching it to copy the gameplay. They're teaching it perception.

[00:05:33]That's an enemy. That's a gap. That's a power-up.

[00:05:36]Kirby is in the air. The path goes right.

[00:05:39]They took around 5,000 game frames and had a stronger AI model write structured description of what's happening in each one.

[00:05:47]Then they train their model on those descriptions.

[00:05:50]For phase two, they teach the AI to act.

[00:05:53]So, this is the reinforcement learning phase, and now it understands what it's looking at.

[00:05:58]And it lets it play. Lets it fail. It lets it learn from the consequences.

[00:06:03]So, think of it like this. SFT teaches the vocabulary. RL teaches the street smarts.

[00:06:08]You need to know what a stop sign is before you can learn when to run one.

[00:06:17]Let's get into this a little bit more.

[00:06:18]What is reinforcement learning concretely? Touch on this in a couple other videos, but this is kind of core to this paper and to this experiment.

[00:06:26]And the loop is fairly simple. The agent looks at the screen, picks an action, the game responds, um and the agent gets a reward signal.

[00:06:36]Maybe it's positive if made progress, negative if it took damage. And it updates its strategy slightly.

[00:06:43]You repeat this for thousands of times, and it gradually gets better.

[00:06:46]Now, the specific algorithm is usually called PPO, proximal policy optimization. And the key idea is that you don't change too much at once. If the AI discovers that jumping works well in one situation, PPO makes sure it doesn't over correct and start jumping everywhere.

[00:07:02]Small, stable improvements.

[00:07:06]So, here's where it kind of gets interesting.

[00:07:08]In reinforcement learning, you usually need two things, an actor and a critic.

[00:07:12]The actor makes the decisions, and that's our VLM. That's the model. The critic evaluates situations. It says, "This state looks promising." or "You're in trouble."

[00:07:22]The obvious approach is to use another big language model as a critic.

[00:07:26]Have it read the game state and judge it. But that's extremely expensive. A lot of the previous papers have been using frontier models, and you need to run two massive models, and it's actually kind of unstable. It tends to break during training.

[00:07:39]So, what the the team found out is that the critic doesn't need to be smart. It doesn't need language. It doesn't need to reason.

[00:07:46]It just needs to look at the pixels and output a single number. How good is the situation on a scale?

[00:07:52]And crucially, it only needs to do this once per game turn.

[00:07:56]Not once per word the VLM generates.

[00:07:58]That's the turn level versus token level distinction.

[00:08:02]And it's what makes the whole thing really affordable.

[00:08:05]Cheap, fast, and stable.

[00:08:11]So, before we go into this, let's kind of break down of what the cheap critic is. And it's called a CNN.

[00:08:18]It's a convolutional neural network.

[00:08:23]I'll be mispronouncing that word.

[00:08:25]Um and this is actually one of the oldest tools in deep learning, and it's perfectly suited for this job.

[00:08:32]Uh the CNN works by sliding small filters across an image.

[00:08:36]Kind of like looking through a tiny window that you move around the picture.

[00:08:41]The first layer of filters detects simple things.

[00:08:44]Edges, lines. The next layer combines those into shapes.

[00:08:48]Circles, rectangles.

[00:08:50]The next layer recognizes objects.

[00:08:54]Enemies, platforms, the player character.

[00:09:00]At the end, all that gets compressed into a single number.

[00:09:04]How promising does this game state look?

[00:09:06]So, that's it.

[00:09:08]That's the entire critic's job.

[00:09:11]No language, no reasoning, just pixels in and numbers out. And it's tiny compared to the VLM. Fast run, faster train. That's the whole trick behind this paper.

[00:09:28]So, let me put the whole picture together with an analogy.

[00:09:31]The VLM is like a race car driver. It's expensive, it's smart, it can read the track, think about strategy, can make complex decisions. That's your actor.

[00:09:39]The CNN critic is like a cheap driving instructor sitting in the passenger seat. They don't know how to drive as well as the driver. They can't explain racing theory, but they can look out the window and say, "Good line." or "You're about to hit a wall."

[00:09:53]You don't need a second world You don't need a second world-class driver to judge every micro decision the first driver makes. You just need someone who can look at the road and give you an obvious thumbs up or thumbs down.

[00:10:04]So, that's the separation. Expensive reasoning for decisions, cheap vision for evaluation. And that's what makes the 100-plus turn reinforcement learning actually work.

[00:10:19]So, you might ask, "Why do we even need a critic, right?" Some recent methods, like GRPO and reinforce plus plus, skip being critic entirely. It's been kind of a trendy move.

[00:10:30]They just look at the final outcome and try to figure out things from there. And for simple short tasks, that works. It's fine. But for 100-plus turns, it breaks down. The reward signal is too spread out. Like trying to figure out which ingredient ruined a recipe by only tasting the final dish.

[00:10:45]When a recipe has 80 ingredients, you need somebody tasting along the way.

[00:10:49]And the paper shows this directly.

[00:10:50]Critic-free methods are unstable and weak at this scale. You need value estimation.

[00:10:55]They also add one more trick, which was positive advantage filtering.

[00:11:00]Instead of heavily punishing the model for bad moves, which can make VLMs unstable and weird, they mostly just reinforce the good moves. Actions that work better than expected get amplified.

[00:11:10]Actions that work worse mostly get ignored rather than punished. It's just a cleaner learning signal.

[00:11:16]So, did it work?

[00:11:18]Yes, pretty dramatically, actually. So, these are the results of their paper that they found in this task. Odysseus achieves over 3x the average game progress of frontier models like GPT-4, Botson it, 4.6, Gemini 3 Flash, models that are massively larger and a lot more expensive, obviously.

[00:11:36]And compared to critic-free RL methods, uh like GRPO and reinforce plus plus, it's not even close.

[00:11:48]PPO with the CNN critic is far more stable and reaches a much higher performance. So, the takeaways for long horizon embodied agents, you probably need value estimation. The current hyper around critic-free LM training doesn't transfer to this kind of setting.

[00:12:04]So, here's kind of the full recipe and the actual contribution of this paper.

[00:12:08]Perception focused SFT on around 5,000 frames teaches a model to see.

[00:12:14]A VLM actor uh then picks that action with structured reasoning. Cheap CNN turn-level critic that estimates state value.

[00:12:24]Yeah, PPO for stable policy updates.

[00:12:26]And then positive advantage filtering to avoid destabilizing the VLM.

[00:12:31]And auto curriculum uh to balance learning across different difficulty levels. So, none of these ideas are individually brand new, but wired together in a specific way, they make something work that didn't work before.

[00:12:45]Long horizon VLM agents trained on interaction.

[00:12:51]So, this was the original experiment. And this is what we're going to try to recreate. Playing a baby Odysseus.

[00:13:00]It's the same recipe, just doing it on my laptop. Originally, they had used Super Mario Land, uh the classic. Um and the model they were using was Gwen 3 VL 8B.

[00:13:09]An 8 billion parameter model. And they were using research GPUs um that I don't have access to to actually do this. What we're going to be doing is Kirby's Dream Land, an equally good classic Game Boy game from 1992.

[00:13:23]We're going to be trying to use small VLM 2 for this one.

[00:13:29]Um the much smaller vision model than this Qwen 3 model they were using it. Um but it's what I have to use because the GPU we're using is my RTX 3070 and 8 gig and this is a laptop GPU. I'm going to be running this actually on my laptop not my usual PC that I'm going to. I'm going to use Pi Boy emulator.

[00:13:49]Um and the perception perception training uses the same structured annotation approach and the critic is the same lightweight CNN structure.

[00:13:56]Everything is faithful to the paper's design just kind of shrunk down to fit on my hardware um that you may own yourself.

[00:14:06]So now we're going to be moving into the terminal a little bit and I'm going to be doing this in Codex actually, I decided.

[00:14:15]So let's go over there now.

[00:14:18]Okay, and this is the spec script that I I handed to Codex.

[00:14:23]Uh you can see the goal to build a laptop laptop scale recreation of the core idea from the Odysseus Mario paper.

[00:14:32]A vision language actor learns a long horizon game perception focus SFT, reinforcement learning and lightweight CNN critic.

[00:14:40]So this is the reference paper like I said.

[00:14:44]Uh Super Mario Land they used they used this Qwen 8 billion instruct model.

[00:14:49]So this is what we're going to be trying to do.

[00:14:52]And this is going to be the core architecture. We're going to start with Kirby emulator frames.

[00:14:57]Going to use the small VLM model structured output perception reasoning action action parser then Pi Boy executes the action um which is like a a Game Boy emulator that is run through Python.

[00:15:13]Um then you have the frame reward and CNN critic which estimates the value. PPO updates actor policy and then it's just a continued training loop from there basically.

[00:15:25]Well, I handed this to Codex.

[00:15:27]So, I thought this would be a good opportunity to try Codex's new goal function.

[00:15:32]So, I set approvals to be auto review, which basically um Codex will review whether it actually needs my approval, whether something has high risk or low risk.

[00:15:45]Seems like most of what it would have asked me to do is low risk.

[00:15:48]Um and then you see pursuing goal down here. It's been running now for an hour and a half.

[00:15:53]And once I gave it the data, I gave it the ROM uh for Kirby's Dream Land, and I gave it um the spec script, it was able to run it, and currently it was downloaded a bunch of dependencies that it need for the SFT.

[00:16:10]Had some kind of bottleneck, um which it had to make these kind of changes, you could see.

[00:16:15]Training a format adapter, ran strict live, the um roll out. It built this mixed format in game play SFT curriculum. So, now it's gotten through that. So, that would took up a lot of the hour and a half that we've been running right now. It's just trying to uh like troubleshoot all of this um the training curriculum get a proper SFT run in. And that's nice because I was I was away from the computer for most of this hour and a half. So, I didn't have to sit here and and live through it, which is what I usually have to do with these experiments. You can see it's running the SFT right now. So, let's just see um how it runs.

[00:16:56]What we get.

[00:16:59]Okay, so that run finished, and you now lost down to 0.012. So, that's pretty good.

[00:17:05]Now, it's evaluating strict lenient lenient action success and action diversity on held out frames now.

[00:17:12]So, we'll see if it can handle different types of uh frames.

[00:17:20]Okay, so it made some progress on that run. Let's see.

[00:17:24]Uh but we didn't quite have gameplay level um bolts yet.

[00:17:30]Still running different training runs.

[00:17:33]The next targeted SFT step is this new data set.

[00:17:39]Going to be using this kind of validated right right right A right B progress cycle.

[00:17:46]So, it's going to be running that. Um but I'm going to just let this run overnight, I think.

[00:17:54]So, I don't know how long it's going to go.

[00:17:56]It's at We're at an hour and 47 minutes.

[00:18:00]And should be okay if I just let it run overnight, right?

[00:18:04]Well, we're going to find out. So, I will see you guys in the morning.

[00:18:10]Okay, good morning, everyone.

[00:18:12]Uh we're back at the desk.

[00:18:15]And you can see I was like 95% sure this would be stopped at some point.

[00:18:22]But I just got to my computer and you can see 12 hours, 12 minutes.

[00:18:26]It's just been running non-stop.

[00:18:29]I think I should stop it.

[00:18:31]See what it's been doing for the 11 hours or ever long since I was actually working on this.

[00:18:40]Ding ding. It seems like it's working on the PPO phase at least. I think it did finish the SFT.

[00:18:49]That's crazy. I'm amazed uh I didn't run into my my limits here. I'm only currently on the $100 plan for a Codex.

[00:18:57]And it's been running non-stop for over 12 hours now.

[00:19:02]Uh it said take a second to review all the work done. Are you still on the path to recreating a core the core process described described in the paper spec file creating a mini VLM that can generalize and play Kirby better than the base models consider better possible alternatives.

[00:19:18]I'm going to try to audit the current reports.

[00:19:23]Just want to see if it's right on the right path.

[00:19:25]It's been running for 12 hours. It shouldn't I don't know. That seems long for something like this.

[00:19:32]Okay, short answer we are aligned with the engineering pipeline of the spec, but we are drifting away from the core scientific target if we keep optimizing scripted routed cues policies.

[00:19:42]The specs core spirit does not find the best Kirby action script. It is closer to build a VLM style actor that sees frames, right?

[00:19:49]Feature structured action with SFT improve gameplay.

[00:19:52]We have done a lot of the pipeline correctly, but the strongest results are coming from search assisted action schedules.

[00:19:58]Not from the mini VLM that understands the screen and generalizes. We went a little bit off track. Like he was chasing these tiny games from PPO and you can try to optimize that as much as you like apparently for 12 hours, but that's not necessarily the best use of our time here.

[00:20:17]Use of our tokens. Next phase to visual policy generalization.

[00:20:25]Uh that is more faithful to the specs core goal and more PPO variants on memorized scripted cues. Yeah, we don't want it to memorize.

[00:20:34]Okay, so let's let's take that little pivot.

[00:20:40]Okay, so I told it to make that pivot.

[00:20:42]Remember the core goal is not necessarily to get the best Kirby playing model.

[00:20:46]Um it is to train the VLM based on the process of the paper and to see what the outcome is there.

[00:20:52]This was my first time using goal and maybe I was a little bit my prompt was not the best.

[00:20:58]Um I think I need to be a little bit more explicit with what I want it to do and when it should give up.

[00:21:09]Uh going to Okay, so it's going to be making that move now.

[00:21:22]And we'll see what we get from here.

[00:21:24]Yeah, I'm still learning how to use goal. I'm amazed it ran for 12 hours.

[00:21:28]Um but I don't know if that was the best use of our time, although I was sleeping, so.

[00:21:35]I guess we got some some data out of it.

[00:21:40]I think yeah, I think I need to play with goal a little bit more to see that like the right prompting.

[00:21:45]I need to think a little bit more about how to structure the goal.

[00:21:51]Because it really will just run forever.

[00:21:55]So, okay, let's see what we get out of this.

[00:21:58]Okay, so digging into this little bit deeper, I think the issue previously was kind of went off track um and it kind of had a limited data set.

[00:22:06]We need thousands of stills like the paper had I think over 5,000.

[00:22:10]Not the hundreds that we have here.

[00:22:14]Um and then there was some issue with the labels. A single still may not be enough.

[00:22:19]You either need to use two to four frame history, frame and compact RAM state or both.

[00:22:27]So, it's still the same principle. We just need more clearer labels.

[00:22:32]Um balance actions intentionally.

[00:22:36]They need a little bit of tweaking, but the core innovation is not memorizing optimal curvy strip. It is SFT AVLM into a game playing actor that improve it with RL using the visual value of feedback that we talked about.

[00:22:52]Um I think that's the pivot we're going to take.

[00:22:56]My concern was that the small VLM was just too small to to actually properly execute this, but it says Codex says it's okay as long as we we implement these changes.

[00:23:07]So, let's do that and see what we get.

[00:23:10]Okay, so we finished the larger SFT on the 5,000 still corpus, and we did get better results so far.

[00:23:21]Um you see the assessment was at the larger data set improved the VLM from one action collapse to limited action diversity, but still not a competent life controller yet. The next paper line step is to use this SFT adapter as a stage one VLM artifact and move on to the RL distillation stage with the CNN actor critic path. So, it's not uh quite up to the standard we need right now, but that's okay.

[00:23:48]Uh but we're making progress. So, we're going to move on to this RL stage that we talked about.

[00:23:54]This is one of my first exper- like ML experiments in Codex, and while it's doing a good job actually executing it, I feel like this kind of stuff is not it's not very good at explaining what it's doing in clear terms for me. I feel like Claude always is a better explainer of exactly what the issues are.

[00:24:15]Feels more um just clear and concise in the way it describes what it's doing.

[00:24:24]Even though I think just generally I think Codex is handling the actual work better.

[00:24:30]Um this is my observation from using this today. Okay, so we did the first run of the CNN policy, and this is kind of uh the results that we got. So, some of the terms here that come up, PPO is the reinforcement learning algo used to improve the policy from emulator rewards. The two things you need to know here are sampled policy and greedy policy.

[00:24:54]So, greedy policy is it always takes the single highest scoring action.

[00:24:58]And the sampled policy is instead of always taking the single highest scoring action, the model randomly samples from its action probabilities. So, higher probability actions are more likely, but there's a certain randomness involved.

[00:25:12]Um and the the policy itself, CNN policy is a small neural network that looks at the game frames and chooses controller actions.

[00:25:19]That's what um some of these terms are.

[00:25:22]So, before the PPO, the greedy CNN did nothing useful. Sampled CNN did some what useful things.

[00:25:30]Um but after PPO, you could see the greedy CNN was still weak, but it definitely did improvement from progress at zero to progress at 518.

[00:25:40]Um but the real improvement was sampled CNN, which went from uh 3,117 to 4,808 um in terms of the game progress.

[00:25:53]So, the concrete improvement was reward improved from 154 to 248. And like I said, the progress improved up to 4,808. And eight, so.

[00:26:03]60% uh improvement in reward, 54% progress improvement. So, that is um the caveat obviously being that this improvement only showed up when sampling actions.

[00:26:15]The still deterministic greedy version is still somewhat poor.

[00:26:19]So, PPO improved the policy policy distribution, but it did not yet produce a reliable single best action at each frame. Let's see what the next stage will be for this.

[00:26:30]Okay, so the next steps. I keep having to kind of reinforce Codex that we're just trying to recreate the paper, and it's okay if we fail.

[00:26:38]Uh it keeps trying to like add things extra um in order to get a better model, which makes sense if you're actually trying to create an ultimate Kirby playing model.

[00:26:48]Maybe we'll do that, but right now we're just trying to recreate the paper. So, we want to try to stay as close to that as possible, even if it means failing at the end. But, we're we're having some improvement. The next move is going to run more tuned CNN PPO uh from the SFT prior checkpoint, evaluate the sampled CNN policy as the RL policy, and then document whether we see any improvements.

[00:27:09]So, we'll move on from there, and we'll see what we get.

[00:27:16]Okay, so it completed that uh training run, and you can see the key results here.

[00:27:21]The pre-RL S CNN SFT prior sampled baseline max progress got us to 3,117.

[00:27:29]In the strict longer PPO best sampled checkpoint, we got max progress of 4,690.

[00:27:35]So, this is the cleanest faithful result so far.

[00:27:38]At the SFT prior plus CNN PPO involved improved sampled gameplay.

[00:27:46]Um so, there are some limitations. The greedy best checkpoint still failed.

[00:27:51]Um so, it's not perfect, but the conclusion is that the paper style RL stage does improve stochastic CNN policy, but not into a stable deterministic controller using the setup that I have, which like I said, this is a mini version of this setup using a different model and different GPUs than they were using. So, just to show you what this looks like in the actual game, uh this is actually rendered with the agent playing Kirby.

[00:28:21]And this is going to be the sample prior to any of the training, prior to our SFT or the PPO training with CNN. So, show you what it looks like.

[00:28:33]So, this it might be difficult to see. It's a little bit clippy.

[00:28:37]Um just the nature of the recording.

[00:28:40]But, you see you get stuck there.

[00:28:42]But, in terms of progress, he gets to around 2,000 uh 400 in terms of progress in the stage. And now look at our trained sample.

[00:28:55]So, this one will go 4,400 plus uh move uh in terms of progress.

[00:29:03]Might not always be clear, but you can see he's moving along a little bit more.

[00:29:14]And that's as far as we get. First, so for both of them, he just went 180 steps, and they counted how far they got into the level in 180 steps.

[00:29:24]So, I asked it to try to remove that to see how far we can get with Kirby still surviving. And it saw that the average max progress increased to 5,358.

[00:29:35]The total best observed run being 8,760.

[00:29:39]Um, so the best run went to 388 steps.

[00:29:43]It's quite a bit larger than the original cap that we had just for training purposes. And now I'm going to try to record a video of this longer run so you can kind of see what what happens.

[00:29:54]Okay, so this is the video of the extended run.

[00:29:57]And let's see how far Kirby gets here.

[00:30:00]So, this is just letting it run until he dies.

[00:30:05]This is our agent after the full training on a long horizon. This one went 388 steps.

[00:30:13]Quite a long uh Doing pretty good. Got stuck there.

[00:30:19]See, can he get over this?

[00:30:21]Uh, he did it. Good job.

[00:30:23]Doing good.

[00:30:26]Yes.

[00:30:27]Uh, he fell down. Okay, so that's the end.

[00:30:30]Uh, but he did pretty good. He got pretty far into the level considering it's just an agent.

[00:30:34]Um, and this is a very small model. Like I said, this is s m o l b l m, very small.

[00:30:41]0.5 billion parameters, not even a billion parameter model. So, very lightweight model, but it was able to get a pretty good result after the the training we ran through.

[00:30:54]So, there you have it. That's our recreation of this Odysseus paper.

[00:31:01]Uh decent results.

[00:31:03]Little bit of a tricky tricky experiment. Not totally clean, but I think we got decent results in the end and showed how their approach towards training LLMs for long horizon tasks, using kind of the fun example of a Mario game or a Kirby game, um kind of showed how their novel training approach really made it possible to improve on those long horizon tasks.

[00:31:29]So, I hope you found this uh video interesting.

[00:31:32]Kind of an interesting idea for If you'd like, please leave a comment.

[00:31:36]Please subscribe. If you like this video, please leave a like.

[00:31:39]And then I'll see you in the next experiment. Thank you for watching.