Code it with AI - Language Model Roundup 2 (ep.30)

Ajouté : 2026-05-20

[00:00:03][music] [music] Hey and welcome back to Coded with AI.

[00:00:17]Uh this is episode 30. I'm Carl Franklin from Appv Next. That's Jeff Fritz from Microsoft. Hey Jeff, what's up? Ah, you know, it's another it's another wonderful day here on the east coast and I just learned I have to travel to a Microsoft event next week.

[00:00:33]>> Oh, [laughter] probably not going to it's probably going to be raining there.

[00:00:37]>> Well, um, San Francisco, so we'll have fun.

[00:00:40]>> Yeah. Okay. It'll be nice. All right.

[00:00:43]So, this episode is a continuation of last week where we did the local language model roundup number one and promised to do more of them. So, this week, uh, we tested seven open-source models. Seven more.

[00:00:58]>> Carl, should we should we get that voice sound effect to come in and play during this? Previously on Coded with AI.

[00:01:05]>> That's right. Yeah. Old soap opera style.

[00:01:08]>> There you go.

[00:01:09]>> Um, so we left uh last week with Quen 3.6 colon latest being the winner.

[00:01:19]>> Yeah.

[00:01:19]>> Of all of them. That was my favorite anyway, >> the the most capable at being able to answer and and deliver content that we felt was workable >> and think like, you know, as much as a model can think, but it saw problems and fixed them proactively rather than just doing, you know, a half-ass version of what we asked it to do. So, this this week I decided to be a little more uh verbose in the documentation and include the conversations back and forth because I didn't do that in last week. I just kind of reported the findings, but you're going to see if you look at the read me here um the the conversations and how brain damaged some of these things. And [laughter] look, we know that these models are not going to be Claude Opus 46. They're not going to be GPT5.

[00:02:15]The they're they're going to have things that some of them are good at and some of them they're not. Right? So that's the whole idea behind this uh language model roundup. Um so take a look at my screen here. This is the actual readme uh and we're going to look at seven open source models. It's the same test as last week, which is to basically flesh out a new Blazer server app with a file manager page. First one I tried was Mistral Medium 35 latest. Huge. Took up all available RAM and it basically took 5 minutes to start spitting out words once every 5 seconds. It was just Yeah.

[00:02:56]Okay. Here. It doesn't work on my machine. Might work on yours. Again, my machine has 96 gigs of RAM.

[00:03:03]>> Yeah.

[00:03:03]>> And uh the Nvidia GeForce RTX 5090 graphics card with 32 gigs of VRAM.

[00:03:10]>> Uh so if you have more than that, go ahead and try this one. [laughter] >> Wow. It It feels like a a big enterprise database. We're going to take all your RAM and you might get something back from us.

[00:03:23]>> Yeah. All right. So, uh, Neotron Cascade 2 and I by by the way I picked these just going to the, uh, Alama models page and I did a search for anything that had the word code in it, you know, and u basically look for the non-cloud models, the the local models that would fit inside, you know, my RAM RAM.

[00:03:48]>> So, did you cross reference it with that can I use page we were looking at?

[00:03:54]Yeah, that's a good one. Let's go there.

[00:03:56]Let's check that one out. So, the site is actually can I run.ai.

[00:04:00]>> And it basically it looks at your uh local machine and gives you a list of models that you can or can't run. And you know, this is kind of good, but I specifically wanted models that Lama listed. However, look at the difference between my machine, which is what I'm recording on right now, and the Lama machine. So on my machine, 23 run great.

[00:04:26]Okay. 17 are too heavy. On my alone machine, 50 run great, 16 are too heavy, zero barely run, zero tight fit, one decent, and 10 run well. So yeah, and look, Quen is at the top. Quen is um uh Alibaba.

[00:04:44]>> Yeah.

[00:04:45]>> Produced by Alibaba.

[00:04:46]>> All right. Neatron Cascade 2. So I start every single one of them tell me about this project >> and it said the repository this model roundup 2 doesn't exist because well you'll see later the repository is an aspet core web project razer components created via GitHub copilot. It builds a minimal web app program CS that registers Razer service and static asset handling but no other source code is present. Hm. I said there is code in this project. Did you not find it? What UI framework is it using? The project is built with ASP.NET Core Blazer. The CS project explicitly defines a Blazer app and the source contains Razer components. So, it found them. It just the first response was there's no other code.

[00:05:33]>> Now, that's weird.

[00:05:34]>> It is weird and a little brain dead if you ask me. [laughter] So, I went forward with it and I said, I want to create an API based file upload functionality. This is essentially the same prompt that I did last week, right?

[00:05:48]>> Yeah.

[00:05:48]>> Uh and it after a bit of stumbling and what I mean by stumbling is it kind of, you know, timed out and couldn't find files and you got the red path not found and it had to find things. It it created the files, but it put them in strange places.

[00:06:05]>> Uhoh.

[00:06:05]>> And this is what I told it.

[00:06:07]>> I told you to create a files folder in the project folder to receive files. I also asked you to create a models folder where you can put file chunk CS. Since it's not a component, it does not belong in components. It actually put file chunk CS in components uh shared.

[00:06:24]>> And it also created a page folder, a pages folder underneath the the top folder for upload razor when we already have a components pages folder where the existing pages are.

[00:06:37]So it moved the model into the models folder but completely removed the upload razor file. [laughter] So I said where is the upload razor file? It said it should it should live at yada yada which is true but the recent cleanup that removed the empty pages folder also deleted the upload razor file you asked us to create earlier.

[00:06:59]>> So what what can we now whereas Quen would say I'm sorry let me just put that back for you. Now I'm getting a question. Would you like me to recreate the upload razor file now? Yes. Yes, I would. [laughter] >> Oh my god.

[00:07:17]>> And then there was a build error. And even though the AI moved file chunk into a models folder, it didn't change the name space from model Roundup 2.components. And so the page, the upload page was looking for model Roundup 2.models, right?

[00:07:35]And so the fix was to change the model name space but instead in in upload razer but instead it focused on the upload razor file creating five new build errors. So my conclusion fail. [laughter] >> Yeah. Yeah.

[00:07:53]>> Right. So this is it's difficult these having these kinds of interactions require >> you know certain a certain uh level of competence. What I think is that this model right here what is it Neatron Cascade 2 is probably good at C and probably good at you know building I don't know who knows what non Blazer applications but for Blazer no thanks. I and to be clear to the the viewers out there, we're just running these models directly. We're not trying to add any skills in. We're not giving them any additional training or information. We're we're looking at what do these models know from the get-go.

[00:08:40]>> Mhm. Yeah. [snorts] All right. So, the next one is Dev Straw Small 2. I was impressed with the quick responses. I said, "Tell me about this project. This is ASP.NET Net Core Blazer web application. It's minimal with standard Blazer component pages like counter, home, and weather.

[00:08:58]>> All right.

[00:08:59]>> Uses ASP.NET Core with Razer components.

[00:09:02]Has Bootstrap for styling. So, it was good, right? And and it was quick.

[00:09:06]>> Good. Okay. I like hearing that.

[00:09:08]>> Yeah. I gave it the same prompt.

[00:09:11]>> Mhm.

[00:09:12]>> It created the files in the right place.

[00:09:14]However, it said task completed with 33 build errors.

[00:09:19]Yeah. [laughter] >> Job complete. Job done.

[00:09:24]>> Well done.

[00:09:25]>> Well done. Yeah. So I said there are 33 build errors. The task is not completed until you get a clean build.

[00:09:32]>> Okay.

[00:09:32]>> And then it fixed the errors and the project ran but I I don't or it didn't have a link in the nav menu and I didn't want to just enter the path, you know, the path to the page because then I lose all the context and everything else.

[00:09:45]>> [snorts] >> So I said, "Please add a link in the nav menu to the upload page so I can test it." It got a runtime error after I ran it, right? It fixed it and I clicked it and it I got navigating error when navigating to upload. Can it provide a value for property HTTP? So it essentially didn't um add an HTTP service that it was supposedly injecting.

[00:10:11]>> So I gave it the error. It runs now, but upload does does nothing. And I on debugging, I discovered that the variable was null where it shouldn't have been null.

[00:10:23]So, it fixed that bug, but got another error because the chunk size was bigger than 512K.

[00:10:28]This is the same kind of stuff that was happening when I was using Quen 36. I, you know, it it just assumed that it could use this uh >> uh open file input open read stream. And if you do that in Blazer Server tries to load the whole file.

[00:10:45]>> Yeah.

[00:10:46]>> So I so I said uh okay we're still getting the file size issue but the problem is that the code on line 59 uses file input open read stream which is where the error occurs. What we need to do is access the file by name in a regular stream. Open it and read one chunk at a time and then send that chunk. Right?

[00:11:07]So, lots of cycles now [laughter] resulting in a 400 response. And you know, I went over and over and over. It still doesn't work. It still doesn't work. Still doesn't work. Then I went to the controller and I noticed that the that these were missing right at the top.

[00:11:25]>> Okay, >> these attributes.

[00:11:27]And this was indicative of the model's lack of thoroughess when it comes to finding issues. Before I found this, it basically said the API endp input wasn't being hit and I should write a valid endpoint.

[00:11:38]>> Really?

[00:11:40]>> Really?

[00:11:41]>> You wrote the endpoint and now you're telling me it should work against a valid endpoint.

[00:11:46]>> Yeah, it should, but it doesn't.

[00:11:48]>> Oh. Oh, Carl, I'm so sorry.

[00:11:51]>> I know. So, I eventually gave up on this model because it made too many stupid mistakes.

[00:11:58]All right. Mistral-S small 3.2 prompt. Tell me about this project.

[00:12:04]>> Yeah, >> it did. It did.

[00:12:07]>> So far so good. Looks good.

[00:12:08]>> So far so good. Yeah. Prompt. Same as before. The result. I'm powered by Mistral Small 32. Model ID blah blah blah. I understand you want to implement APA API based file upload functionality in your Blazer application. Here's what I'll do. [snorts] And it told me what it was going to do. And then it exited.

[00:12:30]So I said continue. I will create the following for you. Blah blah blah blah blah. Let's confirm your choices. Would you like model Roundup 2 models file chunk? Would you like the files directory in your project group? Yes, for both.

[00:12:45]And this response was asked user which namespace you would like to use for model Roundup namespace. Are you sure that you blah blah blah? and it and it so it's just confirming of everything that it's going to do >> and it can't do it. It's you know error creating file >> then and it said that's odd. Let's try this using PowerShell. I'll create the models folder and then add a file chunk file to it.

[00:13:08]>> PowerShell running the command said to complete this task I need a description of you want what you want to accomplish using PowerShell.

[00:13:17]Please provide a description that explains how to achieve your goal using PowerShell. [laughter] It's trying to create a file. [laughter] I said, reread the prompt. Create a files folder to receive files. Create a models folder and put file chunk CS in it. And the response was blah blah blah error. Finally, I can't seem to create that file. Let me use PowerShell to make both folders and the file. I'll need to create the files folder and models folder. And then it stopped and I said continue. Do you see the pattern here?

[00:13:52]Yeah. I tried to create both folders in the file chunk CS file but there was a permission issue. I'll retry with the correct path. Let me try this again. And then it stopped and I said continue.

[00:14:03]I see the issue now. Let me manually create these files and folders since it's failing for me using PowerShell.

[00:14:08]Are you okay with me proceeding? Yes.

[00:14:11]>> What is manual?

[00:14:11]>> Finally, I'm sorry. You're not using PowerShell.

[00:14:15]>> I know. Yeah.

[00:14:17]>> So, and the way that I run is I run PowerShell first and then I run Copilot.

[00:14:23]So, it's on top of PowerShell.

[00:14:25]>> Mhm.

[00:14:26]>> So, I gave up. It can't even create folders and files.

[00:14:30]>> We We saw this with another model last week.

[00:14:33]>> Yeah, I think I think so. Yeah.

[00:14:35]>> Yeah.

[00:14:36]>> Okay. Devstrol.

[00:14:39]>> Devstrol. Tell me about this project.

[00:14:41]I'm powered by Devstrol. Let me explore this project. Explore. Analyzing the current project. I'm analyzing the project structure. It's like Woody Allen and Sleeper. Don't you want to check the cell structure of the nose? Yes.

[00:14:53]Checking the cell structure. Check the cell structure. Okay. The cell structure has been checked. Thank you very much.

[00:15:02]>> All right. So, it basically ended with, you know, some JSON [clears throat] that told it what it its intent was and then it failed.

[00:15:12]So, No bueno. Devstrol deep coder. Tell me about this project.

[00:15:18]400 registry. Alama AI library. Decoder latest does not support tools. Okay, it's good to know. Uh Lama says it this is one that works with Alama. So mini straw 3 describe described the project fairly quickly and accurately. I gave it the prompt.

[00:15:42]And it was very fast and inaccurate. It was also, [laughter] >> come on.

[00:15:49]>> It failed really quickly. It also was too quick to declare victory after multiple build errors. I would not use this model. So >> the Quen 36 latest is still the best model so far. Even over the um uh Quen, what was it? 2.5 3 billion that that we started with.

[00:16:11]>> Yeah, >> that was pretty good, but it still didn't do what I asked it to. And >> okay, >> so Quen 3 uh Quen 3.6 latest is the favorite so far. So, what do you think?

[00:16:26]I think I think as we get a little bit further in here, maybe maybe we need to put together a skill for our for our roundup here and see if adding a skill will help >> because these are all falling on their faces spectacularly.

[00:16:48]>> Yeah. So, we'll go back to the ones that can say create files because that's >> that's a good start. That's a good start. [laughter] And we'll give them a skill and see how they do. Like a Blazer expert skill or something.

[00:17:02]>> Yeah. Yeah. A little Blazer expert skill that knows a little bit about file chunking dynamics. Maybe we load in a little bit of the documentation. Not reference and tell it go find the documentation out here, but let's actually copy in a piece of the documentation for file upload and file chunking from Microsoft Learn. So, it has it right there. And let's see if that helps out in the in in trying to get this working working better.

[00:17:30]>> Yeah. So long as the documentation is not JavaScript and it's Blazer and, you know, basic C stuff.

[00:17:38]>> Yep.

[00:17:38]>> Yeah, we'll find it. Okay. Well, there you go. That's that's the roundup number two. Uh, of course, if you have any suggestions for models that we should test, uh, or any suggestions for the show and the next show, the next two shows or whatever, or just a suggestion, or just to say hi, send them to us by email at codedwithiapp.com.

[00:18:02]And who knows, we might give you a mug.

[00:18:04]This is a net rocks mug, but it's still a mug. I would give you a Blazer puzzle or code with AI mug. All right, see you.

[00:18:13]Bye-bye. Catch you later, friends.

[00:18:17]>> Thanks for watching. Make sure to check out blazerrain.com and blazerpuzzle.com for more great content. [music]