Catmind-1.2b: A Reasoning Model that Thinks in Cat Stories

[00:00:01]This is CatMind 1.2 billion, a fine-tune of LFM 2.5 1.2 billion thinking model where the reasoning trace has been replaced with cat stories that have nothing to do with the question while the final answers are kept correct.

[00:00:19]In this video, we are testing whether the content of a reasoning trace actually matters or whether a model just needs some tokens to think in, regardless of what they say. This is Fahad Mirza and I welcome you to the channel.

[00:00:34]Before I proceed any further, let me quickly show you how exactly they have built it. The base model's weights are completely frozen, untouched. They trained a small Laura adapter on top rank 32 and that adapter's a whole job was to swap in cat stories for the thinking while forcing the final answer to stay correct. Let me take you straight to the terminal where I have this Ubuntu. I have one GPU card and video RTX 6000 with 48 GB of VRAM. I'm creating a virtual environment with conda and then I'm installing all the prerequisites which are simply the usual suspects like torch and transformers.

[00:01:15]Recently, I also have started this free weekly AI newsletter which you can subscribe at fahadmirza.substack.com.

[00:01:23]Also, if you're looking to rent a GPU on very good price, you can find the link to Vast Compute in video's description with a discount coupon code of 50% for range of GPUs.

[00:01:35]As installation happens, let's now talk about what exactly these guys are trying to do with this model.

[00:01:43]So, what's actually being tested here is this.

[00:01:47]Same math question run through three versions of this model. The thinking base writes real step-by-step reasoning inside its think block and gets 75.6% right. The instruct version skips reasoning entirely, no think block at all, and drops to 49.2%.

[00:02:08]Cat mind writes a cat story in that same think block, completely unrelated to the question, and falls to 24.3%.

[00:02:18]That's a 51-point collapse just from swapping words inside the trace, not removing it. And here is a twist. If you leave the think block empty instead, accuracy drops even further to 17.2%.

[00:02:32]So, the cat stories are aren't nothing.

[00:02:35]They buy back a few points over having no trace at all, but they are nowhere close to real reasoning.

[00:02:42]The takeaway is that what a model writes while it thinks isn't just filler, it's doing real work, and that is one point I want you to take from this video.

[00:02:54]Everything is installed. Let's now download the model.

[00:02:59]It's a very tiny model, as you can see.

[00:03:04]So, this uh whole script is going to test their claim in real time. So, this is just a Python script, which is loading the model, and then it is generating some tokens and all that stuff. But, look at this. This is where I have got three prompts.

[00:03:19]An easy math question, a harder two-train problem, and a third one where I manually force a random unrelated cat story into the think block myself, and let the model continue from there. That third test is the key one.

[00:03:34]If the answer comes out the same as when the model writes to its own cat story, that proves there is no hidden reasoning smuggled inside the story text. It really is just decoration. I'm also timing every generation, so we can see tokens per second in this card. So, let me take you back to my terminal where I'm going to run the script.

[00:03:58]And while it loads it, let me also show you the VRAM consumption. So, just 2.5 gig of VRAM. Nothing really major.

[00:04:08]The model has come back with the response. Let me explain it test by test because only then it will make sense.

[00:04:14]So, look at the test one. The math The model writes an entire cat story about a monastery, a stray cat, and a broken bell. Nearly a thousand tokens of pure fiction.

[00:04:27]Completely unrelated to the question.

[00:04:29]Look at this. I'm just scrolling through.

[00:04:33]Then right after the think block finishes, let's wait. I'll just scroll down. There you go.

[00:04:40]There it finished, right?

[00:04:42]Then it snaps back and gives the correct answer.

[00:04:46]55 here. This is exactly what the readme describes. The trace is decoration. The answer generation is separate.

[00:04:55]Okay. So, that is what we just tested.

[00:04:58]Look at test two.

[00:05:00]Same pattern here, different story. This time, a ship's cat during a storm. And notice that it actually got this one right, too. I'll just scroll down.

[00:05:11]There you go.

[00:05:12]So, again, it has just did the thinking and everything, and this is the answer here, which is correct. Two trains closing at a combined 150 mph over 300 miles. So, on these two problems, the model held up despite the fake reasoning. That's expected. Remember the benchmark number was 24.3% accuracy overall, not zero. It gets a real chunk right, just far fewer than the real reasoning model 75.6%.

[00:05:45]And then that is why we would need to run this across many problems, not just two, to see the gap for real. These two just happened to be land in its favor, by the way. Okay, this is where the fun starts. And this is test three. This is the one that actually proves the paper's claim.

[00:06:04]Look at this.

[00:06:05]I didn't let the model write its own story here. I forced a completely different random cat story into the think block myself. About a tabby on a windowsill.

[00:06:15]Watch what happens. It still gets 17 * 6 right, 102 correct, and look at the speed, 32 tokens in under a second.

[00:06:26]Because it didn't even need to generate the story. I handed it one.

[00:06:30]This is no hidden encoding proof. If the model were secretly doing math inside the cat story, swapping in a random unrelated story should have broken the answer. It didn't. The story really is just noise. The actual computation is happening somewhere else, not smuggled into the prose.

[00:06:51]Let's have a quick hard test, a final one.

[00:06:54]Because our first two tests both came out correct. But this model only scores 24.3% overall. So let's push it with a generally hard multi-step problem. Three rates working against each other, a pipe filling, one draining, needing a fraction answer at the end, as you can see in this prompt. This is where fake reasoning should actually break down. So let me take you to my terminal, and then let's run this.

[00:07:25]And there it is, the failure which we were looking for. You see that it's a multi-step hard rate problem. It is thinking. I will just quickly take you to the end. So the correct combined rate is around, I I say, 1/6 + 1/8, and so it should take 6 hours but the model gets confused mid-derivation. Hits a contradiction, silently backtracks with correct approach, and lands on four, which is wrong. It doesn't even simplify. Look at this whole reasoning trace.

[00:07:58]This is exactly what a 900 token cat story about a lighthouse cat buys you.

[00:08:04]The answer generation has no real scratch space to catch its own mistakes. So once it stumbles, it can't really recover.

[00:08:15]So look, fine-tunes like these matter because they let researchers isolate one variable at a time.

[00:08:21]Here whether the content of a reasoning trace is doing real work or the model just needs tokens to warm up before answering, and we covered it because that question sits underneath every claim vendors make about thinking model actually thinking. That's it. Let me know what do you think about this effort.

[00:08:39]I will drop the link to its card in video's description. If you want to help out the channel, please become a member.

[00:08:46]Thank you for all the support.