Why Small Language Models Matter for Deterministic Output

Ajouté : 2026-05-17

[00:00:00]The reason why turn the temperature down to zero is not necessarily a guaranteed solution to end all stochasticity is that when the LLMs are executing inference in the back end, it's it can be spread over many machines cuz the models are too large to run on a single box. And then when you're getting the results, the serial order of those results for a single inference pass could be out of order. When you get out of order results for the inference results, that means that the answer can change.

[00:00:30]And a bunch of other things. Now, if you have a small enough model like an SLM that runs on a single node and you turn the temperature all the way down, you can probably guarantee for certain cases that you will get deterministic output. But you can't rely on this in a general sense. You can absolutely try to turn the temperature down to zero.

[00:00:51]That doesn't work, right?