Neural Network Scaling: Depth vs. Width Strategy

Added: 2026-05-28

[00:00:00]The depth curve kind of goes like this.

[00:00:01]It jumps up pretty fast. That's like present throughout our paper. For width, it grows a little bit more slowly. And so that the kind of take away from that is that if you are a bit more resource constrained, scaling along depth might be better cuz there's fewer parameters with a smaller model to a smaller number to a learnable parameters. Width is expensive. Width is expensive, exactly.

[00:00:20]And in general, of course, like more parameters is also going to be more expensive. So that's just like another consideration it's to think about when using these networks, I suppose.

[00:00:28]>> Yeah.