DeepSeek splits its experts into shared + routed. Same accuracy, 30% less compute.

本站添加: 2026-05-18

[00:00:00]DeepSeek splits its experts into two groups. [music] One always fires, the rest are picked. Same accuracy, 30% less training compute. Most mixture of experts models pick the top eight experts out of 256. Pure routing.

[00:00:13][music] Every token goes through whichever experts the router picks. Pure routing wastes capacity. Basic grammar gets relearned by half the experts.

[00:00:20]>> [music] >> So, DeepSeek splits the pool. One expert is shared, it fires every token and owns the common stuff. The other 255 are routed. The router picks eight. [music] Now, those eight specialize. The shared expert always runs, so your compute floor goes up. Push past two shared and you're [music] sliding back to a dense model. So, when you compare mixture of experts checkpoints, the shared two routed split is the design lever. Pure top K like Mistral is the old shape.

[00:00:47]Shared plus routed is the new one. One always fires, eight get picked, 30% [music] cheaper.