Llama Made Crystal Clear · chapter 1: What Llama Actually Is
Dense versus mixture-of-experts, using Scout's own active-versus-total parameter split
2026-09-07
Below: the paragraph from the book that builds this idea, then the diagram itself (Figure 1.2), and a recap. About a minute of reading.
Figure 1.2 puts the two architectures side by side. Notice that the dense model on the left lights up every block, while the mixture-of-experts model on the right lights up only a couple of its sixteen expert blocks per token, even though all sixteen are sitting there, fully trained and ready.
Recap
- The idea: Dense versus mixture-of-experts, using Scout's own active-versus-total parameter split.
- The picture: Figure 1.2, from chapter 1 ("What Llama Actually Is") of Llama Made Crystal Clear.
- Go deeper: the chapter builds this step by step, with recipes and sources at the end.
This diagram is one of many in Llama Made Crystal Clear.
Every chapter opens with the gist, draws the hard ideas, and ends with recipes and sources.
Get the book