Models & architecture
Mixture of experts (MoE)
Definition
Mixture of experts is an architecture where only a subset of the model activates per token. A router selects a few specialised sub-networks, so a very large model runs at the cost of a much smaller one.
In a dense model every parameter participates in every token. In an MoE model, layers contain many "expert" sub-networks and a learned router activates only a handful for each token.
The result is a favourable trade: total parameter count can be very large while active parameters per token stay modest. A model with 400 billion total parameters might activate only 30 billion per token, giving the knowledge capacity of the former at closer to the inference cost of the latter.
The catch is memory. All experts must be loaded even though few are used per token, so MoE models are cheap to run but expensive to host — which matters most for self-hosting decisions.
Related terms
Parameters
Parameters are the learned numerical weights inside a model, adjusted during training. Parameter count — 7B, 70B, 405B — is a rough proxy for capacity, but not a reliable predictor of quality.
Transformer
The transformer is the neural network architecture behind virtually all modern language models. Its key innovation is self-attention, which lets every token directly consider every other token in the input.
Inference cost
Inference cost is what you pay to run a model on a request, almost always billed per million input and output tokens. Output tokens typically cost three to five times more than input tokens.
Quantization
Quantization reduces the numerical precision of model weights — from 16-bit to 8-bit or 4-bit — shrinking memory use and speeding inference at a modest cost in quality.
Large language model (LLM)
A large language model is a neural network trained on vast amounts of text to predict the next token in a sequence. That single objective, at sufficient scale, produces the ability to write, summarise, translate, reason and code.
Attention
Attention is the mechanism that lets a model weigh how much each token should influence the interpretation of every other token. It is how a model resolves what a pronoun refers to, or which clause modifies which noun.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.