Models & architecture
Attention
Definition
Attention is the mechanism that lets a model weigh how much each token should influence the interpretation of every other token. It is how a model resolves what a pronoun refers to, or which clause modifies which noun.
For each token, attention computes query, key and value vectors. The query is compared against every key to produce weights, and those weights determine how much each token's value contributes to the output. Repeated across many "heads" and layers, this builds increasingly abstract representations.
Different attention heads specialise. Some track syntax, some resolve coreference, some attend to positional patterns. This specialisation is emergent rather than designed.
Attention is also why prompt position matters. Models attend unevenly across long contexts, which produces the well-documented tendency to under-weight material in the middle of a long document.
Related terms
Transformer
The transformer is the neural network architecture behind virtually all modern language models. Its key innovation is self-attention, which lets every token directly consider every other token in the input.
Lost in the middle
Lost in the middle is the tendency of models to use information at the start and end of a long context well while under-weighting material in the middle. Position affects whether the model actually uses what you provided.
Context window
The context window is the maximum number of tokens a model can consider at once — your prompt, any attached documents, the conversation history, and the response it generates. Exceed it and the earliest content gets dropped.
Large language model (LLM)
A large language model is a neural network trained on vast amounts of text to predict the next token in a sequence. That single objective, at sufficient scale, produces the ability to write, summarise, translate, reason and code.
Parameters
Parameters are the learned numerical weights inside a model, adjusted during training. Parameter count — 7B, 70B, 405B — is a rough proxy for capacity, but not a reliable predictor of quality.
Reasoning model
A reasoning model is trained to generate extended internal deliberation before answering. It trades latency and cost for substantially better performance on maths, logic, coding and multi-step problems.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.