Models & architecture
Transformer
Definition
The transformer is the neural network architecture behind virtually all modern language models. Its key innovation is self-attention, which lets every token directly consider every other token in the input.
Introduced in 2017, the transformer replaced recurrent architectures that processed text sequentially. Self-attention allows the model to weigh relationships between all positions simultaneously, which both captures long-range dependencies better and parallelises efficiently on GPUs.
That parallelism is arguably the more consequential property: it made training on internet-scale data economically feasible, which is what enabled the capability jumps that followed.
Self-attention costs scale quadratically with sequence length, which is the fundamental reason context windows were historically small and why extending them is an active research area.
Related terms
Large language model (LLM)
A large language model is a neural network trained on vast amounts of text to predict the next token in a sequence. That single objective, at sufficient scale, produces the ability to write, summarise, translate, reason and code.
Context window
The context window is the maximum number of tokens a model can consider at once — your prompt, any attached documents, the conversation history, and the response it generates. Exceed it and the earliest content gets dropped.
Parameters
Parameters are the learned numerical weights inside a model, adjusted during training. Parameter count — 7B, 70B, 405B — is a rough proxy for capacity, but not a reliable predictor of quality.
Attention
Attention is the mechanism that lets a model weigh how much each token should influence the interpretation of every other token. It is how a model resolves what a pronoun refers to, or which clause modifies which noun.
Reasoning model
A reasoning model is trained to generate extended internal deliberation before answering. It trades latency and cost for substantially better performance on maths, logic, coding and multi-step problems.
Multimodal model
A multimodal model accepts or produces more than one type of data — commonly text plus images, and increasingly audio and video. It processes them in a shared representation rather than through separate pipelines.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.