Models & architecture
Large language model (LLM)
Definition
A large language model is a neural network trained on vast amounts of text to predict the next token in a sequence. That single objective, at sufficient scale, produces the ability to write, summarise, translate, reason and code.
Modern LLMs are transformer networks with billions to trillions of parameters. Training has two broad phases: pre-training on a very large text corpus to learn language and world knowledge, then post-training (instruction tuning and preference optimisation) to make the model follow instructions and behave helpfully.
The surprising result of the last several years is how much capability emerges from next-token prediction alone. Nothing in the objective mentions reasoning or translation; those abilities appear as consequences of predicting text well at scale.
The framing also explains the failure modes. A model optimised to produce plausible continuations will produce plausible-sounding falsehoods when it lacks knowledge — which is exactly what hallucination is.
Related terms
Transformer
The transformer is the neural network architecture behind virtually all modern language models. Its key innovation is self-attention, which lets every token directly consider every other token in the input.
Hallucination
A hallucination is model output that is fluent and confident but factually wrong — invented citations, non-existent functions, fabricated statistics. It stems from models being trained to produce plausible text, not verified truth.
Parameters
Parameters are the learned numerical weights inside a model, adjusted during training. Parameter count — 7B, 70B, 405B — is a rough proxy for capacity, but not a reliable predictor of quality.
Fine-tuning
Fine-tuning continues training a pre-trained model on your own examples to specialise its behaviour. It is the right tool for teaching consistent style, format or classification behaviour — not for adding knowledge.
Attention
Attention is the mechanism that lets a model weigh how much each token should influence the interpretation of every other token. It is how a model resolves what a pronoun refers to, or which clause modifies which noun.
Reasoning model
A reasoning model is trained to generate extended internal deliberation before answering. It trades latency and cost for substantially better performance on maths, logic, coding and multi-step problems.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.