Models & architecture
Parameters
Definition
Parameters are the learned numerical weights inside a model, adjusted during training. Parameter count — 7B, 70B, 405B — is a rough proxy for capacity, but not a reliable predictor of quality.
A model with 7 billion parameters has 7 billion adjustable numbers determining how it transforms input into output. More parameters mean more capacity to store patterns and knowledge, and more compute needed to train and run the model.
Parameter count has become a weak quality signal. Training data quality, data volume, and post-training technique matter enormously, and well-trained smaller models routinely outperform larger, older ones. A modern 8B model can beat a two-year-old 70B model on many tasks.
For self-hosting, parameter count directly determines memory requirements. A rough guide at common quantisation levels: multiply parameter count in billions by roughly 0.5-1 to get required VRAM in gigabytes.
Related terms
Large language model (LLM)
A large language model is a neural network trained on vast amounts of text to predict the next token in a sequence. That single objective, at sufficient scale, produces the ability to write, summarise, translate, reason and code.
Quantization
Quantization reduces the numerical precision of model weights — from 16-bit to 8-bit or 4-bit — shrinking memory use and speeding inference at a modest cost in quality.
Open-weights model
An open-weights model has publicly downloadable parameters, letting anyone run it on their own hardware. Llama, Mistral, Qwen and DeepSeek are prominent examples.
Fine-tuning
Fine-tuning continues training a pre-trained model on your own examples to specialise its behaviour. It is the right tool for teaching consistent style, format or classification behaviour — not for adding knowledge.
Transformer
The transformer is the neural network architecture behind virtually all modern language models. Its key innovation is self-attention, which lets every token directly consider every other token in the input.
Attention
Attention is the mechanism that lets a model weigh how much each token should influence the interpretation of every other token. It is how a model resolves what a pronoun refers to, or which clause modifies which noun.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.