Models & architecture
Quantization
Definition
Quantization reduces the numerical precision of model weights — from 16-bit to 8-bit or 4-bit — shrinking memory use and speeding inference at a modest cost in quality.
A 70-billion-parameter model at 16-bit precision needs roughly 140GB of memory, beyond any consumer GPU. Quantized to 4-bit it needs around 40GB, which brings it within reach of high-end workstations.
Quality loss is smaller than intuition suggests. 8-bit quantization is often indistinguishable from full precision. 4-bit is usually acceptable for general use with some degradation on precise reasoning. Below 4-bit, degradation becomes noticeable.
This is the technology that makes local model running practical, and it is why you see model files labelled Q4, Q5, Q8 and similar.
Related terms
Parameters
Parameters are the learned numerical weights inside a model, adjusted during training. Parameter count — 7B, 70B, 405B — is a rough proxy for capacity, but not a reliable predictor of quality.
Open-weights model
An open-weights model has publicly downloadable parameters, letting anyone run it on their own hardware. Llama, Mistral, Qwen and DeepSeek are prominent examples.
Inference cost
Inference cost is what you pay to run a model on a request, almost always billed per million input and output tokens. Output tokens typically cost three to five times more than input tokens.
Large language model (LLM)
A large language model is a neural network trained on vast amounts of text to predict the next token in a sequence. That single objective, at sufficient scale, produces the ability to write, summarise, translate, reason and code.
Transformer
The transformer is the neural network architecture behind virtually all modern language models. Its key innovation is self-attention, which lets every token directly consider every other token in the input.
Attention
Attention is the mechanism that lets a model weigh how much each token should influence the interpretation of every other token. It is how a model resolves what a pronoun refers to, or which clause modifies which noun.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.