Models & architecture
LoRA
Definition
LoRA (Low-Rank Adaptation) is a fine-tuning method that trains small adapter matrices instead of updating all model weights. It cuts the compute and storage cost of fine-tuning by orders of magnitude.
Rather than modifying billions of parameters, LoRA inserts small low-rank matrices alongside existing weights and trains only those. The resulting adapter is typically megabytes rather than gigabytes.
This has practical consequences beyond cost. Adapters can be swapped at runtime, so one base model can serve many specialised behaviours. You can maintain separate adapters per customer or per task without hosting separate models.
Quality is usually close to full fine-tuning for style and format adaptation, which covers most real use cases.
Related terms
Fine-tuning
Fine-tuning continues training a pre-trained model on your own examples to specialise its behaviour. It is the right tool for teaching consistent style, format or classification behaviour — not for adding knowledge.
Parameters
Parameters are the learned numerical weights inside a model, adjusted during training. Parameter count — 7B, 70B, 405B — is a rough proxy for capacity, but not a reliable predictor of quality.
Quantization
Quantization reduces the numerical precision of model weights — from 16-bit to 8-bit or 4-bit — shrinking memory use and speeding inference at a modest cost in quality.
Large language model (LLM)
A large language model is a neural network trained on vast amounts of text to predict the next token in a sequence. That single objective, at sufficient scale, produces the ability to write, summarise, translate, reason and code.
Transformer
The transformer is the neural network architecture behind virtually all modern language models. Its key innovation is self-attention, which lets every token directly consider every other token in the input.
Attention
Attention is the mechanism that lets a model weigh how much each token should influence the interpretation of every other token. It is how a model resolves what a pronoun refers to, or which clause modifies which noun.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.