Risks & limitations
Model drift
Definition
Model drift is when a hosted model's behaviour changes over time as the provider updates it, causing prompts that worked reliably to degrade without any change on your side.
Providers update models continuously — safety adjustments, efficiency improvements, capability upgrades. Behaviour on your specific prompts can shift as a result, sometimes subtly.
This is a real operational risk for anything built on a hosted API. A pipeline that has run correctly for months can start failing because an update changed formatting tendencies or how strictly instructions are followed.
Defences: pin specific model versions where the provider offers them, run your eval set on a schedule rather than only after your own changes, and monitor output shape in production. Drift is much cheaper to catch with a failing test than with a user complaint.
Related terms
Evaluation (evals)
Evaluation is systematically measuring AI output quality against a test set, rather than judging from a handful of examples. It is what makes prompt and model changes safe to ship.
Determinism
A deterministic system returns identical output for identical input. Language models are only approximately deterministic even at temperature 0, because floating-point and infrastructure variation introduce small differences.
Open-weights model
An open-weights model has publicly downloadable parameters, letting anyone run it on their own hardware. Llama, Mistral, Qwen and DeepSeek are prominent examples.
Hallucination
A hallucination is model output that is fluent and confident but factually wrong — invented citations, non-existent functions, fabricated statistics. It stems from models being trained to produce plausible text, not verified truth.
Prompt injection
Prompt injection is an attack where malicious instructions hidden in content the model processes override the developer's intended behaviour. It is the most serious unsolved security issue in LLM applications.
Lost in the middle
Lost in the middle is the tendency of models to use information at the start and end of a long context well while under-weighting material in the middle. Position affects whether the model actually uses what you provided.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.