Risks & limitations
Bias
Definition
Bias in AI is systematic skew in outputs that reflects patterns in training data or training process — including demographic stereotypes, cultural assumptions, and over-representation of dominant viewpoints.
Models learn from human-generated text, which contains human biases. Without deliberate intervention, models reproduce and can amplify them: associating occupations with genders, defaulting to Western cultural assumptions, or performing worse in under-represented languages.
Post-training mitigation reduces the most overt cases but does not eliminate subtler forms. Bias in what a model considers a "normal" example, or in the quality gap between languages, persists.
For applications touching hiring, lending, healthcare or legal decisions, this is a compliance matter, not just an ethical one. Test outputs across demographic variations of the same input and measure whether results differ when they should not.
Related terms
Evaluation (evals)
Evaluation is systematically measuring AI output quality against a test set, rather than judging from a handful of examples. It is what makes prompt and model changes safe to ship.
Guardrails
Guardrails are the constraints that keep an AI system inside acceptable behaviour — input validation, output filtering, topic restrictions, action limits and human approval gates.
Hallucination
A hallucination is model output that is fluent and confident but factually wrong — invented citations, non-existent functions, fabricated statistics. It stems from models being trained to produce plausible text, not verified truth.
Prompt injection
Prompt injection is an attack where malicious instructions hidden in content the model processes override the developer's intended behaviour. It is the most serious unsolved security issue in LLM applications.
Lost in the middle
Lost in the middle is the tendency of models to use information at the start and end of a long context well while under-weighting material in the middle. Position affects whether the model actually uses what you provided.
Model drift
Model drift is when a hosted model's behaviour changes over time as the provider updates it, causing prompts that worked reliably to degrade without any change on your side.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.