Cost & performance
Inference cost
Definition
Inference cost is what you pay to run a model on a request, almost always billed per million input and output tokens. Output tokens typically cost three to five times more than input tokens.
The formula is straightforward: (input tokens ÷ 1,000,000 × input rate) + (output tokens ÷ 1,000,000 × output rate). What surprises teams is how fast it compounds at volume — a cost per call that looks negligible becomes significant across a million calls.
The largest available savings are usually architectural rather than negotiated. Route simple tasks to small models; most workloads do not need a flagship. Cache repeated context. Trim retrieved chunks to what is actually needed. Cap output length explicitly.
Because output is far more expensive than input, asking for concise responses is often the single highest-leverage cost change available.
Related terms
Token
A token is the unit of text an AI model reads and writes — roughly four characters or three-quarters of a word in English. Models price by the token and cap context by the token, so token count determines both what you pay and what fits.
Prompt caching
Prompt caching stores the processed form of a repeated prompt prefix so subsequent requests reusing it are cheaper and faster. Cached input tokens typically cost a fraction of normal input tokens.
Quantization
Quantization reduces the numerical precision of model weights — from 16-bit to 8-bit or 4-bit — shrinking memory use and speeding inference at a modest cost in quality.
Reasoning model
A reasoning model is trained to generate extended internal deliberation before answering. It trades latency and cost for substantially better performance on maths, logic, coding and multi-step problems.
Latency
Latency is the delay before and during a response. Two measures matter: time to first token, which governs perceived responsiveness, and tokens per second, which governs how fast the rest arrives.
Streaming
Streaming sends a response token by token as it is generated rather than waiting for completion. It does not reduce total generation time but dramatically improves how fast the response feels.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.