The process of using a trained model to generate outputs from new inputs. It is the phase that consumes computational resources in production and defines operational cost.
Detailed explanation
Inference is the phase where an already-trained AI model processes new inputs and generates outputs. Unlike training (which happens once), inference happens on every call in production, and is what defines the operational cost of an AI product. Inference cost depends on three factors: model size, volume of tokens processed (input + output), and provider pricing. ZML released LLMD, a free multi-chip inference tool that reduces hardware lock-in. For product managers, optimizing inference — choosing the right model, reducing context, and routing intelligently — is the primary AI FinOps lever.
How to use it in product decisions
Manage inference as a service chain, not merely a price per million tokens. Include queues, cache, network, tools, retries, and human review in the latency and cost users perceive. Define an SLO for each use case and route simple tasks to smaller models when evaluations allow it. Monitor tokens, time to first token, total duration, errors, and cost per completed task. The best optimization preserves quality and predictability, especially during peaks and long-tail cases.