A technique to train a smaller model from the outputs of a larger model, preserving capability for a specific task at a fraction of the inference cost.
Detailed explanation
Model distillation is a technique where a smaller model ("student") is trained to mimic the outputs of a larger, more capable model ("teacher"). The result is a model that performs comparably to the larger model on a specific task, but at a fraction of the inference cost. Shopify demonstrated this in production: their distillation pipeline cut AI costs by up to 30x, and in several cases the smaller model performed equal to or better than the generic large model. The principle is that large models excel at generalist tasks, but when the task is specific and well-defined, a distilled model for that domain is more efficient in cost, latency, and even quality. For PMs, distillation is one of the most powerful AI FinOps levers.
How to use it in product decisions
Use distillation when the task is stable, repetitive, and supported by enough quality examples. Before replacing a general model with a smaller one, create an evaluation set with common cases, tail cases, and expensive failures. Compare quality, latency, cost, and maintenance, not only average accuracy. Keep a fallback route to the larger model and monitor distribution shifts. The gain is real only when the distilled model preserves the outcome users perceive and reduces total operating cost.