Running a trained model on new input to get a prediction or generated output, the stage where a model does its everyday work.
Model Inference is the stage in which a trained model is put to use. Once a model has been fully trained, it is deployed into production. When a user or system sends it a prompt or data point, the model computes and returns a result. That computation is called inference.
Inference needs substantial computing power, delivered in real time, so optimizing the infrastructure horizon for low-latency inference is a central challenge in enterprise AI architecture.
Organizations often deploy dedicated on-premises GPU clusters for intensive inference workloads to avoid high, variable public cloud fees.
See also Machine Learning, On-Premises, Large Language Model, Fine-Tuning and MLOps.