The process of reducing the precision of a model's weights (e.g., from 32-bit floats to 8-bit integers) to shrink its memory footprint and accelerate inference.
Model Quantization makes it practical to deploy Large Language Models at the edge or on constrained On-Premises hardware. Compressing the weights to lower precision, often with techniques such as GPTQ or AWQ, lets the model use a fraction of the VRAM while keeping accuracy close to the original.
See also On-Premises, Neural Network, Private AI, Large Language Model, Fine-Tuning and Model Inference.