GLOSSARY > MODEL QUANTIZATION

Model Quantization

The process of reducing the precision of a model's weights (e.g., from 32-bit floats to 8-bit integers) to shrink its memory footprint and accelerate inference.

Model Quantization is essential for deploying Large Language Models at the edge or on constrained On-Premises hardware. By mathematically compressing the neural network weights—often via techniques like GPTQ or AWQ—the model consumes a fraction of the VRAM while retaining near-original accuracy. This technique is a cornerstone of modern Private AI mobilization.

Explore More

Related Terms

Large Language Model

A deep learning algorithm that can recognize, summarize, translate, predict, and generate human language.

Read Definition

Fine-Tuning

The process of taking a pre-trained foundational AI model and training it further on a highly specific, proprietary dataset.

Read Definition

Model Inference

The live phase where a trained AI model processes new, unseen data to make predictions or generate outputs.

Read Definition