GLOSSARY > TOKENIZATION
Tokenization
The process of breaking down raw text into smaller discrete sub-word units (tokens) that a machine learning model can mathematically process.
Tokenization bridges the gap between human language and mathematical vectors. Understanding token economics is critical for managing LLM inference costs and context window limits. Efficient tokenizers can compress information density, allowing models to process larger documents with less compute drag.
Explore More