The process of breaking down raw text into smaller discrete sub-word units (tokens) that a machine learning model can mathematically process.
Tokenization is the first step in turning human language into the vectors a model computes on. Token counts determine LLM inference costs and how much text fits in a context window. Efficient tokenizers represent the same text in fewer tokens, so models can process longer documents with less compute.
See also Context Window, Large Language Model, Fine-Tuning and Model Inference.