Splitting text into tokens, the word pieces a language model reads and counts, before the model processes it.
Tokenization is the first step in turning human language into the vectors a model computes on. Token counts determine LLM inference costs and how much text fits in a context window. Efficient tokenizers represent the same text in fewer tokens, so models can process longer documents with less compute.
See also Context Window, Large Language Model, Fine-Tuning and Model Inference.