Tokenization

What it is

Tokenization is the process of converting raw text into a sequence of integer IDs that an LLM can process. It is not a pre-processing detail — it determines the model's effective vocabulary, its handling of rare words, its multilingual performance, and directly drives API cost.

The core problem it solves is representing an open-ended string space with a fixed-size embedding table while keeping sequence lengths tractable. Character-level models have very long sequences; word-level models fail on out-of-vocabulary tokens and morphological variation.

Three algorithms dominate: Byte-Pair Encoding (BPE, Sennrich 2016) merges the most frequent byte pairs iteratively until the target vocabulary size is reached — GPT-4 uses cl100k with 100,256 tokens. WordPiece (Schuster 2012) maximises log-likelihood of the training corpus and is used by BERT-family models. SentencePiece (Kudo 2018) operates on raw Unicode without pre-tokenisation, making it language-agnostic — Llama 3 uses a 128,256-token SentencePiece vocabulary. Tokenizers are trained separately from the model, then frozen. A critical quirk: most tokenizers split numbers digit-by-digit ('2024' → ['20','24'] or ['2','0','2','4']), making arithmetic brittle unless the model sees abundant numeric training data.

Choose cl100k (tiktoken) when building on OpenAI models; use the model's bundled tokenizer otherwise — never mix tokenizers. Mismatched tokenizers produce silently wrong token counts and budget overruns.

Why it matters

• GPT-4's cl100k (100,256 tokens) vs. Llama 3's SentencePiece (128,256 tokens) are not interchangeable — mixing tokenizers produces silently wrong token counts and budget overruns.
• SentencePiece's language-agnostic Unicode handling matters directly for multilingual SAP deployments (DACH, EMEA).
• The digit-splitting quirk ('2024' → ['20','24']) is a concrete reason to distrust LLM arithmetic in financial-reporting use cases without validation.

Key points

• BPE merges frequent byte pairs; the merge table is part of the model artefact and must be shipped with it.
• GPT-4 cl100k: 100,256 tokens. Llama 3 SentencePiece: 128,256 tokens. Never assume token counts are interchangeable.
• ABAP code tokenises poorly in most English-trained vocabularies — 1 ABAP LOC ≈ 8–15 tokens vs 3–5 for Python.
• Numbers are usually split digit-by-digit; instruct models to output numbers as formatted strings when precision matters.
• Tiktoken (OpenAI) and HuggingFace tokenizers both produce counts; always use the target model's tokenizer.
• A tokenizer is trained once and frozen with its model — swapping model families (e.g. GPT-4 to Llama 3) invalidates every prior token-cost estimate and requires re-benchmarking from scratch, not a proportional adjustment.
• German compound nouns and other agglutinative constructions tokenize less efficiently than their English equivalents under most vocabularies trained on English-skewed corpora — a DACH-market deployment should benchmark German-language traffic separately, never extrapolate from an English-only test.
• BPE's byte-level fallback guarantees every input string is tokenizable — nothing is ever truly out-of-vocabulary, just potentially very inefficient to encode.

Terms used on this page

BPE

Byte-Pair Encoding: iteratively merges the most frequent adjacent byte pairs in training text to build a subword vocabulary.

Vocabulary

The fixed set of token IDs the model was trained with; out-of-vocabulary inputs are impossible by construction in BPE — every byte is representable.

SentencePiece

Google's tokenizer library that trains a BPE or unigram LM model directly on raw Unicode, requiring no language-specific pre-tokenisation.

cl100k

OpenAI's 100,256-token BPE vocabulary used by GPT-4 and text-embedding-3 models, implemented in the tiktoken library.

Token budget

The maximum number of tokens (prompt + completion) a model can process in one call; exceeding it raises a context-length error.

Sources

1. Sennrich et al. 2016 — Neural Machine Translation of Rare Words with Subword Units (BPE)
2. Kudo & Richardson 2018 — SentencePiece
3. OpenAI tiktoken documentation
4. OpenAI tokenizer playground
5. Meta AI — Llama 3 Model Card
6. HuggingFace tokenizers library
7. SAP Business AI — official product page
8. SAP Generative AI — official product page
9. Hugging Face — Tokenizer summary: BPE, WordPiece and SentencePiece compared
10. Google — SentencePiece: official repository and documentation
11. SAP Help Portal — Generative AI hub in SAP AI Core: model access overview
12. SAP Help Portal — Metering and pricing for generative AI on SAP AI Core