Concept card

Tokenization

As of 2026-07-23

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is Tokenization?

Tokenizer choice silently sets a model's effective vocabulary, multilingual performance and API cost — and splitting numbers digit-by-digit is the specific quirk that makes LLM arithmetic unreliable without heavy numeric training data.

What it is

Tokenization is the process of converting raw text into a sequence of integer IDs that an LLM can process. It is not a pre-processing detail — it determines the model's effective vocabulary, its handling of rare words, its multilingual performance, and directly drives API cost.

The core problem it solves is representing an open-ended string space with a fixed-size embedding table while keeping sequence lengths tractable. Character-level models have very long sequences; word-level models fail on out-of-vocabulary tokens and morphological variation.

Three algorithms dominate: Byte-Pair Encoding (BPE, Sennrich 2016) merges the most frequent byte pairs iteratively until the target vocabulary size is reached — GPT-4 uses cl100k with 100,256 tokens. WordPiece (Schuster 2012) maximises log-likelihood of the training corpus and is used by BERT-family models. SentencePiece (Kudo 2018) operates on raw Unicode without pre-tokenisation, making it language-agnostic — Llama 3 uses a 128,256-token SentencePiece vocabulary. Tokenizers are trained separately from the model, then frozen. A critical quirk: most tokenizers split numbers digit-by-digit ('2024' → ['20','24'] or ['2','0','2','4']), making arithmetic brittle unless the model sees abundant numeric training data.

Choose cl100k (tiktoken) when building on OpenAI models; use the model's bundled tokenizer otherwise — never mix tokenizers. Mismatched tokenizers produce silently wrong token counts and budget overruns.

Why it matters

GPT-4's cl100k (100,256 tokens) vs. Llama 3's SentencePiece (128,256 tokens) are not interchangeable — mixing tokenizers produces silently wrong token counts and budget overruns.
SentencePiece's language-agnostic Unicode handling matters directly for multilingual SAP deployments (DACH, EMEA).
The digit-splitting quirk ('2024' → ['20','24']) is a concrete reason to distrust LLM arithmetic in financial-reporting use cases without validation.

Key points

BPE merges frequent byte pairs; the merge table is part of the model artefact and must be shipped with it.
GPT-4 cl100k: 100,256 tokens. Llama 3 SentencePiece: 128,256 tokens. Never assume token counts are interchangeable.
ABAP code tokenises poorly in most English-trained vocabularies — 1 ABAP LOC ≈ 8–15 tokens vs 3–5 for Python.
Numbers are usually split digit-by-digit; instruct models to output numbers as formatted strings when precision matters.
Tiktoken (OpenAI) and HuggingFace tokenizers both produce counts; always use the target model's tokenizer.
Tokenization is mastered only when it changes a named buyer decision.
Start with the semantic contract and control model before demonstrating the tool.
Use current SAP, analyst, study, KG, and news signals as evidence, not decoration.
Separate verified facts from directional trends and modeled assumptions.
Define owner, metric, threshold, support path, and rollback before scaling.

Terms used on this page

BPE: Byte-Pair Encoding: iteratively merges the most frequent adjacent byte pairs in training text to build a subword vocabulary.
Vocabulary: The fixed set of token IDs the model was trained with; out-of-vocabulary inputs are impossible by construction in BPE — every byte is representable.
SentencePiece: Google's tokenizer library that trains a BPE or unigram LM model directly on raw Unicode, requiring no language-specific pre-tokenisation.
cl100k: OpenAI's 100,256-token BPE vocabulary used by GPT-4 and text-embedding-3 models, implemented in the tiktoken library.
Token budget: The maximum number of tokens (prompt + completion) a model can process in one call; exceeding it raises a context-length error.
Decision owner: The accountable person who accepts the trade-off and funds the next action.
Semantic contract: The shared definition of business terms, metrics, entities, and access rules used by tools and teams.
Control plane: The layer that applies policy, access, lineage, monitoring, and escalation across the operating model.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks.

Open in the app →