Concept card

Token Economics

As of 2026-10-10

Edited by Cédric Mary, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is Token Economics?

Prompt caching is the biggest cost lever most SAP teams miss — a 2,000-token Joule system prompt called 500 times a day saves ~$2,700/year per agent, a real saving that multiplies across 20 agents.

Token economics is the discipline of understanding how API cost, latency, and throughput trade off across model tiers, deployment modes, and prompt design choices. It is what separates an SAP AI integration with a predictable, controlled cost base from one that quietly burns budget with nobody able to explain why the invoice looks the way it does.

The core problem

Inference cost is non-linear and highly sensitive to how a prompt is built. The same business question, asked with a lean, well-structured five-hundred-token prompt versus a bloated ten-thousand-token prompt stuffed with unnecessary context, can differ by an order of magnitude in cost — and the more expensive call is not automatically the more accurate one. Most teams discover this only after the first large invoice, because token cost is invisible at design time and only becomes visible in production usage.

Why output costs more than input

Every major provider prices output tokens several times higher than input tokens, and the reason is architectural rather than commercial. Input tokens can be processed in parallel across the whole prompt in a single forward pass. Output tokens cannot: a language model generates text autoregressively, one token at a time, where each new token depends on every token generated before it, so the computation cannot be batched the same way. That sequential dependency is also why output generation dominates latency — a long input processes almost instantly relative to a long output, which has to be produced token by token.

Why it matters

Output tokens cost 3-6× more than input tokens across all providers — a structural reason to keep generated output terse in high-volume Joule flows.
Batch APIs cut cost 50% for anything that doesn't need to render in under 5 seconds — nightly report generation and bulk contract summarisation qualify; real-time Joule chat doesn't.
Self-hosted Llama 3.3 70B at ~$0.009/M tokens (60% utilisation) reframes the build-vs-buy decision for very high-volume workloads.

Key points

Output tokens cost 3–6× more than input tokens — minimise completion length before optimising prompt length.
Prompt caching (OpenAI: 50% off ≥1024-token repeated prefixes; Anthropic: $0.30/M read vs $3.00/M uncached) pays off at >26 reuses.
Self-hosted Llama 3.3 70B on H100: ~$0.009/M tokens amortised vs $2.50/M for GPT-4o — 278× cheaper at scale.
Gemini 2.0 Flash at $0.10/$0.40 per M tokens is the cost frontier for high-volume extraction tasks as of Q2 2026.
Batch APIs (OpenAI, Anthropic) offer 50% discount for non-real-time workloads — use for nightly SAP report generation.
A mixed architecture (Sonnet for reasoning, Haiku for extraction) cuts cost by 60-80% versus a single frontier tier.
Progressive summarisation over a 40-turn agent run: a 100K-token context brought under 8K saves about 92% on follow-up calls.
A mandatory token budget per agent and per use case, built into SAP AI governance from design time.
SAP Joule system prompts run 4,000-8,000 tokens before any user input — adding 2,000 tokens of business rules doubles the input cost.
The cheapest lever is the call that never happens: RAG plus pruning of tool results before any model optimisation.

Terms used on this page

Prompt caching: Provider-side mechanism that re-uses the KV cache from a repeated prompt prefix, charging a lower read rate than a full re-computation.
Batch API: Asynchronous inference endpoint (OpenAI, Anthropic) that accepts up to 50k requests, processes overnight, and charges 50% of synchronous pricing.
Input token: A token in the prompt (system + user turn + tool definitions); billed per million at the provider's input rate.
Output token: A token generated by the model; billed at the provider's output rate, typically 3–6× the input rate.
Token budget: Maximum tokens (prompt + completion) a model processes per call; exceeding it raises an error.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →