LLM Cost Engineering on SAP — AI Units, Token Budgets, Caching
As of 2026-09-25
What is LLM Cost Engineering on SAP?
C128 and C333 explain how SAP attributes and prices generative AI spend; this card is the practitioner's toolkit for shrinking it — explicit prompt caching (Anthropic Claude and Amazon Nova only, five-minute default TTL), batch consumption for bulk native LLM calls at a documented lower cost than synchronous calls, and the token-budget and rate-limit settings that turn an open-ended model call into a bounded one.
Where cost engineering sits next to cost attribution
C128 covers how SAP AI Core spend gets attributed — tokens to GenAI tokens to capacity units, resource-group grain, the labels you add for chargeback. C333 covers the commercial layers Joule and embedded AI sit in — AI Units, per-agent-action pricing. Neither is about shrinking the bill itself; that is what this card does. The metering chain both cards describe is still the starting point: input and output tokens (output priced slightly higher) convert to GenAI tokens at model-specific rates SAP documents in SAP Note 3437766, and GenAI tokens become the capacity units on the invoice. Every lever below acts somewhere on that chain — by avoiding tokens entirely (caching), by avoiding the synchronous premium (batching), or by bounding the tokens a call is allowed to spend (budgets and limits).
Why it matters
- A RAG pipeline with a long, static system prompt and no cache_control is paying full price on that prefix every single call — for Claude or Nova models, that is a discount left entirely on the table.
- Routing a nightly bulk-classification job through orchestration instead of batch consumption both costs more per SAP's own documentation and adds client-side rate-limit handling that batch consumption doesn't require.
- An unset max_completion_tokens is not a safe default — it silently authorizes the model's own maximum, which is exactly the kind of detail that turns a pilot's token estimate into a production invoice surprise.
Key points
- Metering chain (shared with C128): input/output tokens -> GenAI tokens (model-specific rate, SAP Note 3437766) -> capacity units billed.
- Explicit prompt caching (cache_control) works only for Anthropic Claude (up to 4 breakpoints: tools, system, messages) and Amazon Nova (system/messages, ~1 breakpoint). OpenAI/Gemini get implicit caching only, unconfigurable.
- Default cache TTL is 5 minutes; select Anthropic models support an optional 1-hour TTL.
- Batch consumption is documented as cheaper than synchronous calls, retries automatically, and needs no client-side rate-limit handling — but supports native LLM calls only, not orchestration (no grounding/masking/filtering).
- Batch consumption is EU/US only, excluding prod-euonly and sovereign cloud, and requires an object store secret (S3, Azure Blob, GCS, Alibaba OSS, or HANA Cloud Data Lake).
- RPM rate limits are per model per tenant; a 429 costs latency and retry risk, not extra billing per se — raising the limit does not lower spend.
- max_completion_tokens is mandatory for Anthropic models; left unset for any model, orchestration applies that model's own maximum.
- Fallback lists (config.modules as an array) handle 408/429/5xx on non-streaming calls by switching model instead of retrying blindly.
Terms used on this page
- GenAI token
- A virtual metering unit representing model-specific token usage, used to compute capacity-unit billing (SAP Note 3437766).
- cache_control
- The orchestration V2 field marking a prompt block as a cache breakpoint, supported only for Anthropic Claude and Amazon Nova models.
- Implicit caching
- Default, unconfigurable context caching enabled for OpenAI and Gemini models in orchestration.
- Batch consumption
- Asynchronous processing of many native LLM requests from a file, documented as cheaper than synchronous calls but unavailable for orchestration requests.
- RPM limit
- A per-model, per-tenant cap on Requests Per Minute; exceeding it returns HTTP 429.
- Fallback list
- config.modules passed as an array so orchestration switches models on an unsupported region or, without streaming, on 408/429/5xx.
Sources
- SAP AI Core docs (SAP-docs GitHub, Sep 2026) — Prompt Caching (cache_control, supported models, TTL, response fields)
- SAP AI Core docs — Batch Consumption (native-LLM-only, regions, object store providers, documented cost reduction)
- SAP AI Core docs — Rate Limit Management (RPM per model/tenant, 429, resource-group thresholds)
- SAP AI Core docs — Metering and Pricing for Generative AI (tokens, GenAI tokens, capacity units, worked example)
- SAP AI Core docs — Orchestration Workflow V2 (model block, max_completion_tokens, timeout, max_retries)
Full card available to members. What the full card adds: the full decision framework · the common pitfalls and their fix · the cheat sheet · the code blocks · the facts worth quoting.