LLM Cost Engineering on SAP — AI Units, Token Budgets, Caching

Where cost engineering sits next to cost attribution

“AI Cost Attribution in SAP AI Core — Tokens, Capacity Units and Resource Groups” covers how SAP AI Core spend gets attributed — tokens to GenAI tokens to capacity units, resource-group grain, the labels you add for chargeback. “SAP AI Units & the SAP Business AI Commercial Model” covers the commercial layers Joule and embedded AI sit in — AI Units, per-agent-action pricing. Neither is about shrinking the bill itself; that is what this page does. The metering chain both cards describe is still the starting point: input and output tokens (output priced slightly higher) convert to GenAI tokens at model-specific rates SAP documents in SAP Note 3437766, and GenAI tokens become the capacity units on the invoice. Every lever below acts somewhere on that chain — by avoiding tokens entirely (caching), by avoiding the synchronous premium (batching), or by bounding the tokens a call is allowed to spend (budgets and limits).

Why it matters

• A RAG pipeline with a long, static system prompt and no cache_control is paying full price on that prefix every single call — for Claude or Nova models, that is a discount left entirely on the table.
• Routing a nightly bulk-classification job through orchestration instead of batch consumption both costs more per SAP's own documentation and adds client-side rate-limit handling that batch consumption doesn't require.
• An unset max_completion_tokens is not a safe default — it silently authorizes the model's own maximum, which is exactly the kind of detail that turns a pilot's token estimate into a production invoice surprise.

Key points

• Metering chain (shared with “AI Cost Attribution in SAP AI Core — Tokens, Capacity Units and Resource Groups”): input/output tokens -> GenAI tokens (model-specific rate, SAP Note 3437766) -> capacity units billed.
• Explicit prompt caching (cache_control) works only for Anthropic Claude (up to 4 breakpoints: tools, system, messages) and Amazon Nova (system/messages, ~1 breakpoint). OpenAI/Gemini get implicit caching only, unconfigurable.
• Default cache TTL is 5 minutes; select Anthropic models support an optional 1-hour TTL.
• Batch consumption is documented as cheaper than synchronous calls, retries automatically, and needs no client-side rate-limit handling — but supports native LLM calls only, not orchestration (no grounding/masking/filtering).
• Batch consumption is EU/US only, excluding prod-euonly and sovereign cloud, and requires an object store secret (S3, Azure Blob, GCS, Alibaba OSS, or HANA Cloud Data Lake).
• RPM rate limits are per model per tenant; a 429 costs latency and retry risk, not extra billing per se — raising the limit does not lower spend.
• max_completion_tokens is mandatory for Anthropic models; left unset for any model, orchestration applies that model's own maximum.
• Fallback lists (config.modules as an array) handle 408/429/5xx on non-streaming calls by switching model instead of retrying blindly.

Terms used on this page

GenAI token

A virtual metering unit representing model-specific token usage, used to compute capacity-unit billing (SAP Note 3437766).

cache_control

The orchestration V2 field marking a prompt block as a cache breakpoint, supported only for Anthropic Claude and Amazon Nova models.

Implicit caching

Default, unconfigurable context caching enabled for OpenAI and Gemini models in orchestration.

Batch consumption

Asynchronous processing of many native LLM requests from a file, documented as cheaper than synchronous calls but unavailable for orchestration requests.

RPM limit

A per-model, per-tenant cap on Requests Per Minute; exceeding it returns HTTP 429.

Fallback list

config.modules passed as an array so orchestration switches models on an unsupported region or, without streaming, on 408/429/5xx.

Sources

1. SAP AI Core docs (SAP-docs GitHub, Sep 2026) — Prompt Caching (cache_control, supported models, TTL, response fields)
2. SAP AI Core docs — Batch Consumption (native-LLM-only, regions, object store providers, documented cost reduction)
3. SAP AI Core docs — Rate Limit Management (RPM per model/tenant, 429, resource-group thresholds)
4. SAP AI Core docs — Metering and Pricing for Generative AI (tokens, GenAI tokens, capacity units, worked example)
5. SAP AI Core docs — Orchestration Workflow V2 (model block, max_completion_tokens, timeout, max_retries)
6. Anthropic — Prompt caching documentation (primary source behind SAP's Claude cache_control support)
7. AWS — Amazon Bedrock prompt caching user guide (primary source behind SAP's Amazon Nova cache_control support)
8. SAP Help Portal — Use a service key, SAP AI Core service guide
9. SAP Help Portal — Register your object store secret, SAP AI Core service guide (prerequisite for batch consumption)
10. GitHub SAP-docs — Prepare and upload your input file for a batch job (SAP-authored mirror)
11. GitHub SAP-docs — Create a batch job (SAP-authored mirror)
12. SAP Support Launchpad — SAP Note 3437766, model information, token conversion rates and rate limits
13. Prior Labs — TabPFN-3.5 changelog (50% discount on Prior Labs API and MCP, 15-29 Sep 2026)
14. SAP — Understanding SAP's Business AI Commercial Model, version 10.2026 (AI Units per action, p. 7)