Concept card

Inference Economics — Cost, Latency, Quantization, Distillation

As of 2026-07-23

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is Inference Economics — Cost, Latency, Quantization, Distillation?

At 200K daily Joule sessions, a 10% token reduction alone saves $36K-$252K a year — which is why prompt engineering is billable consulting work, not a nice-to-have.

What it is

Inference economics is the discipline of running LLMs at the cost, latency, and throughput that enterprise workloads can sustain. For SAP practitioners, this means understanding the per-1K-token math for BTP-hosted Joule, knowing which optimisation levers are available within the SAP trust boundary, and being able to build a business case for AI spend that CFOs will approve.

The cost of inference has three components: compute (GPU time per forward pass), memory bandwidth (KV cache read/write), and I/O (network + storage). In managed API pricing (OpenAI, Anthropic, SAP BTP Generative AI Hub), these are bundled into a per-token rate: input tokens billed at one rate, output tokens at a higher rate (output generation is autoregressive and therefore compute-intensive). On SAP BTP Generative AI Hub as of 2026, GPT-4o pricing follows the OpenAI commercial schedule passed through SAP's reseller margin — approximately $0.005-$0.015 per 1K input tokens and $0.015-$0.060 per 1K output tokens depending on model and tier. A Joule S/4HANA help-desk assistant with 200K daily sessions, 1K input + 500 output tokens per session = 200M input + 100M output tokens/day = approximately $1,000-$7,000/day at list price. At that scale, every 10% token reduction saves $36K-$252K/year — which is why prompt engineering is a billable consulting skill.

Why it matters

Output tokens cost 3-4x input tokens because generation is autoregressive, so trimming output length saves more than trimming input.
TTFT scales with input length — a 10K-token prompt takes roughly 10x longer to first token than a 1K-token one, which breaks sub-2-second interactive Joule assistants unless input is bounded and streamed.
INT8 quantization halves compute and memory for under 1% quality loss, while INT4 saves 4x but costs 3-8% quality — fine for summaries, not for financial decision support.

Key points

Inference cost = input tokens × input rate + output tokens × output rate; BTP Generative AI Hub passes through OpenAI list pricing with SAP reseller margin (~$0.005-0.060/1K tokens depending on model).
Time-to-first-token (TTFT) scales with input length; inter-token latency (ITL) is roughly constant per step — interactive Joule workflows must bound input length and enable streaming.
INT8 quantization: ~2× memory/compute reduction, < 1% quality loss — production-safe for most SAP tasks. INT4: ~4× reduction, 3-8% quality loss — acceptable for low-stakes summarisation only.
Distillation trains a small student model (7B) to mimic a large teacher (GPT-4-class) on a specific task — 10-100× inference cost reduction, 70-90% task-specific accuracy retained.
At 200K daily SAP Joule sessions (1K input + 500 output tokens), list-price cost is $1,000-$7,000/day; 10% token reduction = $36K-$252K/year saving.
Batch vs. interactive trade-off: for nightly SAP analytics batch jobs, maximise throughput (large batch size, INT8); for interactive Joule assistants, minimise TTFT (short inputs, streaming, prefix caching).
Inference Economics — Cost, Latency, Quantization, Distillation is mastered only when it changes a named buyer decision.
Start with the semantic contract and control model before demonstrating the tool.
Use current SAP, analyst, study, KG, and news signals as evidence, not decoration.
Separate verified facts from directional trends and modeled assumptions.

Terms used on this page

TTFT (Time-to-first-token): Latency from request submission to when the first output token is produced; dominated by the prefill pass over the full input context.
Quantization: Reducing the numerical precision of model weights (e.g. FP16 → INT8 → INT4) to lower memory footprint and inference compute, with a controlled quality trade-off.
Distillation: Training a smaller student model to mimic a larger teacher model's output distribution on a target task; produces a task-specific model 10-100× cheaper at inference.
Throughput: Number of tokens generated per second across all concurrent requests; maximised by large batch sizes and quantization; the relevant metric for batch analytics workloads.
Decision owner: The accountable person who accepts the trade-off and funds the next action.
Semantic contract: The shared definition of business terms, metrics, entities, and access rules used by tools and teams.
Control plane: The layer that applies policy, access, lineage, monitoring, and escalation across the operating model.
Evidence grade: A label that separates verified fact, directional signal, modeled assumption, and field observation.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks.

Open in the app →