Concept card

Latency vs Throughput Trade-offs — TTFT, TPOT, SLA Design

As of 2026-07-23

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is Latency vs Throughput Trade-offs — TTFT, TPOT, SLA Design?

Chat, code-completion, batch jobs and voice agents each need a different TTFT/TPOT target — mixing a latency-sensitive chat endpoint with a throughput-hungry batch job guarantees the batch job tanks chat p99.

What it is

LLM serving has three orthogonal performance metrics that an SLA must address separately. Time-To-First-Token (TTFT) measures the wait from request submission to the first emitted token — dominated by the prefill phase, in which the model processes the full input prompt in one parallel pass. Time-Per-Output-Token (TPOT, also written as inter-token latency, ITL) measures the gap between successive emitted tokens — dominated by the decode phase, in which the model generates one token per forward pass. End-to-end latency is TTFT + (output tokens × TPOT). Throughput (tokens/sec, aggregated across concurrent requests) is the system-wide capacity number.

Why it matters

Larger continuous batches raise throughput but also raise TTFT (new requests wait for the next scheduling tick) and TPOT (more requests share the same GPU clock per token) — the trade-off can't be eliminated, only reduced.
Concrete SLA targets differ by workload: chat wants p95 TTFT under 500ms, code-completion under 200ms, batch summarisation ignores TTFT entirely to maximise batch size, voice agents want sub-100ms TPOT but tolerate 500-800ms TTFT.
The architect's fix is splitting workloads across separate endpoint tiers pointing at the same model file — one low-latency small-batch endpoint, one high-throughput large-batch endpoint.

Key points

Three orthogonal metrics: TTFT (prefill-bound), TPOT/ITL (decode-bound), throughput (system-wide tokens/sec aggregated across concurrent requests).
End-to-end latency = TTFT + (output_tokens × TPOT) — both must appear in any LLM SLA.
Continuous batching raises throughput but raises TTFT (requests wait for next scheduling tick) and TPOT (requests share clock); paged attention reduces but does not eliminate the trade-off.
Chat target: p95 TTFT < 500ms, TPOT 30-50ms. Code-completion: TTFT < 200ms. Batch jobs: throughput-first, TTFT irrelevant. Voice agents: TPOT < 100ms.
Split workloads across endpoint tiers — low-latency + small-batch for chat, high-throughput + large-batch for offline; never mix on one endpoint.
Benchmark at p50/p95/p99 — averages hide the long-tail latency that kills user experience.
Latency vs Throughput Trade-offs — TTFT, TPOT, SLA Design is mastered only when it changes a named buyer decision.
Start with the semantic contract and control model before demonstrating the tool.
Use current SAP, analyst, study, KG, and news signals as evidence, not decoration.
Separate verified facts from directional trends and modeled assumptions.

Terms used on this page

TTFT: Time-To-First-Token — time from request submit to first emitted token; bounded by prefill compute over the full input prompt.
TPOT / ITL: Time-Per-Output-Token or Inter-Token Latency — time between successive emitted tokens during decoding; bounded by memory bandwidth at autoregressive generation.
Continuous batching: Serving technique where new requests join an in-flight batch at each decode step, instead of waiting for batch completion; standard in vLLM, TensorRT-LLM, SGLang.
Paged attention: Memory-management technique borrowed from virtual memory; KV-cache is stored in fixed-size pages to reduce fragmentation when serving variable-length sequences.
Decision owner: The accountable person who accepts the trade-off and funds the next action.
Semantic contract: The shared definition of business terms, metrics, entities, and access rules used by tools and teams.
Control plane: The layer that applies policy, access, lineage, monitoring, and escalation across the operating model.
Evidence grade: A label that separates verified fact, directional signal, modeled assumption, and field observation.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks.

Open in the app →