Concept card

Tensor Parallelism vs Pipeline Parallelism — Multi-GPU Inference

As of 2026-07-24T14:00:00Z

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is Tensor Parallelism vs Pipeline Parallelism — Multi-GPU Inference?

Tensor parallelism needs NVLink and scales to 8 GPUs in one node; pipeline parallelism tolerates slower cross-node links but wastes compute to the pipeline bubble — production stacks combine both.

Tensor Parallelism and Pipeline Parallelism are the two foundational strategies for serving a large language model whose weights are too large to fit on a single GPU. A seventy-billion-parameter model in standard half-precision needs roughly one hundred and forty gigabytes of memory before any headroom for the key-value cache is added — far beyond the eighty gigabytes on a single top-tier data-centre GPU. Splitting the model across multiple devices is not optional, and the way that split is designed directly determines the latency, throughput and interconnect cost of the whole deployment.

How Each Strategy Splits the Model

Tensor Parallelism splits every weight matrix horizontally across a set of GPUs: each layer of the model runs on every GPU simultaneously, with each device computing its own shard of the matrix multiplication, and the partial results are summed across devices before the next layer begins. Because every layer still executes in lockstep across all GPUs, tensor parallelism keeps end-to-end latency low — there is no waiting for earlier stages to drain. The cost shows up on the interconnect: every layer triggers a full synchronisation step across devices, so a very fast GPU-to-GPU link is effectively mandatory. On slower interconnects the synchronisation overhead dominates and throughput collapses. Tensor parallelism scales cleanly within a single high-bandwidth node but degrades quickly once it has to cross to a second node, where the added network latency compounds across dozens of transformer layers.

Why it matters

TP triggers an AllReduce every layer, so PCIe-only nodes collapse under the bandwidth demand — NVLink (900GB/s on H100 SXM) is mandatory, and cross-node TP adds latency that compounds across 80+ transformer layers.
PP's pipeline bubble fraction is (P-1)/(M+P-1) — at 8 micro-batches and pipeline depth 4 that's 27% idle time, acceptable for batch serving but unacceptable for single-stream interactive workloads.
A 70B model in FP16 needs ~140GB, far beyond a single H100's 80GB, so any serving stack at that scale must choose a split strategy — there's no way around it.

Key points

TP splits weight matrices horizontally — every GPU runs every layer in parallel, AllReduce sums partial results; mandatory NVLink, scales best within one node up to TP=8.
PP splits the model vertically by layer ranges — activations flow GPU-to-GPU once per forward pass; works over InfiniBand/Ethernet across nodes.
TP minimises single-request latency at the cost of interconnect bandwidth; PP minimises interconnect traffic at the cost of pipeline-bubble idle time.
Production rule: TP within node (NVLink-bound), PP across nodes (network-bound); Llama-3.1-405B typically TP=8 × PP=2 on 16 H100s.
Frameworks: vLLM ≥0.5, TensorRT-LLM, SGLang, DeepSpeed-Inference all expose both knobs; defaults are rarely optimal for your workload mix.
Anti-pattern: TP across PCIe (no NVLink) — AllReduce saturates the bus and throughput collapses by 30-50% on time-to-first-token and 60-80% on sustained throughput.
Bubble fraction formula: (P-1)/(M+P-1) where P = pipeline depth, M = micro-batch count; target <20% for interactive workloads.
KV-cache memory scales as batch_size × context_length × num_layers × hidden_dim × 2 (K+V) × bytes_per_element — size this before choosing TP degree.
SAP BTP AI inference hosts in 2026 are NVLink SXM pods; verify NVLink topology before specifying TP>4 or the guarantee is void.
Sequence parallelism (SP) extends TP to the attention's sequence dimension — used in vLLM when context length exceeds 8K to avoid OOM on the KV-cache shard.

Terms used on this page

AllReduce: Collective communication primitive that sums partial tensors across all participating GPUs and broadcasts the result; the dominant cost in tensor-parallel forward passes — on NVLink H100 takes ~5 µs per layer at TP=8.
Pipeline bubble: Idle GPU time in pipeline parallelism while later stages wait for earlier stages on the first/last micro-batches of a step; fraction = (P-1)/(M+P-1) where P is pipeline depth and M is micro-batch count.
NVLink: NVIDIA's high-bandwidth GPU-to-GPU interconnect (900 GB/s bidirectional on H100 SXM5 NVSwitch) — prerequisite for efficient tensor parallelism; PCIe bandwidth (~64 GB/s) is 14× slower and collapses under AllReduce traffic.
KV-cache: Cached key/value attention tensors retained across decoding steps; consumes GPU memory proportional to context length × batch size × num_layers × hidden_dim × 2; the primary memory budget constraint when sizing TP degree.
Sequence Parallelism (SP): Extension of tensor parallelism that shards the input sequence across GPUs during attention computation, reducing per-GPU KV-cache footprint at long context lengths; implemented in vLLM and Megatron-LM.
TTFT: Time To First Token — the latency from request submission to first output token; the dominant UX metric for interactive applications; TP=8 within a NVLink node minimises TTFT; PP adds bubble latency that degrades TTFT.
Micro-batch: A subdivision of the global batch in pipeline parallelism; multiple micro-batches flowing through the pipeline simultaneously suppress the pipeline bubble and improve GPU utilisation.
Tensor shard: One GPU's share of a weight matrix in tensor parallelism; for a hidden dimension d=8192 at TP=4, each GPU holds a 2048-column shard of each weight, computing its partial matrix multiplication before AllReduce.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →