Concept card

LoRA + QLoRA — Parameter-Efficient Fine-Tuning at Production Scale

As of 2026-07-24T14:00:00Z

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is LoRA + QLoRA — Parameter-Efficient Fine-Tuning at Production Scale?

LoRA freezes the base model and trains only small rank-r matrices injected into attention projections — a 128x reduction in trainable parameters that lets a 70B model fine-tune on a single GPU instead of an 8-GPU cluster.

LoRA (Low-Rank Adaptation) and its quantised variant QLoRA are the production-default technique in 2026 for adapting a large language model to a specific domain or behaviour without retraining all of its weights. Instead of updating every one of a model's seven to seventy billion parameters, LoRA freezes the base weights entirely and injects a small pair of trainable low-rank matrices into each attention projection. The trainable footprint shrinks by two to three orders of magnitude, which is the difference between needing an eight-GPU cluster and fine-tuning a seventy-billion-parameter model on a single high-end accelerator.

How it works

Why it matters

The math is exact and cheap: at r=16, d=4096, that's 131,072 trainable parameters per weight matrix versus 16,777,216 for full fine-tuning — a 128x cut, proven by Dettmers et al. to match full-precision quality within noise.
QLoRA's 4-bit NF4 quantization drops memory another 4x, putting 70B fine-tuning on a single 48GB GPU (L40S, A6000, H100 PCIe) — a hardware-cost unlock, not just an efficiency one.
Skipping the MLP gate projections (gate_proj, up_proj) and targeting only attention leaves ~30% of adaptable parameters untrained — a specific, avoidable configuration mistake.

Key points

Mechanism: freeze base weights W; inject trainable rank-r matrices B and A so effective weight is W + (α/r)×BA; train only B and A — 100-1000× fewer parameters than full fine-tuning.
QLoRA: adds 4-bit NF4 quantisation to the frozen base, cutting memory 4× further; 70B fine-tunes on a single 48GB GPU; matches full-precision quality within statistical noise (Dettmers et al. 2023, Vicuna + MMLU).
Hyperparameters: r=8-32 for behaviour adaptation, r=64 for complex reasoning; alpha=2r; LR=1e-4 to 3e-4 (10× higher than full fine-tuning); 3-5 epochs over 10K-100K curated examples.
Target modules: all four attention projections (q_proj, k_proj, v_proj, o_proj) plus MLP gates (gate_proj, up_proj); omitting MLP gates leaves ~30% of adaptable capacity unused.
Hot-swap serving: multiple adapters share one base model in GPU memory; per-tenant or per-task adapter selected at request time with ~1ms switch latency in vLLM's LoRA serving mode.
When to use LoRA vs alternatives: LoRA for task-specific behaviour with 10K-1M labelled examples; RAG (C203) for retrieval-augmented knowledge; CPT (C257) for raw domain language pre-training.
Adapter merge for zero-latency: run `peft.merge_adapter()` before ONNX export — merged model is identical in size and speed to the base, with the LoRA update baked in.
EU AI Act Art. 9-49 documentation obligation applies to fine-tuned adapters used in high-risk decision systems — the adapter training data, evaluation results, and performance delta vs base must be logged.
Cost benchmark: QLoRA fine-tuning a 7B model for SAP document classification on 50K examples takes ~2h on one A100 at ~€5 total compute cost; deploying the merged adapter adds zero inference overhead.
Failure mode: adapter that underperforms the base on ≥20% of held-out task categories — root cause is usually insufficient or mislabelled training data, not rank selection.

Terms used on this page

LoRA: Low-Rank Adaptation; freezes base weights W and trains low-rank factors B (d×r) and A (r×k) so effective weight is W + (α/r)×BA; reduces trainable parameters 100-1000× with negligible quality loss for behaviour adaptation.
QLoRA: LoRA combined with 4-bit NF4 quantisation of the frozen base weights; enables 70B-parameter fine-tuning on a single 48GB GPU at full-precision quality (Dettmers et al. 2023, arXiv:2305.14314).
NF4 (Normal Float 4): A 4-bit data type optimised for normally-distributed weight values in LLMs; stores each weight in 4 bits vs 16 bits for BF16, reducing memory 4× with less quantisation error than standard INT4 or FP4 on transformer weight distributions.
Adapter hot-swap: Serving pattern where the base model is loaded once in GPU memory and per-task or per-tenant LoRA adapters are loaded/unloaded at request time; vLLM's LoRA serving mode supports this with ~1ms switch latency.
Rank r: The inner dimension of the LoRA decomposition B (d×r) and A (r×k); controls adaptable capacity — r=16 at d=4096 gives 131,072 trainable parameters per weight matrix vs 16,777,216 for full fine-tuning (128× reduction).
Alpha (LoRA scaling): Scaling constant in W_eff = W + (α/r)×BA that controls how strongly the LoRA update is applied; alpha=2r is the canonical starting point; higher alpha amplifies adapter signal and can cause training instability.
PEFT (Parameter-Efficient Fine-Tuning): Family of techniques — LoRA, QLoRA, Prefix Tuning, IA³, Adapter layers — that adapt large pre-trained models by training a tiny fraction of parameters; LoRA is the dominant production choice in 2026 due to zero inference overhead after merge.
Merge adapter: Operation that adds the BA product permanently into the base weight W, producing a standard dense model with the LoRA update baked in; `peft.merge_adapter()` in HuggingFace PEFT; eliminates the runtime cost of adapter computation.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →