LoRA + QLoRA — Parameter-Efficient Fine-Tuning at Production Scale

What it is

LoRA (Low-Rank Adaptation) and its quantised variant QLoRA are the production-default technique in 2026 for adapting a large language model to a specific domain or behaviour without retraining all of its weights. Instead of updating every one of a model's seven to seventy billion parameters, LoRA freezes the base weights entirely and injects a small pair of trainable low-rank matrices into each attention projection. The trainable footprint shrinks by two to three orders of magnitude, which is the difference between needing an eight-GPU cluster and fine-tuning a seventy-billion-parameter model on a single high-end accelerator.

Why it matters

• The math is exact and cheap: at r=16, d=4096, that's 131,072 trainable parameters per weight matrix versus 16,777,216 for full fine-tuning — a 128x cut, proven by Dettmers et al. to match full-precision quality within noise.
• QLoRA's 4-bit NF4 quantization drops memory another 4x, putting 70B fine-tuning on a single 48GB GPU (L40S, A6000, H100 PCIe) — a hardware-cost unlock, not just an efficiency one.
• Skipping the MLP gate projections (gate_proj, up_proj) and targeting only attention leaves ~30% of adaptable parameters untrained — a specific, avoidable configuration mistake.

Key points

• Mechanism: freeze base weights W; inject trainable rank-r matrices B and A so effective weight is W + (α/r)×BA; train only B and A — 100-1000× fewer parameters than full fine-tuning.
• QLoRA: adds 4-bit NF4 quantisation to the frozen base, cutting memory 4× further; 70B fine-tunes on a single 48GB GPU; matches full-precision quality within statistical noise (Dettmers et al. 2023, Vicuna + MMLU).
• Hyperparameters: r=8-32 for behaviour adaptation, r=64 for complex reasoning; alpha=2r; LR=1e-4 to 3e-4 (10× higher than full fine-tuning); 3-5 epochs over 10K-100K curated examples.
• Target modules: all four attention projections (q_proj, k_proj, v_proj, o_proj) plus MLP gates (gate_proj, up_proj); omitting MLP gates leaves ~30% of adaptable capacity unused.
• Hot-swap serving: multiple adapters share one base model in GPU memory; per-tenant or per-task adapter selected at request time with ~1ms switch latency in vLLM's LoRA serving mode.
• When to use LoRA vs alternatives: LoRA for task-specific behaviour with 10K-1M labelled examples; RAG (C203) for retrieval-augmented knowledge; CPT (C257) for raw domain language pre-training.
• Adapter merge for zero-latency: run `peft.merge_adapter()` before ONNX export — merged model is identical in size and speed to the base, with the LoRA update baked in.
• EU AI Act Art. 9-49 documentation obligation applies to fine-tuned adapters used in high-risk decision systems — the adapter training data, evaluation results, and performance delta vs base must be logged.
• Cost benchmark: QLoRA fine-tuning a 7B model for SAP document classification on 50K examples takes ~2h on one A100 at ~€5 total compute cost; deploying the merged adapter adds zero inference overhead.
• Failure mode: adapter that underperforms the base on ≥20% of held-out task categories — root cause is usually insufficient or mislabelled training data, not rank selection.

Terms used on this page

LoRA

Low-Rank Adaptation; freezes base weights W and trains low-rank factors B (d×r) and A (r×k) so effective weight is W + (α/r)×BA; reduces trainable parameters 100-1000× with negligible quality loss for behaviour adaptation.

QLoRA

LoRA combined with 4-bit NF4 quantisation of the frozen base weights; enables 70B-parameter fine-tuning on a single 48GB GPU at full-precision quality (Dettmers et al. 2023, arXiv:2305.14314).

NF4 (Normal Float 4)

A 4-bit data type optimised for normally-distributed weight values in LLMs; stores each weight in 4 bits vs 16 bits for BF16, reducing memory 4× with less quantisation error than standard INT4 or FP4 on transformer weight distributions.

Adapter hot-swap

Serving pattern where the base model is loaded once in GPU memory and per-task or per-tenant LoRA adapters are loaded/unloaded at request time; vLLM's LoRA serving mode supports this with ~1ms switch latency.

Rank r

The inner dimension of the LoRA decomposition B (d×r) and A (r×k); controls adaptable capacity — r=16 at d=4096 gives 131,072 trainable parameters per weight matrix vs 16,777,216 for full fine-tuning (128× reduction).

Alpha (LoRA scaling)

Scaling constant in W_eff = W + (α/r)×BA that controls how strongly the LoRA update is applied; alpha=2r is the canonical starting point; higher alpha amplifies adapter signal and can cause training instability.

PEFT (Parameter-Efficient Fine-Tuning)

Family of techniques — LoRA, QLoRA, Prefix Tuning, IA³, Adapter layers — that adapt large pre-trained models by training a tiny fraction of parameters; LoRA is the dominant production choice in 2026 due to zero inference overhead after merge.

Merge adapter

Operation that adds the BA product permanently into the base weight W, producing a standard dense model with the LoRA update baked in; `peft.merge_adapter()` in HuggingFace PEFT; eliminates the runtime cost of adapter computation.

Sources

1. arXiv: LoRA — Low-Rank Adaptation of Large Language Models (Hu et al. 2021)
2. arXiv: QLoRA — Efficient Finetuning of Quantized LLMs (Dettmers et al. 2023)
3. Hugging Face PEFT library documentation
4. NIST AI Risk Management Framework 1.0 — model lifecycle governance
5. arXiv: DoRA — Weight-Decomposed Low-Rank Adaptation (Liu et al. 2024)
6. EU AI Act — Art. 9-49 documentation requirements for high-risk AI systems
7. SAP Business AI — official product page
8. arXiv — Houlsby et al., "Parameter-Efficient Transfer Learning for NLP" — the original Adapter-tuning paper (2019, fetched 2026-09-27)
9. arXiv — Li & Liang, "Prefix-Tuning: Optimizing Continuous Prompts for Generation" (2021, fetched 2026-09-27)
10. vLLM documentation — LoRA Adapters, multi-adapter hot-swap serving (fetched 2026-09-27)
11. SAP Help Portal — generative AI hub overview, SAP AI Core (fetched 2026-09-27)
12. SAP Community — SAP Joule for Developers: end of the promotional period and the path forward (2026, fetched 2026-09-27)