RAG vs Fine-tuning vs In-Context Learning — Decision Frame for SAP Knowledge

Every SAP AI project that puts a language model over enterprise knowledge hits the same fork within its first two weeks: do you retrieve the knowledge at query time (RAG), bake it into the model's weights (fine-tuning), or simply hand it to the model in the prompt (in-context learning)? Getting this choice wrong is expensive in three different ways — doubled infrastructure cost, months of added latency to a go-live date, or a model that confidently hallucinates SAP transaction codes in front of a client — and the difference between an experienced AI architect and an enthusiastic pilot team is usually visible in this one decision.

The three approaches

In-context learning (ICL) places the relevant examples or knowledge directly inside the prompt at inference time. It needs no training pipeline, deploys in hours, and is the correct default for prototypes and low-volume use cases. Its ceiling is the context window — 128K tokens for Joule models on BTP — and every token in that window is paid for and processed on every single call, so cost scales linearly with how much context you stuff in, regardless of how much of it the model actually needs. The model's parametric knowledge is also frozen at training cutoff, so ICL alone cannot make a model aware of anything that happened after that date.

Why it matters

• In-context learning deploys in hours and is the right default for prototypes, but everything must fit inside the 128K-token Joule window and cost scales per call.
• RAG is the production standard because it scales to millions of pages (SAP Help Portal), stays fresh without retraining, and gives auditable source citations.
• Fine-tuning is reserved for cases requiring consistent output format or implicit behavioural change — not for injecting fresh knowledge.

Key points

• ICL: no training, deploys in hours, knowledge in the prompt — right for PoC and frequently-changing SAP content (release notes, price lists); limited by context window size.
• RAG: external vector index, retrieval at query time, handles arbitrarily large corpora, stays fresh without retraining, provides source citations — production standard for SAP knowledge applications.
• Fine-tuning: updates model weights, improves format consistency and implicit SAP knowledge, reduces per-call token cost — justified only when format compliance is non-negotiable AND knowledge is stable AND call volume exceeds ~1M/day.
• SAP documentation changes quarterly: fine-tuned models go stale every release cycle; RAG stays current by re-indexing new documentation.
• Hybrid pattern: RAG (factual grounding) + ICL (few-shot formatting) + fine-tuning (output structure only) wins 80%+ of SAP enterprise deployments.
• HANA Cloud Vector Engine eliminates the need for a separate vector database; SAP AI Core hosts embedding and completion models — all within BTP trust boundary.

Terms used on this page

RAG (Retrieval-Augmented Generation)

Architecture that retrieves relevant document chunks from a vector index at inference time and injects them as context into the LLM prompt, grounding responses in up-to-date, citable sources.

Fine-tuning

Process of updating a pre-trained model's weights on a domain-specific labelled dataset to specialise its behaviour, format, or implicit knowledge.

In-context learning (ICL)

Technique where task instructions, examples, and/or knowledge are placed directly in the prompt at inference time; requires no training.

Few-shot prompting

Form of ICL where 3-10 input-output examples are included in the prompt to demonstrate the desired response format or reasoning pattern.

Knowledge cutoff

The date up to which a model's parametric (weight-baked) knowledge extends; ICL and fine-tuning are both bounded by it for facts not supplied at inference time — only RAG can surface information created after cutoff, because it retrieves rather than recalls.

Orchestration grounding

SAP's term (in the generative AI hub) for the RAG mechanic configured as a module in the orchestration service — retrieves and injects relevant content before the LLM call; the production implementation of the RAG pattern this card describes, on SAP's own stack.

Retrieval count (k)

The number of top-ranked chunks a RAG pipeline returns per query; too small a k risks missing the answer, too large a k reintroduces lost-in-the-middle degradation and inflates token cost — tuned per corpus and query type, not fixed globally.

Catastrophic forgetting

The risk that fine-tuning a model on a narrow dataset degrades its general capabilities outside that dataset's scope; a reason to keep fine-tuning narrowly scoped to format or behaviour rather than using it as a general knowledge-injection method.

Sources

1. Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)
2. SAP AI Core — model fine-tuning guide on BTP
3. Gao et al. — Retrieval-Augmented Generation for Large Language Models: A Survey (2023)
4. SAP HANA Cloud Vector Engine — developer guide
5. SAP Business AI — official product page
6. SAP Joule (work companion) — official product page
7. SAP Help Portal — Generative AI Hub overview, SAP AI Core (2026)
8. Hu et al. — LoRA: Low-Rank Adaptation of Large Language Models, arXiv:2106.09685 (2021)
9. OpenAI — Fine-tuning guide (platform documentation)
10. Brown et al. — Language Models are Few-Shot Learners (GPT-3, in-context learning), arXiv:2005.14165 (2020)
11. Anthropic — Prompt engineering / few-shot prompting documentation
12. Wei et al. — Finetuned Language Models Are Zero-Shot Learners, arXiv:2109.01652 (2021)