Concept card

RAG Grounding via SAP Knowledge Graph

As of 2026-07-24T14:00:00Z

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is RAG Grounding via SAP Knowledge Graph?

RAG is the right Joule grounding mode for unstructured document retrieval — contracts, manuals, policy Q&A — distinct from, but combinable with, SAP Knowledge Graph traversal.

What it is

Retrieval-Augmented Generation, or RAG, is the technique by which a foundation model pulls in relevant passages from an external knowledge store before it writes an answer, instead of relying only on the patterns it absorbed during training. Inside the SAP Joule architecture, RAG is one of three grounding modes a designer can choose for an agent — the other two being a direct Analytic Model query against structured SAC or Datasphere data, and traversal of the SAP Knowledge Graph, the entity-relationship layer built from S/4HANA, SuccessFactors, and Ariba metadata. RAG is specifically the right tool for unstructured content: contract clauses, maintenance manuals, policy PDFs, knowledge-base articles — anything that lives as prose rather than as rows and columns or as typed relationships between business objects.

The reason this matters commercially is simple. A foundation model trained on public internet text has never seen a customer's internal procurement policy, so it cannot answer "what are the payment terms for a tier-2 vendor?" from its weights alone. Left unguarded, it will either refuse or, worse, generate a plausible-sounding but wrong answer. RAG closes that gap in two stages: an ingestion stage that turns the enterprise's documents into searchable vector embeddings ahead of time, and a retrieval stage that, at the moment a question is asked, finds the passages closest in meaning to the question and hands them to the model as context. The model then composes its answer from that retrieved material rather than from memory, which is what makes the answer traceable back to a source document.

Why it matters

A Joule agent asked about tier-2 vendor payment terms can't answer from training weights — the answer lives in a procurement policy PDF, which only RAG-based retrieval can surface.
Chunk size (roughly 300-800 tokens) is the first architectural decision — too small fragments context, too large degrades embedding quality.
Choosing RAG versus Knowledge Graph traversal versus Analytic Model query is a mode selection, not a default — picking wrong misgrounds the answer.

Key points

RAG = one of three Joule grounding modes: use for unstructured document retrieval (contracts, procedures, manuals); use Analytic Model for structured KPIs; use Knowledge Graph for entity traversal.
SAP AI Foundation Vector Store: EU data plane compliant (Frankfurt/Amsterdam); access governed at index level, not document level — separate indexes per access tier for sensitive content.
Pipeline: chunk (300–800 tokens) → embed (ada-002 or multilingual-e5) → metadata tag → HNSW/IVF-Flat index → cosine similarity search at inference time.
Hallucination suppression: 'answer only from provided context; if not present, say so' — standard instruction in every Joule RAG prompt.
Hybrid grounding: Knowledge Graph identifies entities (supplier ID, contract IDs) + RAG retrieves clause text from those contracts — combine both for highest-quality procurement/legal agents.
Citation display is both a user-trust feature and an Article 13 EU AI Act transparency obligation for high-risk use cases.
Most missed design decision: single shared Vector Store index for heterogeneous document categories = governance failure. One index per access tier.
RAG Grounding via SAP Knowledge Graph is mastered only when it changes a named buyer decision.
Start with the semantic contract and control model before demonstrating the tool.
Use current SAP, analyst, study, KG, and news signals as evidence, not decoration.

Terms used on this page

RAG (Retrieval-Augmented Generation): Technique combining a retrieval system (vector similarity search over a document index) with a generative LLM — the LLM generates its response from retrieved document chunks rather than from training data alone.
Vector Store: Managed database for vector embeddings; supports approximate nearest-neighbor (ANN) similarity search using algorithms like HNSW or IVF-Flat; core infrastructure for RAG pipelines.
Embedding: High-dimensional numerical representation of text produced by an embedding model; semantically similar texts produce geometrically close vectors, enabling similarity search.
Chunking: The process of splitting source documents into segments of 200–800 tokens for embedding; chunk size and overlap are the primary determinants of RAG retrieval quality.
SAP Knowledge Graph: Entity-relationship graph derived from S/4HANA, SuccessFactors, and Ariba object metadata — material-vendor-plant, employee-organisation-role, contract-supplier relationships. Used for entity traversal grounding in Joule agents.
Cosine similarity: The primary similarity metric for vector search — measures the angle between two high-dimensional vectors; value of 1.0 = identical direction (semantically very similar); 0.0 = orthogonal (unrelated).
HNSW (Hierarchical Navigable Small World): Graph-based approximate nearest-neighbor index algorithm optimised for sub-second similarity search at high recall; default index type for most production Vector Store deployments.
Cross-encoder reranker: A model that takes a query + candidate chunk pair as joint input and scores their relevance more precisely than embedding similarity alone; adds 200–400ms latency but significantly improves precision on dense technical documents.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →