Evaluating Generative AI on SAP Data — Building an Evaluation Harness
As of 2026-09-25
Hands-on evaluation harness for generative AI on SAP data, built directly on SAP AI Core's Evaluations capability. Covers the two metric families (system-defined computed metrics such as BERT Score, BLEU, ROUGE, JSON Schema Match, Exact Match, Language Match, and content-filter audit booleans; and LLM-as-a-judge metrics such as Pointwise Correctness, Pointwise Answer Relevance and the RAG-specific Groundedness and Context Relevance metrics), SAP's own documented finding that computed metrics under-perform LLM-as-a-judge metrics for grounding evaluation, custom metrics via structured prompts and rating rubrics, the exact configuration and execution API calls (genai-evaluations-simplified), reading the SQLite raw-metrics output against the tracked aggregate metrics, and the four design choices that turn one run into a re-usable harness: a frozen test set, one metric per failure mode, a pre-declared pass threshold and a re-run trigger tied to SAP Note 3437766 model deprecations.
What you will learn
- Explain why computed metrics (BERT Score, BLEU, ROUGE) underperform LLM-as-a-judge metrics for grounding evaluation, and apply SAP's documented substitution
- Choose system-defined computed and LLM-as-a-judge metrics for a given SAP generative AI use case, and design a custom LLM-as-a-judge metric with a rating rubric when none fits
- Build a valid SAP AI Core evaluation configuration (genai-evaluations-simplified) with dataset, metrics, prompt or orchestration references, and start and poll its execution
- Read an evaluation's SQLite raw-metrics output and its tracked aggregate metrics, and diagnose which test rows failed and why
- Design a re-usable evaluation harness: a frozen test set, one metric per failure mode, a pre-declared pass threshold, and a re-run trigger tied to prompt, model or grounding changes
- Compare two or more models in a single evaluation run and weigh a quality gap against its GenAI-token cost before recommending one for production
Module overview
Who this is for. You have a prompt or a RAG pipeline in the SAP generative AI hub that looks good in a demo, and a steering committee that wants proof it works before it touches a live process. This module builds that proof: a repeatable evaluation harness using SAP AI Core's Evaluations capability, so a model or prompt change becomes a measured decision instead of an impression. It assumes M325 (generative AI hub hands-on) and the fundamentals in M333.
Prerequisites
- M333 (AI & LLM Fundamentals for SAP Consultants) or equivalent working knowledge of tokens, RAG and evaluation basics
- M325 (SAP Generative AI Hub hands-on) or comfort creating an orchestration configuration and a prompt template
- Access to a generative AI hub tenant (or its trial) with the AI API, or willingness to write API calls on paper
- Comfort with JSON and basic REST API calls (curl or an HTTP client)
Outcomes
- Justify, to a steering committee, which metric family (computed vs LLM-as-a-judge) fits a given SAP generative AI use case and why.
- Produce a working SAP AI Core evaluation configuration and execution request for a real or realistic SAP use case.
- Read raw and aggregate evaluation results well enough to locate the failing row, not just the average score.
- Hand over a documented, re-runnable evaluation harness with a frozen test set, chosen metrics and a pass threshold.
Full module available to members. The full module adds: the decision framework · the end-to-end scenario walkthrough · the KPI scorecard · the anti-patterns · the code blocks · the knowledge check · the diagrams.