Evaluating Generative AI on SAP Data

A pre-production harness, not a production watch

SAP AI Core's Evaluations feature, built on the genai-evaluation global scenario, is described by SAP as tooling "for benchmarking large language models and prompts via orchestration configurations": you evaluate prompt-and-model combinations against a dataset you control, comparing options before one goes live. C129 covers a narrower, later slice of the same toolkit — three LLM-as-a-judge metrics used to watch hallucination and drift once a configuration is already in production. This card covers the harness itself: the full metric catalog, custom metrics, and the workflow that runs an evaluation from a registered dataset to a comparison you can act on.

Why it matters

• A team that only watches C129's three production metrics has no systematic way to compare a candidate prompt or model before it ships — that comparison is exactly what the Evaluations feature and this card are for.
• Content Filter on Input/Output looking like a filtering control rather than an audit metric is an easy misread that leads a team to believe it configured filtering when it only measured whether filtering fired in a past run.
• A custom metric with a rubric but no worked examples produces less consistent scores — the kind of quality gap that only shows up once you compare two runs and can't explain why the judge disagreed with itself.

Key points

• Evaluations runs under the genai-evaluation global scenario; it benchmarks prompt+model orchestration configs against your own dataset before production.
• 8 computed metrics: BERTScore, BLEU, ROUGE, Exact Match (all need a reference) + JSON Schema Match, Content Filter on Input, Content Filter on Output, Language Match (no reference needed).
• 8 LLM-as-a-judge metrics: Instruction Following, Correctness, Answer Relevance, Conciseness (1-5, general) + RAG Groundedness, RAG Context Relevance (1-3) + experimental RAG Context Precision, RAG Completeness. None need a reference.
• Custom metrics use a structured prompt: modelConfiguration, definition, evaluationTask, ratingRubric, criteria, evaluationSteps, examples.
• Workflow: register dataset artifact -> save prompt in prompt registry -> (optional) variable mapping -> POST /v2/lm/configurations with executableId genai-evaluations-simplified -> execute -> compare results per record and in aggregate.
• promptTemplateScope (tenant default, or resourcegroup) must match how the prompt template was created.
• C129 uses 3 of these metrics for production hallucination/drift watching; this card covers the full harness used before go-live.
• Content Filter on Input/Output metrics audit whether the filtering module fired on a record — they do not run filtering themselves.

Terms used on this page

genai-evaluation

The global scenario under which SAP AI Core's Evaluations feature runs.

Computed metric

A deterministic, formula-based evaluation metric (e.g. BLEU, ROUGE) as opposed to an LLM-as-a-judge metric.

LLM-as-a-judge metric

An evaluation metric scored by a judge LLM against a rubric rather than by string comparison.

Custom metric

A user-defined LLM-as-a-judge metric built from a structured prompt with a rating rubric and worked examples.

promptTemplateScope

The evaluation configuration parameter (tenant or resourcegroup) telling the run where to find the prompt registry template.

Variable mapping

A configuration reconciling mismatched names between a prompt template's placeholders and a test dataset's attributes.

Sources

1. SAP AI Core docs (SAP-docs GitHub, Sep 2026) — Evaluations (genai-evaluation scenario, use cases)
2. SAP AI Core docs — System-Defined Evaluation Metrics (8 computed + 8 LLM-as-a-judge metrics, reference requirements, scales)
3. SAP AI Core docs — Custom Metrics (structured prompts, rating rubric, worked example)
4. SAP AI Core docs — Create an Evaluation (prerequisites, configuration parameters, promptTemplateScope)