Evaluating Generative AI on SAP Data

A pre-production harness, not a production watch

SAP AI Core's Evaluations feature, built on the genai-evaluation global scenario, is described by SAP as tooling "for benchmarking large language models and prompts via orchestration configurations": you evaluate prompt-and-model combinations against a dataset you control, comparing options before one goes live. “AI Observability in SAP AI Core — Token Usage, Hallucination, Drift” covers a narrower, later slice of the same toolkit — three LLM-as-a-judge metrics used to watch hallucination and drift once a configuration is already in production. This page covers the harness itself: the full metric catalog, custom metrics, and the workflow that runs an evaluation from a registered dataset to a comparison you can act on.

Why it matters

• A team that only watches “AI Observability in SAP AI Core — Token Usage, Hallucination, Drift”'s three production metrics has no systematic way to compare a candidate prompt or model before it ships — that comparison is exactly what the Evaluations feature and this page are for.
• Content Filter on Input/Output looking like a filtering control rather than an audit metric is an easy misread that leads a team to believe it configured filtering when it only measured whether filtering fired in a past run.
• A custom metric with a rubric but no worked examples produces less consistent scores — the kind of quality gap that only shows up once you compare two runs and can't explain why the judge disagreed with itself.

Key points

• Evaluations runs under the genai-evaluation global scenario; it benchmarks prompt+model orchestration configs against your own dataset before production.
• 8 computed metrics: BERTScore, BLEU, ROUGE, Exact Match (all need a reference) + JSON Schema Match, Content Filter on Input, Content Filter on Output, Language Match (no reference needed).
• 8 LLM-as-a-judge metrics: Instruction Following, Correctness, Answer Relevance, Conciseness (1-5, general) + RAG Groundedness, RAG Context Relevance (1-3) + experimental RAG Context Precision, RAG Completeness. None need a reference.
• Custom metrics use a structured prompt: modelConfiguration, definition, evaluationTask, ratingRubric, criteria, evaluationSteps, examples.
• Workflow: register dataset artifact -> save prompt in prompt registry -> (optional) variable mapping -> POST /v2/lm/configurations with executableId genai-evaluations-simplified -> execute -> compare results per record and in aggregate.
• promptTemplateScope (tenant default, or resourcegroup) must match how the prompt template was created.
• “AI Observability in SAP AI Core — Token Usage, Hallucination, Drift” uses 3 of these metrics for production hallucination/drift watching; this page covers the full harness used before go-live.
• Content Filter on Input/Output metrics audit whether the filtering module fired on a record — they do not run filtering themselves.

Terms used on this page

genai-evaluation

The global scenario under which SAP AI Core's Evaluations feature runs.

Computed metric

A deterministic, formula-based evaluation metric (e.g. BLEU, ROUGE) as opposed to an LLM-as-a-judge metric.

LLM-as-a-judge metric

An evaluation metric scored by a judge LLM against a rubric rather than by string comparison.

Custom metric

A user-defined LLM-as-a-judge metric built from a structured prompt with a rating rubric and worked examples.

promptTemplateScope

The evaluation configuration parameter (tenant or resourcegroup) telling the run where to find the prompt registry template.

Variable mapping

A configuration reconciling mismatched names between a prompt template's placeholders and a test dataset's attributes.

Sources

1. SAP AI Core docs (SAP-docs GitHub, Sep 2026) — Evaluations (genai-evaluation scenario, use cases)
2. SAP AI Core docs — System-Defined Evaluation Metrics (8 computed + 8 LLM-as-a-judge metrics, reference requirements, scales)
3. SAP AI Core docs — Custom Metrics (structured prompts, rating rubric, worked example)
4. SAP AI Core docs — Create an Evaluation (prerequisites, configuration parameters, promptTemplateScope)
5. GitHub SAP-docs — View evaluation results (SAP-authored mirror, per-record and aggregate output shape)
6. GitHub SAP-docs — JSON Schema Match metric detail (SAP-authored mirror)
7. GitHub SAP-docs — Language Match metric detail (SAP-authored mirror)
8. GitHub SAP-docs — Grounding (RAG) evaluation metrics detail (SAP-authored mirror)
9. GitHub (Tiiiger) — BERTScore reference implementation, the ICLR 2020 metric SAP's BERT Score computed metric is built on
10. GitHub (mjpost) — SacreBLEU, the standard reproducible BLEU scoring implementation underlying SAP's BLEU computed metric
11. SAP Help Portal — Prompt Registry, generative AI hub (the prompt-registry prerequisite for an evaluation run)
12. SAP Help Portal — Generative AI hub orchestration service overview (the configurations an evaluation benchmarks)