AI Model Evaluation
As of 2026-08-16
4 eval axes: accuracy/faithfulness · relevance · robustness · bias. Senior pattern: 50-200 prompt golden set per skill, versioned, re-run on every change. LLM-as-judge for scale (70-80 % human agreement at 1-2 % cost). Continuous weekly eval in prod, alert on > 5 % drift. Without eval, every change is a coin flip; AI feature becomes unmonitored risk in 6 months.
What you will learn
- Design a golden-set evaluation harness (50-200 versioned prompts per skill) covering accuracy/faithfulness, relevance, robustness, and bias
- Build and validate an LLM-as-judge scoring pipeline against human review (target 70-80% agreement) before trusting it at scale
- Schedule continuous production evaluation — weekly minimum, plus on every model, prompt, or retrieval-index change — with drift alerting above 5% week-over-week
- Run a stratified bias evaluation aligned to EU AI Act Art. 15 and defend the golden-set size and drift-threshold trade-offs to a sponsor
Module overview
Evaluation is the discipline that separates "we shipped AI" from "we shipped AI that works". A skill or agent without a documented eval suite is a liability — a risk you can't measure, a quality you can't defend, a regression you'll discover from users.
The 4 evaluation axes for SAP analytics AI.
- Accuracy / faithfulness — does the answer match the underlying data? For grounded skills: 100 % factual fidelity (no hallucination on numbers). For RAG: citations match retrieved chunks. Measure on golden set of 50-200 known-answer questions.
- Relevance — does the response answer the question asked? Distinct from accuracy: an accurate but off-topic answer is useless. Measure via LLM-judge or human review on representative prompts.
- Robustness — does the system degrade gracefully on edge cases? Adversarial prompts, ambiguous requests, missing data. Measure via 10-30 adversarial prompts per skill (see the red-teaming lab).
- Bias — does the system perform equally across protected attributes (gender, age, geography, role)? Measure via stratified test set, EU AI Act Art. 15 mandate.
Prerequisites
- Intermediate hands-on experience on SAP analytics projects
- Review core concepts first: C087, C029, C025
Outcomes
- Understand the core concepts behind ai model evaluation
- Apply Evaluation in a typical SAP analytics engagement
- Explain the core architecture and decision points for AI Model Evaluation
- Apply a repeatable implementation pattern in a 15-minute lab format
Full module available to members. The full module adds: the decision framework · the end-to-end scenario walkthrough · the KPI scorecard · the anti-patterns · the code blocks · the knowledge check · the diagrams.