Concept card

AI Bias, Fairness & Explainability in SAP AI

As of 2026-10-18

Edited by Cédric Mary, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is AI Bias, Fairness & Explainability in SAP AI?

Bias in SAP AI is tested per decision, group and language, not declared. Fairness metrics (demographic parity, equalised odds, calibration) cannot all hold at once, so the business owner picks one and records why. LLM self-explanations are not audit evidence. Recruitment and credit scoring are EU AI Act Annex III areas (Art. 10 bias examination, Art. 13 transparency, Art. 14 oversight). Generative AI hub filtering and masking add safety, not fairness; per-group evaluations do the measuring.

Bias belongs to a system in context, not to a model alone

Bias is a systematic difference in outcomes or output quality that follows a group characteristic and that the intended purpose does not justify. NIST's AI RMF 1.0 (January 2023) names three categories: systemic bias (in datasets, organisational practices and society), computational and statistical bias (often non-representative samples) and human-cognitive bias (how people read and use AI output). Each can occur without discriminatory intent, and mitigated bias is not the same as fairness. For a consultant, "the model is unbiased" is never testable; "this configuration, on this population, for this decision, meets these thresholds" is.

Where bias comes from

Four entry points recur. Training data: history encodes past decisions, and recital 67 of the EU AI Act notes that biases can be inherent in datasets, especially historical ones. Labelling: guidelines, or past approvals reused as ground truth, import human judgement into the target. Proxies: deleting a protected attribute does not remove it when postcode, part-time status, career gaps, first name or CV language correlate with it; in SAP data a cost centre or plant can play that role. Deployment context: a model validated on one population is reused on another, or its output feeds the next decision, the loop Article 10(2)(f) describes as "data outputs influence inputs for future operations".

Generative models add the foundation model's pre-training, which the customer neither controls nor sees. NIST AI 600-1 (July 2024), the Generative AI Profile, lists "Harmful Bias or Homogenization" among twelve risks.

Why it matters

Recruitment, performance monitoring and credit scoring are named Annex III high-risk areas of the EU AI Act, so bias examination (Art. 10) and human oversight (Art. 14) become delivery requirements for SAP HR and finance AI.
SAP's own AI Ethics Handbook treats employment/HR as a high-risk domain and lists discrimination as a red line, so customers and partners are expected to classify use cases the same way.
Generative AI hub filters and masking are safety layers; only per-group evaluations and counterfactual tests show whether French, German and English users, or different candidates, are treated alike.

Key points

Bias is tested per decision, population and language; "the model is unbiased" is not a testable claim.
NIST AI RMF 1.0 (Jan 2023): systemic, computational/statistical and human-cognitive bias; mitigated bias is not the same as fair.
Demographic parity, equalised odds and calibration cannot all hold at once when base rates differ (arXiv 1609.05807, 1709.02012).
LLM chain-of-thought explanations can misstate the real reason (arXiv 2305.04388); log inputs, sources and prompt versions instead.
AI Act Annex III point 4 (employment) and 5(b) (credit scoring of natural persons) are high-risk; Art. 10(2)(f)-(g) require bias examination and mitigation.
Art. 14 names automation bias: a human reviewer is only a control if able and likely to override.
Generative AI hub filtering and masking are safety layers; data masking person names cover English and locations the US only (SAP docs).
Digital Omnibus dates (Annex III to 2 Dec 2027) come from law-firm reporting and a Parliament resolution of 16 Jun 2026; verify on EUR-Lex.

Terms used on this page

Demographic parity: Fairness criterion requiring the same rate of positive decisions in every group.
Equalised odds: Criterion requiring equal true-positive and false-positive rates across groups; equal opportunity relaxes it to true-positive rate.
Calibration: A predicted score corresponds to the same real probability in every group.
Proxy variable: A feature (postcode, part-time status, first name) correlated with a protected attribute that reintroduces it after removal.
Counterfactual pair: Two inputs identical except for one group-related element, used to detect different treatment.
Model card: Short document reporting intended use, evaluation procedure and performance across groups for a model.
Global vs local explanation: Global describes overall model behaviour; local explains one individual decision.
Automation bias: Excessive reliance on automated output; named in AI Act Art. 14 as something overseers must stay aware of.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →