Concept card

AI Red-teaming + Safety Evaluation — Adversarial Prompts, Jailbreaks, Harm Eval

As of 2026-07-23

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is AI Red-teaming + Safety Evaluation — Adversarial Prompts, Jailbreaks, Harm Eval?

Red-teaming is now a compliance requirement, not a research nicety — EU AI Act Art. 9/15 documentation for a high-risk system runs 200-800 person-days, and the minimum scope for SAP deployments spans five specific checks.

What it is

AI red-teaming is the structured adversarial probing of a model and its deployed system to surface unsafe, biased, leak-prone, or jailbreak-susceptible behaviours BEFORE production exposure. In 2026 it is no longer a research curiosity — under EU AI Act Art. 9 (risk management) and Art. 15 (accuracy, robustness, cybersecurity) high-risk system providers must demonstrate red-team coverage; the Art. 9-49 documentation effort runs 200-800 person-days per high-risk system per the Munich applied-AI study cited in the Analytics Legends research ledger.

A red-team campaign has four work streams. Capability probing surfaces what the model can do — including capabilities the provider did not intend (writing exploit code, synthesising biothreat instructions, producing CSAM). Prompt-injection and jailbreak probing tests whether system prompts, retrieved-context boundaries, and tool-use guards can be bypassed by hostile user input. Bias and harm evaluation runs scenario suites (BBQ, RealToxicityPrompts, HarmBench, DecodingTrust) and surfaces disparate performance across demographic axes. Misuse evaluation probes specific high-risk uses (election manipulation, fraud, targeted harassment) and documents refusal-rate plus residual leak rate.

Why it matters

Cost is quantified, not abstract: the Munich applied-AI study cited puts Art. 9-49 documentation effort at 200-800 person-days per high-risk system.
The methodology is already codified across labs (Anthropic's RSP, OpenAI's preparedness framework, DeepMind's Frontier Safety Framework) and standards bodies (NIST AI RMF, MITRE ATLAS, OWASP LLM Top 10) — this isn't ad hoc.
For SAP specifically, the minimum red-team scope names five concrete checks: prompt-injection over retrieved-content boundaries, Joule tool-use data leaks across tenants, refusal-rate on HR/payroll queries, bias in candidate-screening outputs, and OWASP LLM-01 jailbreak resistance.

Key points

Four work streams — capability probing, prompt-injection / jailbreak, bias and harm evaluation, misuse evaluation; each documented in the Art. 9 risk register.
EU AI Act obligations — Art. 9 (risk management) + Art. 15 (accuracy, robustness, cybersecurity) require demonstrated red-team coverage for high-risk systems by 2026-08-02 (Annex III) / 2027-08-02 (product-embedded). Per the Analytics Legends research ledger.
Frameworks — Anthropic Responsible Scaling Policy, OpenAI Preparedness, DeepMind Frontier Safety Framework, NIST AI RMF 1.0, MITRE ATLAS, OWASP LLM Top 10.
Eval suites — BBQ (bias), RealToxicityPrompts, HarmBench, DecodingTrust, ToxiGen; supplement with domain-specific scenarios.
SAP-specific scope — prompt-injection over retrieved content, Joule tool-use tenant-crossing, sensitive-data refusal rate, candidate-screening bias, OWASP LLM-01 jailbreak resistance.
AI Red-teaming + Safety Evaluation — Adversarial Prompts, Jailbreaks, Harm Eval is mastered only when it changes a named buyer decision.
Start with the semantic contract and control model before demonstrating the tool.
Use current SAP, analyst, study, KG, and news signals as evidence, not decoration.
Separate verified facts from directional trends and modeled assumptions.
Define owner, metric, threshold, support path, and rollback before scaling.

Terms used on this page

Red-teaming: Structured adversarial probing of an AI system to surface unsafe, biased, leak-prone or jailbreak-susceptible behaviours before production deployment; codified under EU AI Act Art. 9 risk-management for high-risk systems.
Jailbreak: Adversarial prompt pattern that bypasses a model's safety training or system-prompt guardrails, causing it to produce content it would normally refuse; catalogued by OWASP LLM-01 and tracked in HarmBench.
Prompt injection: Attack where hostile content embedded in a tool result, retrieved document, or user input rewrites the model's effective instructions; OWASP LLM Top 10 #1 risk in production deployments.
Responsible Scaling Policy (RSP): Anthropic's voluntary commitment to define AI Safety Level (ASL) thresholds and to require specific red-team gates, deployment controls, and security measures before crossing each threshold; emulated by OpenAI's Preparedness and DeepMind's Frontier Safety frameworks.
Decision owner: The accountable person who accepts the trade-off and funds the next action.
Semantic contract: The shared definition of business terms, metrics, entities, and access rules used by tools and teams.
Control plane: The layer that applies policy, access, lineage, monitoring, and escalation across the operating model.
Evidence grade: A label that separates verified fact, directional signal, modeled assumption, and field observation.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks.

Open in the app →