AI Red-Teaming for SAP — Adversarial Prompt Testing

What red-teaming is, and whose job it is

AI red-teaming is the adversarial counterpart of functional testing: instead of checking that the system does what it should, testers try to make it do what it must not — leak data, ignore its instructions, take unauthorised actions, produce harmful or false content, or run up cost. SAP states that Joule is red-teamed by an internal SAP AI red team and external evaluators and that every external LLM integration is risk-assessed and benchmarked (SAP Community, 2025 and August 2026). That covers SAP's product. It does not cover what you add: your prompts and orchestration configs, the documents you ingest for grounding, the tools your Joule Studio agents may call, the filter thresholds you set and the backend roles Joule inherits through principal propagation (C120). Those are the customer's attack surface and the customer's test.

Why it matters

• Indirect prompt injection arrives through documents, free-text fields and tool outputs, not through the chat box — a test plan that only types into Joule misses the most likely attack.
• SAP's red-teaming of Joule does not cover your prompts, grounding repositories, agent tools or backend roles; that attack surface is yours to test.
• Platform changes such as Prompt Shield no longer scanning system messages (13 August 2026) can silently remove a defence — only repeated tests reveal it.

Key points

• Red-teaming = adversarial testing of the whole AI system (prompts, retrieval, tools, filters, roles), before go-live and after every relevant change.
• SAP red-teams Joule (internal SAP AI red team + external evaluators); customers must test their own configurations and agents.
• EU AI Act: Art. 15(5) resilience against AI-specific attacks for high-risk systems; Art. 55(1)(a) adversarial testing is a GPAI systemic-risk provider duty.
• Digital Omnibus (in force 27 Jul 2026): Annex III high-risk obligations from 2 Dec 2027, Annex I from 2 Aug 2028.
• Taxonomy: OWASP Top 10 for LLM Applications 2025 — prompt injection, sensitive information disclosure, excessive agency, system prompt leakage, vector weaknesses, unbounded consumption…
• SAP-specific vectors: documents ingested by grounding, free-text fields read by agent tools, over-broad backend roles inherited via principal propagation.
• Automate with generative AI hub Evaluations: adversarial dataset + Content Filter on Input/Output metrics + custom judge metric for expected safe behaviour.
• Re-test triggers: model upgrade or deprecation, filter changes (Prompt Shield 13 Aug 2026), new tools, new repositories, new user groups.

Terms used on this page

AI red-teaming

Structured adversarial testing that tries to make an AI system violate its intended behaviour, security or policies.

Direct prompt injection

A user's input instructs the model to ignore or override its instructions.

Indirect prompt injection

Malicious instructions embedded in content the system retrieves or reads — documents, records, tool outputs — rather than typed by the user.

Excessive agency

OWASP LLM06: an LLM-based system granted more functions, permissions or autonomy than its purpose requires.

System prompt leakage

OWASP LLM07: disclosure of system instructions that may contain rules, references or secrets.

Vector and embedding weaknesses

OWASP LLM08: risks from poisoned, stale or over-shared content in retrieval stores.

Unbounded consumption

OWASP LLM10: requests or loops that exhaust capacity, rate limits or budget.

Expected safe behaviour

The response an adversarial test case should produce (refusal, context-only answer, confirmation request), used as the scoring reference.

Sources

1. OWASP Top 10 for LLM Applications 2025
2. EU AI Act — Article 15 (accuracy, robustness, cybersecurity)
3. EU AI Act — Article 55 (obligations for GPAI models with systemic risk)
4. European Commission — AI Omnibus enters into force (27 Jul 2026)
5. SAP Community — Trust by Design: Security, Guardrails & Governance in Joule (Aug 2026)
6. SAP Community (SAP) — Principles to Practice: Securing SAP Business AI
7. SAP AI Core — Content Filtering (SAP-docs)
8. SAP AI Core — System-Defined Evaluation Metrics (SAP-docs)
9. SAP AI Core service guide incl. What's New and usage restriction (PDF, 4 Sep 2026)
10. Integrating Joule with SAP Solutions — document grounding caveats (PDF, 14 Sep 2026)
11. SAP News Center — AI agents work at scale: AI Governance Assistant, EU AI Act + NIST classification (22 Sep 2026)
12. MLflow — LLM evaluation documentation
13. Microsoft / Azure — PyRIT, Python Risk Identification Tool for generative AI (GitHub)