Concept card

AI Observability — Token Usage, Hallucination, Drift

As of 2026-07-24T14:00:00Z

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is AI Observability — Token Usage, Hallucination, Drift?

Hallucination detection in production runs on a fast secondary model (cross-encoder or entailment classifier, under 100ms, under 10% of the main model's token cost) scoring every response against its grounding chunks.

AI observability is the practice of continuously measuring how a generative AI system behaves once it is live — not before, in a demo, but in the messy reality of production traffic. In an SAP context, that means watching three things constantly: how many tokens each Joule or AI Core Model Gateway call consumes, whether the answers a model gives are actually true given the data it was shown, and whether answer quality quietly changes over time as prompts are tweaked, grounding sources are updated, or the underlying model is refreshed behind the API. Without this discipline, a project that scored well in a proof of concept can degrade for months before anyone notices — because nobody is watching the system the way they watch a database or an interface.

Why it matters: enterprise AI is unlike traditional software in one crucial respect — its failure modes are silent. A broken SQL report throws an error. A hallucinating AI assistant instead produces a confident, well-formatted, wrong answer, and the user has no way to tell the difference from a correct one unless they already know the answer. That asymmetry is what makes observability a governance requirement, not a nice-to-have. For a finance or supply chain use case grounded in SAP data, a single unnoticed hallucination that gets acted on can cost more than a year of monitoring tooling.

How it works

Why it matters

A budget alert firing when 7-day rolling token spend exceeds 110% of the 30-day average catches cost overruns before the monthly invoice
The grounding-verification classifier runs at under 100ms and under 10% of the main model's cost, making per-response checking economically viable
The 0.70/0.85 thresholds turn 'possible hallucination' into an automatic reject/review/pass decision rather than a judgment call

Key points

Three observability tiers: token usage (cost), hallucination detection (grounding accuracy), semantic drift (response stability over time).
Grounding score threshold: reject < 0.70, flag 0.70–0.85, pass > 0.85 — calibrate on your domain corpus, not generic defaults.
EU AI Act Article 9 compliance requires structured production logs of model inputs, outputs, grounding sources, and scores — deadline 2026-08-02 for deployed high-risk systems.
Drift regression suite: 50–200 golden Q&A pairs, run weekly, cosine similarity alert on Δ > 0.10 from baseline.
Observability sidecar pattern: lightweight Python app intercepts AI Core response, scores grounding, logs to HANA Cloud time-series, adds < 50 ms latency.
Budget alert: 7-day rolling daily token spend > 110% of 30-day average triggers before month-end invoice surprise.
Citation without verification is theatrical — models hallucinate citations nearly as readily as facts.
AI Observability — Token Usage, Hallucination, Drift is mastered only when it changes a named buyer decision.
Start with the semantic contract and control model before demonstrating the tool.
Use current SAP, analyst, study, KG, and news signals as evidence, not decoration.

Terms used on this page

Grounding score: 0–1 score from a cross-encoder or entailment classifier measuring whether every factual claim in a model response is supported by the retrieved Knowledge Graph chunks — the primary hallucination signal.
Golden dataset: Curated set of 50–200 canonical question-answer pairs used to detect semantic drift via weekly regression; ground truth for your specific domain and business vocabulary.
Semantic drift: Gradual shift in model response quality, tone, or accuracy over time — caused by prompt changes, grounding data changes, or silent model updates — detectable only via regression testing.
Prompt drift: Subset of semantic drift caused by incremental prompt engineering changes accumulating over months without a formal regression gate.
Observability sidecar: Lightweight proxy service that intercepts AI Core Gateway responses, scores grounding, and logs structured observability records to a time-series store — adds < 50 ms latency.
Entailment classifier: NLI (Natural Language Inference) model that labels whether a hypothesis (model claim) is entailed by, neutral to, or contradicted by a premise (source chunk).
Decision owner: The accountable person who accepts the trade-off and funds the next action.
Semantic contract: The shared definition of business terms, metrics, entities, and access rules used by tools and teams.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →