Task Chains & Operations
As of 2026-10-04
Task chains = DAG orchestration. Senior pattern: design alongside data layout week 1, failure modes per task (retry/skip/halt), SLA-driven runtime budget per node, telemetry to BTP Audit Log + monitoring. Work backwards from dashboard SLA to ingestion. Tasks exceeding budget = redesign target.
What you will learn
- Work through a realistic scenario: DAX-40 finance daily close, 7 AM dashboard SLA, 12 sources, 3 transformations, 3 analytic models.
- Recognize and avoid the anti-pattern: One-size-fits-all halt-on-fail — Notify task failure stops the entire chain.
- Apply the module's core decision: Failure mode per task — choose Retry+backoff transient · skip+alert non-critical · halt+alert critical, not One-size-fits-all halt-on-fail.
- Track mastery with the KPI: SLA hit rate (target: > 99 % runs complete on time; red flag: < 95 % = budget overrun pattern).
Module overview
Task chains = the orchestration layer in Datasphere. Without task chains, every Replication Flow + Data Flow runs on its own schedule, dependencies are tribal knowledge, and the 6 AM dashboard refresh fails silently when a source feed is late. Senior consultants design the chain alongside the data layout in week 1.
A task chain = directed acyclic graph (DAG) of:
- Replication Flow runs (CDC delta refresh).
- Data Flow runs (transformations).
- Persisted view refresh.
- HTTP callouts (notify Slack, trigger downstream pipeline).
- Conditional branches (success / failure).
Senior craft signal — failure handling. A junior writes a chain that stops on first failure. A senior designs failure modes per task: retry with backoff for transient errors, alert + skip for non-critical, alert + halt for critical, conditional notification per severity. The chain is auditable and survives 3 AM ops calls.
SLA-driven design. If the business needs the dashboard at 7 AM, the chain must complete by 6:45 AM with 15 min buffer. Work backwards: synthesis → modeling → ingestion. Each task's runtime budget = SLA / chain depth. Tasks that exceed budget = redesign target.
Prerequisites
- Review core concepts first: C007, C008, C041
Outcomes
- Work through a realistic scenario: DAX-40 finance daily close, 7 AM dashboard SLA, 12 sources, 3 transformations, 3 analytic models.
- Recognize and avoid the anti-pattern: One-size-fits-all halt-on-fail — Notify task failure stops the entire chain.
- Apply the module's core decision: Failure mode per task — choose Retry+backoff transient · skip+alert non-critical · halt+alert critical, not One-size-fits-all halt-on-fail.
- Track mastery with the KPI: SLA hit rate (target: > 99 % runs complete on time; red flag: < 95 % = budget overrun pattern).
Full module available to members. The full module adds: the decision framework · the end-to-end scenario walkthrough · the KPI scorecard · the anti-patterns · the code blocks · the knowledge check · the diagrams.