Data Flows & Transformations
As of 2026-10-04
Data Flow = ETL surface for what Replication Flow can't do (multi-source joins, file ingest, complex derivations, Type 2 SCD). Senior rule: prefer Replication Flow; reach for Data Flow only when transformation is needed. Master 7 operators (Source/Target/Join/Projection/Aggregation/Lookup/Filter + Script). Push filters and projections to source. Header comment on every flow = inheritance protection.
What you will learn
- Work through a realistic scenario: B2B SaaS, sales-rep performance reporting needs S/4 (orders) + SuccessFactors (rep details) + monthly CSV (territory targets). 4-week build.
- Recognize and avoid the anti-pattern: Data Flow used for single-source CDC — Higher cost, higher latency than Replication Flow.
- Apply the module's core decision: Data Flow vs Replication Flow — choose Replication for single SAP source CDC; Data Flow when 2+ sources or transformation needed.
- Track mastery with the KPI: Run-time P95 (target: < planned window; red flag: > window = pushdown missed or under-CU).
Module overview
Data Flow is the ETL surface in Datasphere — for transformations Replication Flow can't do (joins across sources, complex business logic, external CSV/file ingestion, lookups, derivations). The senior rule: prefer Replication Flow when possible; reach for Data Flow only when transformation is genuinely needed.
When Data Flow wins.
- Multiple sources (S/4 + SuccessFactors + Salesforce) joined into one fact.
- File-based sources (CSV uploads, monthly Excel inputs).
- Complex business derivations (revenue allocation, currency conversion at custom rates, tax derivation).
- Slowly-changing dimensions (Type 2 history tracking).
- Pre-aggregation when raw data is too large or too sensitive to expose.
When Data Flow loses.
- CDC from a single SAP source → use Replication Flow (cheaper, native).
- Real-time / sub-minute latency → Data Flow runs in batches; latency floor ~minutes.
- Transformation that should logically live in the analytic model layer → put it in a view, not a Data Flow.
Prerequisites
- Review core concepts first: C008, C006, C005
Outcomes
- Work through a realistic scenario: B2B SaaS, sales-rep performance reporting needs S/4 (orders) + SuccessFactors (rep details) + monthly CSV (territory targets). 4-week build.
- Recognize and avoid the anti-pattern: Data Flow used for single-source CDC — Higher cost, higher latency than Replication Flow.
- Apply the module's core decision: Data Flow vs Replication Flow — choose Replication for single SAP source CDC; Data Flow when 2+ sources or transformation needed.
- Track mastery with the KPI: Run-time P95 (target: < planned window; red flag: > window = pushdown missed or under-CU).
Full module available to members. The full module adds: the decision framework · the end-to-end scenario walkthrough · the KPI scorecard · the anti-patterns · the code blocks · the knowledge check · the diagrams.