Concept card

Data Flows & Transformations

As of 2026-07-24T14:00:00Z

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is Data Flows & Transformations?

Since Replication Flows took over default ingestion, Data Flows have narrowed to three legitimate jobs — Python enrichment, unfederatable cross-source joins, and heavy pre-aggregation.

What Data Flows are, and where they sit in the Datasphere toolkit

Data Flows are Datasphere's visual, graph-based transformation pipelines: a canvas where a modeller drags source nodes, join nodes, aggregation nodes, optional embedded Python script nodes, and a target table onto a directed acyclic graph, then activates the flow so it materialises a result set into a Datasphere local table. They exist alongside two other ways of moving and shaping data in Datasphere — Replication Flows, which handle straightforward copy-and-cast ingestion from a source to a target, and SQL or graphical views, which reshape data live inside HANA's set-based engine without persisting anything. Data Flows are deliberately the narrowest of the three: since Replication Flows took over as the default ingestion mechanism, a Data Flow only earns its complexity when the job genuinely needs something the other two cannot do.

The anatomy of a flow

A Data Flow has five conceptual stages. Source nodes bring in local Datasphere tables, federated remote tables, or uploaded files such as CSV or Parquet. Join nodes combine sources — inner, left, right, full, or cross — inside the flow rather than in a downstream view. Transform nodes apply projections, filters, calculated columns, window functions, or an embedded Python script running in a sandboxed, pandas-compatible environment. Aggregation nodes group rows and compute measures. A target node writes the materialised result to a Datasphere local table on activation, which can be triggered on demand or scheduled.

The Python option, and its real limits

Why it matters

Using a Data Flow where a Replication Flow or SQL view would do adds materialization overhead and maintenance burden with no upside.
Python nodes run sandboxed with no internet egress and a curated library whitelist — plan enrichment around that constraint, not a full Python environment.
Per-flow CU consumption is visible in Tenant Monitor — heavy Python transformation is a real, trackable cost, not a free lunch.

Key points

Five-stage anatomy: source · join · transform · aggregate · target.
Three legitimate use cases: Python enrichment · cross-source join · heavy materialised aggregate.
Replication Flow is default for ingestion (companion C005); Data Flow is the niche.
Python nodes 5-10× slower than SQL; sandboxed (no egress, curated libs).
Idempotency: TRUNCATE target first or UPSERT semantics.
Schedule + monitoring + alerting mandatory; silent failure is the #1 pitfall.
CU consumption: 1-3 CU baseline + row-volume scaling.
Joule-assisted authoring: preview Q3 2026, H1 2027 GA.
Data Flows & Transformations is mastered only when it changes a named buyer decision.
Start with the semantic contract and control model before demonstrating the tool.

Terms used on this page

Data Flow: Datasphere transformation pipeline — visual DAG of source · join · transform · aggregate · target nodes.
Python node: Sandboxed Python 3.x transformation step (pandas, numpy, scikit-learn-light). No internet egress.
DAG (Directed Acyclic Graph): Pipeline structure where nodes flow forward without cycles. Each Data Flow is a DAG.
Idempotency: Property where re-running the flow produces identical results. Requires TRUNCATE-target or UPSERT semantics.
Activation: Trigger that runs the flow. On-demand or scheduled (cron-like).
Replication Flow vs Data Flow: Replication Flow = default ingestion (copy). Data Flow = niche transformation (Python · cross-source · heavy aggregate).
Joule-assisted authoring: Natural-language Data Flow drafting. Preview Q3 2026, H1 2027 GA target.
Tenant Monitor: BDC dashboard surfacing per-flow CU consumption, run history, failure alerts.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →