Databricks-in-BDC Lakehouse Pattern

What it is

Business Data Cloud, SAP's open managed data platform, is built in large part on a strategic partnership with Databricks — a two hundred and fifty million dollar commitment announced in 2024, with zero-copy, bidirectional data sharing as its central technical promise. In practice this means SAP applications such as S/4HANA, SuccessFactors, Ariba, and BW/4HANA expose their data to the Databricks lakehouse through Delta Sharing, the open protocol for sharing Delta Lake tables without physically moving or duplicating the underlying data, while Databricks workloads can write enriched or derived datasets back into SAP-native storage through the same protocol. For an SAP analytics architect, this is the pattern to reach for when an engagement combines Python-heavy data science workloads, large raw-data ingestion volumes, or an existing Databricks investment with a need for governed SAP business data — and it is the wrong pattern, adding unnecessary platform complexity, when the engagement is SAP-anchored business-user modeling with no data-engineering team to operate the Databricks side.

Why it matters

• BDC Connect registers Databricks tables in the BDC catalog with row-level security inherited directly from SAP's own Data Access Controls — no duplicate governance layer to maintain.
• The pattern explicitly solves the 'data gravity trap': trusted SAP data locked in SAP's compute boundary while ML pipelines live in Databricks, previously bridged only by ETL and schema mapping.
• The engagement-fit criterion is stated directly — this is not a universal pattern, it's conditional on the client's existing data science footprint.

Key points

• BDC = SAP's open data platform anchored in $250M Databricks partnership; core protocol = Delta Sharing (zero-copy, bidirectional).
• BDC Connect registers Databricks (and Snowflake, S3/Iceberg, BigQuery) in the BDC data catalog — no custom connectors, uses Unity Catalog Delta Sharing endpoint.
• Unity Catalog row-level security and SAP DAC rules must be aligned — misalignment means a Databricks notebook can see rows a SAP user in the same role cannot.
• Use when: existing Databricks investment + Python-native data team + Spark-scale ML (>500M rows) + non-SAP data in Delta Lake.
• Do not use when: SAP-only estate + business-user modeling + no Databricks skills + latency <500ms required.
• Write-back pattern: Databricks ML outputs (scores, forecasts) → Delta table in Unity Catalog → Delta Sharing → Datasphere remote table → SAC KPI or Joule grounding.
• Dremio, acquired by SAP (announced 4 May 2026, completed 6 July 2026), is intended to add a query-time federation and universal-catalog tier alongside Datasphere and Databricks — integration in progress, no dated GA for Dremio-specific capabilities as of this review; it does not replace the Databricks-in-BDC pattern.

Terms used on this page

Delta Sharing

Open Apache-licensed protocol from Databricks for sharing Delta Lake tables without copying data — readers access signed-URL Parquet files at the source storage location.

Unity Catalog

Databricks' unified governance layer for data and AI assets across all Databricks workspaces — enforces row-level security, column masking, and audit logging.

BDC Connect

SAP's catalog-time federation component in Business Data Cloud — registers external sources (Databricks, Snowflake, S3/Iceberg, BigQuery) in the BDC data catalog without data movement.

Delta Lake

Open-source ACID-compliant storage layer from Databricks that brings reliability to data lakes — tables stored as Parquet + transaction log (Delta Log).

Data Access Control (DAC)

SAP Datasphere's row-level security framework — filters data at query time based on the requesting user's authorisation, enforced on Analytic Models and remote tables.

Zero-copy integration

Data sharing architecture where no physical data copy is created — the consumer reads from the producer's storage location via metadata and signed URLs, eliminating ETL pipelines and storage duplication.

Sources

1. SAP Datasphere — BDC Connect documentation (Help Portal)
2. Delta Sharing — open protocol documentation (delta.io)
3. Apache Arrow Flight specification (used by Dremio query federation)
4. SAP to Acquire Dremio — official SAP News
5. Databricks Docs — Lakehouse architecture
6. Databricks Docs — Unity Catalog data governance
7. SAP Sapphire Orlando 2025: SAP and Databricks open a bold new era of data and AI - BDC2767 — SAP Community (Technology Blog Posts by SAP)
8. The Open Catalog of SAP Data Intelligence — SAP Community (Technology Blog Posts by SAP)
9. GitHub delta-io/delta-sharing — Python client library and open protocol
10. PyPI — delta-sharing Python client package
11. Databricks Docs — MLflow on Databricks
12. Databricks — Unity Catalog product page
13. SAP Help Portal — Sharing Data Products from SAP Databricks to SAP Business Data Cloud (SAP's official procedure for the Databricks-to-BDC return path)
14. SAP Help Portal — Creating and Working with Derived Data Products in SAP Databricks (publishing ML outputs as custom data products)
15. SAP Learning — Sharing data back to SAP Business Data Cloud (training unit on the bidirectional sharing loop)