Unity Catalog

Architecture Patterns
Bernhard Walter
2023-09-28
©2023 Databricks Inc. — All rights reserved 1
Table of Contents
Patterns

1. Software Development Life Cycle setup

2. Cross cloud region data access
3. BI for multiple Business Units
a. In a single cloud-region
b. In multiple cloud-regions
4. Phased Migration to UC
5. ML Setup

©2023 Databricks Inc. — All rights reserved 2

Pattern 1:
Software Development Life
Cycle (SDLC) setup

©2023 Databricks Inc. — All rights reserved 3

Scenario: Software Development Lifecycle
● Customers use different environments for the different stages of development. The minimal
setup is to have two environments “Development & Test” and “Production”
● Most customers also split “Development & Test” into two or more environments: “Development”
and quality assurance environments being called “Test”, “QA”, “Staging”, “Integration”, …
● This scenario describes the recommended “DEV” - “STG” - “PRD” setup but the approach will
also work for more or less environments depending on customer’s setup
● Many enterprise customers also isolate these environments from a storage and compute
perspective using different storage containers, VNets/VPCs and Databricks Workspaces

DEV STG PRD

Databricks Workspace Workspace Workspace

DEV STG PRD

Cloud storage storage container storage container storage container

©2023 Databricks Inc. — All rights reserved

 Unity Catalog Isolation Options (SDLC)

Delegation of Management Cross-Workspace Siloed

(admin isolation) Metastore Admin
Each SDLC environment has its own admin
Unity Catalog DEV DEV STG STG PRD PRD
Metastore Admin catalog Admin catalog Admin catalog
Workspace to
catalog binding
The PRD workspace is fully isolated and has
its own catalog. DEV workspace has access
to DEV and STG catalog
DEV STG PRD
workspace workspace workspace
Storage isolation
This example separates the storage locations
on catalog level (typically sufficient)

UC Access Control DEV STG PRD

Users should only gain access to data/
storage storage storage
metadata based on agreed access rules

©2023 Databricks Inc. — All rights reserved 5

 Software Development Lifecycle setup w/ UC
Approach *
Grant <JA> full access to tables
● There will only be one Metastore per cloud region, which will on PRD data
be used for all related SDLC workspaces (DEV, STG, PRD, …) Grant <UG> full access to Optional: grant <UG> read-only
tables on DEV data access to tables on PRD data
○ Isolate the environments on the Catalog level of the DEV PRD
3-level namespace of Unity Catalog. Assign DEV, STG, DEV
admin
Unity PRD
admin

PRD workspaces to their respective catalog only. Catalog Metastore Catalog

○ Isolate the DEV, STG, PRD data locations by assigning metadata metadata
assign assign
dedicated managed buckets/containers to the access access

catalogs
DEV PRD
○ Isolate admin scope by delegating administration of assigned
Workspace Workspace
assigned
managed managed
the catalogs to different admins for DEV, STG, PRD container/ container/
bucket Work Work bucket
● Catalog names can be combinations of SDLC and business / load load
organizational unit names, e.g. sales_dev, sales_prd,
engineering_dev read/write via read/write via
<DA> token <PA> token
● Access to workspaces, clusters and endpoints needs to be
configured accordingly
DEV bucket/ Delta PRD bucket/
container container
As a best practice, use at minimum 2 workspaces Grant access to <DA>
Lake Grant access to <PA>
(others/prod), but ideally 3 workspaces (dev/stg/prod) *
<DA> DEV System Account for UC <JA> System Account to execute jobs in PRD
<PA> PRD System Account for UC <UG> User Group working in DEV and Prod

©2023 Databricks Inc. — All rights reserved 6

* To simplify the diagram, only DEV and PRD are shown
Pattern 2:
Cross cloud region data access

©2023 Databricks Inc. — All rights reserved 7

 Cross region access - UC integrated
Cloud region 1 (recipient)
1 Databricks-to-Databricks Delta Sharing (provider provides data)
• Region 2 will share tables with the metastore 1 of region 1
meta WS 1
• Region 1 will create catalogs in metastore 1 to hold the tables store 1
from the share of region 2 User
Catalog for Share 1
• Users in region 1 can then discover and access the shared local
ACLs
data from region 2 via metastore 1 Foreign Catalog 2
2 Lakehouse Federation (provider provides compute)
• Region 1 will connect to Databricks in region 2 represented in Unity
metastore 1 by a Foreign Catalog Catalog 1 2

• Users in region 1 can then access the shared data from region $ $
2 via metastore 1
delta
connection
In both cases Share 1 sharing
authn
authn

• region 1 can additionally control access for their users on top meta WS 2
in metastore 1 store 2 2 Cluster or SQl warehouse

• egress costs arise from data leaving region 2 (region 1 Data flow
reading) read/write
Object Store Access Control by
/ UC / Workspace

Note: We strongly do not recommend registering common tables as Cloud region 2 (provider) $
Egress cost from one
region to another
External Tables in more than one metastore

©2023 Databricks Inc. — All rights reserved

 Cross region access - Remote compute
Cloud region 1 (recipient)
3 JDBC / ODBC (provider provides compute) or
4 Databricks SQL Connector for Python (provider provides compute)
meta WS 1
• Users in region 1 will connect to specific compute resources store 1
(clusters or SQL Warehouses) in region 2 User

• Queries will be executed on the compute resource in

Workspace 2
• No integration into metastore of region 1 $

• egress costs arise from data leaving region 2 (region 1 reading) Unity 3
or data leaving region 1 (region 1 writing) Catalog 4
$
user
3 authn
Catalogs
4
meta WS 2
store 2 Cluster or SQl warehouse

Data flow
read/write
Object Store Access Control by
/ UC / Workspace

Egress cost from one

Cloud region 2 (provider) $ region to another

©2023 Databricks Inc. — All rights reserved 9

Managed replication
Relevant in cases when egress costs get too high

Recipient (region 1) Approach:

1. Replicate data in intervals from provider to recipient (driven by the recipient in
Object write
Store Trep region 1)
2 2. Query locally
Trep
Query
meta WS 1
Replication Types:
store 1
User • Managed full replication (only reasonable for small data)
1
Replication • Managed incremental replication (check the used technology whether
incremental replication is supported)
Unity
Catalog
Impact:
$
• Data freshness depends on refresh interval time (business impact)

meta WS 2 • Higher maintenance effort for managing the replication pipeline

store 2 • Egress cost impact
T • Full replication: Full egress cost each time it will be replicated
Object read • Incremental replication: One time cost for the whole data set
Store T
Provider (region 2)

©2023 Databricks Inc. — All rights reserved 10

Pattern 3:
BI for multiple Business Units

©2023 Databricks Inc. — All rights reserved 11

Five core processes
To be taken into account when defining the architecture patterns

Data Production Data Publishing Data Consumption

Creation of data via Ingest & ETL processes A deliberate step to provide access to a Use own data or published data by
or by different business oriented teams. data set for other consumers. others for analysis, reporting, ML, …
What is a meaningful data set? Where to publish which data set? Generate value from data

Data Governance
Metadata management and access control.
Whom to provide which access at which point of time?

Platform Operations
Data teams will work on one or more platforms. Depending on the enterprise, data teams can operate their own platform, or there is a
central team that comes up with a blueprint for the setup.

©2023 Databricks Inc. — All rights reserved 12

Distributed vs Centralized Publishing
Distributed Publishing
consume
● BUs share the data they created themselves BU 2
● Each BU is responsible for data quality (treat published data as a “data product”) and access permissions
● BUs are not isolated from an infrastructure perspective (e.g. no network restrictions)
Central
● Usually there is still a central team to provide platform operations. It is responsible to define secure and
compliant blueprints, setup infrastructure, etc. This team also provides capabilities to ensure private data
BU
can be protected (e.g. by managing access from a network perspective). However, it does not play a role BU 1 BU 3
for publishing and configuring data access.
● The Central BU can also publish common data to other BUs

Centralized Publishing
● A central BU serves as a Hub publish
BU 2 consume
● BUs are isolated from each other, e.g. by network access restrictions
● The central BU provides
● central data storage and the central metastore to publish (data, metadata & ACLs) datasets for all Central
BUs and usually BU
● central data governance and applies quality assurance on all published data BU 1 BU 3
● platform operations (secure and compliant blueprints, infrastructure setups, …)
● Often the central BU publishes own data (via a central data engineering team)

©2023 Databricks Inc. — All rights reserved 13

Flavors of Centralized Publishing
Strictly governed (PULL) Loosely governed (PUSH)
● BU notifies Central BU that a table should be published ● BU copies data to BU folder in the central container (each
● Central BU validates data quality and compliance of BU does only have access to its folder)
metadata to the defined rules ● BU creates schema in its central catalog for published data
● Central BU copies data from BU container to the BU folder in (each BU does only have access to its catalog)
the central container ● BU defines ACLs on the new table
● Central BU creates schema in central UC catalog ● BU sets up a job to regularly update the central data (batch
● Central BU sets up a job to regularly update the centrally or streaming)
published data (batch or streaming) ● Central team mainly provides publishing infrastructure and
Note: This pattern also applies to scenarios where BUs are not applies lightweight data governance (monitoring data
skilled enough to properly publish data, i.e. central team providing quality, validating data is kept up to date, …)
a publishing service instead of primarily applying governance

data data data

Central container BU 1 Pull Central container BU 1
Central B Central B
BU schema table BU schema table
BU1_published X B job BU1_published X B
metadata,
Unity ACLs Unity
Metastore Metastore metadata,
ACLs
schema table schema table
BU_1_prd Y B BU_1_prd Y B job
BU 1 BU 1 data
data data
BU 1 container BU 1 container
B B Push
©2023 Databricks Inc. — All rights reserved 14
Pattern 3.1:
BI for multiple Business Units
Single cloud-region

©2023 Databricks Inc. — All rights reserved 15

Scenario: Single cloud-regions BI

Organization & roles:

Central BU - Provides blueprints for BU platform setups
- Provides central platform management
- Provides data centrally (global and possibly BU specific data)

BU 1: - Works with local data combined with central data

- Can provide created data sets to other BUs (publish data)

BU 2: ...

...

Cloud setup: All BUs in one cloud-region

Use Case: BI use case focussed on Reporting and Analysis via BI Tools

©2023 Databricks Inc. — All rights reserved 16

High level blueprint
Single cloud-region
Central Business Unit (Central BU)
● Maintains central data pipelines
● Provides access to central data for each BU Central BU BU 1
BI Tools
Other Business Units (BU)
● Use own cloud storage for BU data
Databricks Cloud
● Mainly use BI tools and Databricks SQL to work with
Workspaces Storage
own data and centrally provided data (BU 1)
(BU 1)
● Can share data on a central cloud storage or their own Unity
storage - always governed by Unity Catalog
Cloud Databricks Catalog
(details see next slides)
Storage Workspaces
BU 2
Meta
BI Tools
Unity Catalog (central) (central)
store
● Since this setup is in one cloud region, there is a single
metastore covering all BUs (including the Central BU) Databricks Cloud
Workspaces Storage
● Consider Unity Catalog’s workspace binding, and (BU 2)
storage and admin isolation features when setting up (BU 2)
central BU and other BUs.

©2023 Databricks Inc. — All rights reserved 17

Option 1: Distributed Publishing
Single cloud-region
Central BU BU 1
Data Production
● Central Ingest & ETL by the central BU Cloud Storage
● Other BUs create (business) data sets Cloud central
Databricks
Storage data
Workspaces BU 1
Data Publishing (BU 1) DEVBU
data
1
● Central BU publishes data to the central PRD storage and STGBU
data1
into the catalog of the Central BU in UC Central Unity PRD data
BUs publish to their PRD storage and into BU catalogs of UC DEV data
● Catalog BI Tools
(see metadata mgmt later) Databricks
Data Governance (distributed) Meta data published
Workspaces
Central store BU 2 by BU 1
● Central team and each BU can work independently on their (central)
STG data
catalogs (publishing and setting ACLs)
Data Consumption Cloud Storage
Databricks
● Published data will be discovered in the metastore and Workspaces BU 2
Central central
(BU 2)
consumed from PRD storage of the BUs (central or other) DEVBU
data
2
PRD data data
STGBU
data
Platform operations 2
PRD data
● Central team provides platform blueprints and creates
environments for BUs (automated) BI Tools

Publish tables by configuring access permissions

Consume other data
Create and consume own databases/tables
Access metadata 18
©2023 Databricks Inc. — All rights reserved
Option 1: Distributed Publishing Process

catalog . schema . table bucket or container folders

ACLs ACLs
Central_prd schema Z table C Central prd / dataset-C

ACLs ACLs
Unity
Metastore
BU_1_prd schema Y table B BU 2 prd / dataset-B

ACLs ACLs
BU_2_prd schema X table A BU 1 prd / dataset-A

Set ACLs location

©2023 Databricks Inc. — All rights reserved 19

 Option 1: Metadata Management
Permissions on catalog level *

catalog . schema . table/view Owner Central BU 1 BU 2

Central tables,
Central_dev schemas views Central U C None None

Central tables,
Central_stg schemas views Central U C None None

Central tables, U C U ** U **
Central_prd schemas views Central

BU tables,
BU_1_dev schemas views BU 1 None U C None

Unity BU_1_stg
BU tables,
BU 1
schemas views
None U C None
Metastore
BU tables,
BU_1_prd schemas views BU 1 U ** U C U **

BU tables,
BU_2_dev BU 2 None None U C
schemas views
BU tables,
BU_2_stg schemas views BU 2 None None U C

U USE CATALOG BU tables,

BU_2_prd schemas views BU 2 U ** U ** U C
C CREATE SCHEMA

_prd: production
_stg: staging
_dev: development

©2023 Databricks Inc. — All rights reserved * To access tables, USE SCHEMA and SELECT also need to be provided on schema and table level 20
** Catalogs where other BUs can be given access to schemas and tables/views by Central team
Option 2: Centralized Publishing
Single cloud-region
Data Production Central BU BU 1
● Central Ingest & ETL by the central BU
Other BUs create (business) data sets Cloud Storage
● Cloud central and BU published data Databricks
Data Publishing Storage Workspaces BU 1
● Central BU publishes data in central PRD storage and into (BU 1) DEVBU
data
1
the PRD catalog of the Central BU in UC STGBu
data1
Central Unity PRD data
● BUs requests from Central team to publish from their PRD DEV data
storage to central PRD storage and into the BU catalog in Catalog BI Tools
UC that is maintained by Central Databricks
Meta
Data Governance (centralized) Workspaces
Central store BU 2
(central)
● Central team and each BU (for non published) data can STG data
work independently on their catalogs
Cloud Storage
● Central team applies additional quality assurance and Databricks
maintains ACLs in the central BU catalog Workspaces BU 2
Central (BU 2) DEVBu
data
2
Data Consumption PRD data
central and BU published data STGBu
data
2
● Published data will be discovered in the Central catalog PRD data
and consumed from the central PRD storage
BI Tools
Platform operations
● Central team provides platform blueprints, creates
environments for BUs (automated) Publish data sets / tables and configure access permissions
Consume data from other BUs
● Central team could provide common data services
Create and consume own databases/tables
Access metadata 21
©2023 Databricks Inc. — All rights reserved
Isolated, e.g. by network access restrictions
Option 2: Centralized Publishing Process
catalog . schema . table bucket or container folders

Central / Central / …

ACLs ACLs location

BU1_published schema X table B

2. Publish
/ BU 1 / dataset-B

BU2_published schema Y table A metadata * / BU 2 / dataset-A

ACLs ACLs
Unity G
3. Set ACLs G
Metastore
BU_2_prd schema Y table A BU 2 / dataset-A

1. Publish
data *
BU_1_prd schema X table B BU 1 / dataset-B

G Central data governance:

● Control data quality and compliance with publishing rules (naming conventions, …)
● Control data access (set ACLs)

©2023 Databricks Inc. — All rights reserved 22

* For details see push/pull discussion earlier
 Option 2: Metadata Management
Permissions on catalog level *

catalog . schema . table/view Owner Central BU 1 BU 2

Central tables,
Central_dev schemas views Central U C None None

Central tables,
Central_stg schemas views Central U C None None

Central tables,
Central_prd Central U C U ** U **
schemas views
Central tables,
BU1_published schemas views Central U C U ** U **
Central tables, U C U ** U **
BU2_published schemas views Central

Unity BU tables,
BU 1 U C
BU_1_dev schemas views
None None
Metastore
BU tables,
BU_1_stg BU 1 None U C None
schemas views
BU tables, U
BU_1_prd schemas views BU 1 U C None

U USE CATALOG BU tables,

BU_2_dev schemas views BU 2 None None U C
C CREATE SCHEMA
BU tables,
BU_2_stg schemas views BU 2 None None U C
_prd: production
_stg: staging BU tables,
_dev: development
BU_2_prd schemas views BU 2 U None U C

©2023 Databricks Inc. — All rights reserved * To access tables, USE SCHEMA and SELECT also need to be provided on schema and table level 23
** Catalogs where other BUs can be given access to schemas and tables/views by Central team
Granularity of Centralized Publishing

Centralized Publishing does not mean that all production Central BU BU 1

data from a participating BU will be copied over to the containers/buckets tables tables containers/buckets
central storage location: 1
Central 6 … … BU
● In the centralized publishing model, BUs will still DEV data DEV data
maintain their own Dev/Stg/Prd environments 4

● BUs initially create production tables for internal Central

6 1
… … BU
consumption STG data STG data
4
● Some of these tables might be relevant for other BUs,
hence worth publishing Central BU
PRD data PRD data
● as a full copy of the original
partial copy
● as a partial copy (filtered columns or rows, Published BU 1
1’ 1
masking, …) 2 full copy
2 Used by
Note: Choose push / pull publishing according to users
Published BU 2 5 3 of BU 1
customer requirements unpublished
Published Central 6 4

©2023 Databricks Inc. — All rights reserved 24

System Account Mappings
Example dataset 1
Proj 1
Central BU BU X dataset m
ID
Provider DEV Proj 2 dataset 1
Databricks Cloud dataset 1
Workspace Storage
Unity User Shared dataset 2
(PRD) Mgmt
Shared dataset n
SQL (PRD) Proj 1
endpoint(s)
Unity
Cloud Catalog / STG Proj 2
Proj 1 Shared
Storage (STG) Shared
Proj 2
Shared Proj 1
Central
(DEV)
PRD PRD Proj 2

Databricks Proj 1 Shared

(PRD) SA_BU-X_proj-1_dev
Workspace SA_BU-X_proj-1_stg
SA-BU-X_proj-1_prd (DEV or STG) Proj 1 SA_BU-X_proj-1_prd Data Access Pattern
SA-BU-X_proj-2_prd SQL
(STG) SA_BU-X_proj-2_dev ● Read/write
endpoint(s) SA_BU-X_proj-2_stg
Proj 1 SA_BU-X_proj-2_prd ○ ‘SA_BU-X_proj-y_dev’ to ‘/DEV/Proj y’
Proj 1
(DEV) ○ ‘SA_BU-X_proj-y_stg’ to ‘/STG/Proj y’
Different System Proj 2
Accounts (SA) for cloud Different System ○ ‘SA_BU-X_proj-y_prd’ to ‘/PRD/Proj y’
storage level audit if Accounts (SA) for cloud ● Either read/write or read only
needed storage level audit if
needed ○ ‘SA_BU-X_proj-y_dev’ to ‘/DEV/Shared’
○ …
©2023 Databricks Inc. — All rights reserved 25
Pattern 3.2:
BI for multiple Business Units
Multiple cloud-regions

©2023 Databricks Inc. — All rights reserved 26

Scenario: Multiple cloud-regions BI

Organization & roles:

Central BU - Provides blueprints for BU platform setups
- Provides central platform management
- Provides data centrally (global and BU specific data)

BU 1: - Works with local data combined with central data

- Can provide created data sets to other BUs (publish data)

BU 2: ...

...

Cloud setup: BUs are in multiple cloud-regions

Use Case: BI use case focussed on Reporting and Analysis via BI Tools

©2023 Databricks Inc. — All rights reserved

High level blueprint
Multiple cloud-regions
Central Business Unit (BU)
Cloud region 1 Cloud region 2
● Maintains central data pipelines
● Provides access to central data for each BU Central BU BU 1
BI Tools
Other Business Units (BU)
● Use own cloud storage for BU data Databricks Cloud
● Mainly use BI tools and Databricks SQL to work with Workspaces Storage
own data and centrally provided data (BU 1) (BU 1)

● Can share data on a central cloud storage or their

own storage - always governed by Unity Catalog
(details see next slides) Cloud Meta Unity Meta
Unity Catalog Storage store 1 Catalog store 2
(central)
● Since this setup is in multiple cloud regions, there is
one metastore per region.
Databricks
● Consider Unity Catalog’s workspace binding, and Workspaces Databricks Cloud
storage and admin isolation features when setting (central) Workspaces Storage
up central BU and other BUs. (BU 2) (BU 2)

BI Tools
BU 2

©2023 Databricks Inc. — All rights reserved 28

Option 1: Distributed Publishing
Multiple cloud-regions
Cloud region 1 Cloud region 2
Data Production
● Central Ingest & ETL by the central BU Central BU Cloud Storage
● Other BUs create (business) data sets Databricks
data published
1 Workspaces BU 1
Data Publishing Cloud DEVBU
data
by central BU (BU 1) 1
● Central team publishes data to central PRD storage Storage STGBU
data1
and into UC Metastore 1 PRD data
● BUs publish to their PRD storage and into their BI Tools
Central
Metastore (here Metastore 2)
DEV data BU 1
● For Delta Sharing, Metastore 2 will be a recipient of
Metastore 1 to share central metadata with region 2 Meta Unity Meta
If BUs exist in separate cloud regions and want to share data
● store 1 Catalog store 2 published by BU 1
data, Databricks-to-Databricks Sharing needs to be Central
set up for their Metastores (not shown) STG data BU 2
Data Governance (distributed) Databricks BI Tools
3
● All BUs set ACLs for their own and delta-shared data Workspaces Cloud Storage
Data Consumption (central) BU 2
Central 2
● Published data will be discovered in BUs Metastore and Databricks DEVBU
data
2
PRD data
consumed from BUs PRD storage 1 Workspaces STGBU
data
2
data published
Platform operations by central BU
(BU 2) PRD data

● Central team to provide platform blueprint and create

environments for BUs (automated)
Publish tables by configuring access permissions 1 Databricks-to-Databricks Delta Sharing
Consume data from other BUs 2 Lakehouse Federation
©2023 Databricks Inc. — All rights reserved Create and consume own databases/tables 3 JDBC / ODBC 29
Access metadata
Option 1: Distributed Publishing Process
catalog . schema . table bucket or container folders

1. Set ACLs
ACLs ACLs
Unity
Metastore 1
Central_prd schema Z table C Central prd / dataset-C

location Publish and share

2. Share with 1 or 2
Metastore 2 3. Set ACLs for Central data
Shared data (ACLs for step 1 & 3
need to be aligned)
ACLs ACLs
Central publish. schema Z’ table C’

1. Set ACLs
ACLs ACLs
Unity
Metastore 2
BU_1_prd schema Y table B BU 1 prd / dataset-B

Publish and share BU

data with other BUs
ACLs ACLs
BU_2_prd schema X table A BU 2 prd / dataset-A

1 Databricks-to-Databricks Delta Sharing

©2023 Databricks Inc. — All rights reserved 30
2 Lakehouse Federation
 Option 1: Metadata Management
Permissions on catalog level $

catalog . schema . table/view Owner Central BU 1 BU 2

Central_dev Central U C None None

Unity Central_stg Central U C None None

Metastore 1 region 1
Central Central U C * * specific
Central_prd tables U U
databases ACLs

region 2
shared shared * * specific
Central publish. databases tables Central U C U U
ACLs
BU tables,
BU_1_prd BU 1 None # U C U **
databases views
Unity BU_1_… BU 1 None U C None
Metastore 2
BU_2_prd BU 2 None U ** U C
U USE CATALOG
C CREATE SCHEMA BU_2_ … BU 2 None # None U C

_prd: production
_stg: staging
_dev: development
$ to access tables, USE SCHEMA and SELECT also need to be provided on schema and table level
* catalog where other BUs can be given access to schemas and tables/views from Central team
©2023 Databricks Inc. — All rights reserved ** catalogs where other BUs can be given access to schemas and tables/views
31
# if Central needs access to BU data, Delta Sharing or Lakehouse Federation need to be set up
Option 2: Centralized Publishing
Multiple cloud-regions
Cloud region 1 Cloud region 2
Data Production
● Central Ingest & ETL by the central BU Central BU
● Other BUs create (business) data sets central and BU Cloud Storage
published data
Databricks
Data Publishing 1 Workspaces BU 1
● Central team publishes data in central PRD storage Cloud (BU 1) DEVBU
data
1
and into the catalog of the Central BU in Metastore 1 Storage STGBU
data1
● BUs publish by requesting from Central team to PRD data
publish from their PRD storage to central PRD storage Central BI Tools
and into the BU catalog in Metastore 1 DEV data BU 1
● Central BU will configure access control on
delta-shared data in Metastore 2 Meta Unity Meta
● Metastore 2 will be a recipient of Metastore 1 and all store 1 Catalog store 2 Isolated, e.g. by network
access restrictions
published metadata will be shared with Metastore 2 Central
Data Governance (centralized) STG data BU 2
● Access permissions are defined in Unity Catalog. Databricks 3 BI Tools
Central team will maintain access control for all Workspaces Cloud Storage
published data (with input from BUs) (central) BU 2
Central 2
● Central team might apply additional quality control Databricks DEVBU
data
2
PRD data
Data Consumption Workspaces STGBU
data
2
central and BU 1 (BU 2)
● Published data will be discovered in Metastore 2 by PRD data
published data
the BUs and consumed from the central PRD storage
Platform operations
● Central team provides platform blueprints and Publish data sets / tables,configure access permissions 1 Databricks-to-Databricks Delta Sharing
creates environments for BUs (automated) Consume other data 2 Lakehouse Federation
©2023 Databricks Inc. — All rights reserved Create and consume own databases/tables 3 JDBC / ODBC 32
Access metadata
 Databricks-to-Databricks
1
Delta Sharing
2 Lakehouse Federation

Option 2: Centralized Publishing Process G Central data governance

catalog . schema . table bucket or container folders

3. Set ACLs
location
ACLs ACLs
Central_prd schema Z table C Central container / Central / dataset-C

Unity
BU_1_published
ACLs
schema X
ACLs
table B
/ BU 2 / dataset-A
Metastore 1
ACLs ACLs
/ BU 1 / dataset-B

BU_2_published schema Y table A

4. Share with 1 or 2
Metastore 2 5. Set ACLs
ACLs ACLs G
BU 2 (D2D) schema Y’ table A’

ACLs ACLs G
BU 1 (D2D) schema X’ table B’
Unity
ACLs ACLs 2. Publish
Metastore 2
Central (D2D) schema Z’ table C’ metadata *
1. Publish
BU_2_prd schema Y table A BU 2 prd container / dataset-A
data *
BU_1_prd schema X table B BU 1 prd container / dataset-B

©2023 Databricks Inc. — All rights reserved 33

* For details see push/pull discussion earlier
 U USE CATALOG
C CREATE SCHEMA
_prd: production

Option 2: Metadata Management _stg:

_dev:
staging
development

Permissions on catalog level $

catalog . schema . table/view Owner Central BU 1 BU 2

Central_dev Central U C None None

Central_stg Central U C None None

Unity Central_prd
Central
tables Central U C U * U *
Metastore 1 databases
BU
BU_1_published tables Central U C U * U * region 1
databases
specific
BU ACLs
BU_2_published databases
tables Central U C U * U *

shared shared
BU 2 published databases tables Central U C U * U*
region 2
shared shared * * specific
BU 1 published Central U C U U
databases tables
ACLs
shared shared
Central published databases tables Central U C U * U*

Unity BU tables,
BU_1_prd databases views BU 1 None # U C U **
Metastore 2
BU tables,
BU_1_… databases views BU 1 None U C None

BU tables,
BU_2_prd databases views BU 2 None # U ** U C
BU tables,
BU_2_… databases views BU 2 None None U C

$ to access tables, USE SCHEMA and SELECT also need to be provided on schema and table level 34
* catalog where other BUs can be given access to schemas and tables/views from Central team
©2023 Databricks Inc. — All rights reserved ** catalogs where other BUs can be given access to schemas and tables/views
# if Central needs access to BU data, Delta Sharing or Lakehouse Federation need to be set up
System Account Mappings

Same as in Scenario 4.1

©2023 Databricks Inc. — All rights reserved 35

Pattern 4:
Phased Migration to UC

©2023 Databricks Inc. — All rights reserved 36

Migration approach with 2 security zones

Current situation
Large customers (especially on Azure when they have 10s to 100s of workspaces) won’t do a big bang
migration independent of workload type

Two security zones

● UC Zone
All access is managed by Unity Catalog with fine grained permissions.
● Non-UC Zone
Projects that can’t be migrated due to business or technical reasons can be kept in the well know
Standard cluster world and migrated later

©2023 Databricks Inc. — All rights reserved 37

 High level Two-Zone setup
Approach
Non-UC-Zone UC-Zone
1. Split data into buckets/containers for
Hive Unity Unity ○ Project data: Location for the data of non-migrated projects
Metastore Metastore Catalog ○ Common data (optional): Examples could be public data
(weather, sociodemographic data, …) or company data with
a low level of confidentiality
Databricks Databricks
Workspace Workspace ○ Confidential data: No direct access by Non-UC-Zone
Workspaces and available with fine grained security for
Standard Shared / Single
UC-Zone users
cluster user cluster Common data (opt.)
Ingest Curated Final
(Layer of Truth)
e.g. for DS Scala, R and
Confidential data
single user DS/ML bronze silver gold

2. Create two zones

Access ○ One secured by Unity catalog for all migrated use cases and
<SA 2> <SA 4> (r/o) Control projects (data migrated to “Confidential data” if necessary
and metadata migrated from Hive Metastore to UC)
<SA 3> <SA 1> ○ One for non migrated projects based on “classical”
workspaces and security
Project data
Common Delta Lake Confidential 3. Use different System Accounts for the different data locations
data (opt.) data

©2023 Databricks Inc. — All rights reserved 38

 Two Zone setup: Access to UC secured data
Projects might need data from the the UC-Zone, i.e. “Confidential
Non-UC-Zone UC-Zone
data”. However, due to security reasons, this cannot be provided
Hive Unity Unity directly.
Metastore Metastore Catalog Options (not ordered by priority):
●1 Open Delta Sharing (not Databricks-to-Databricks)
Databricks Databricks ●2 Lakehouse Federation, JDBC or Databricks Python connector
Workspace Workspace against DB SQL in the UC-Zone.

Standard connection Shared / Single Notes:

cluster authn user cluster
2 ● To avoid that confidential data will be played back to the
e.g. for DS Scala, R and UC-Zone via “Common data” (if exists), consider making
single user DS/ML “Common data” read only for Non-UC-Zone users
1
● On Azure customers prefer AAD token. While the connector
works with AAD token, creating a user AAD token is a involved
Access
process and usually no option for end users
<SA 2> <SA 4> (r/o) Control

<SA 3> <SA 1>

Project data
Common Delta Lake Confidential
data (opt.) data

©2023 Databricks Inc. — All rights reserved 39

Pattern 5:
ML Setup

©2023 Databricks Inc. — All rights reserved 40

Using MLR and Unity Catalog
Unity Catalog Metastore Compute Storage

1) DS creates features and training

data (RLS and CLM respected)
DS: read
Shared
data schema A table(s)
compute
catalog Secured by row level security
/ column level masking
(not accessible from Single
user compute)
Data
DS: read/write Scientists
(DS) Delta
feature Lake

DS: read/write
Single user
ml schema B train_data
compute
catalog (MLR)
DS: read/write

model
2) DS creates model from
features and training data
using distributed ML, GPU, …
©2023 Databricks Inc. — All rights reserved 41