Databricks ML — Study Notes

Module 3: Unity Catalog

1. What is Governance and Why it Matters

Governance means having rules and controls over your data:
→ Who can see this data?
→ Who can modify this data?
→ Where did this data come from?
→ Who accessed this data and when?
→ Is this data accurate and consistent?

Four pillars of governance:

Access Control — Who can do what

→ Row level: customer can only see their own data
→ Column level: junior analyst can't see salary column
→ Table level: only HR team can access employee tables
→ Catalog level: only admins can create new catalogs

Audit — Who did what and when

→ Unity Catalog records every action automatically
→ Required for compliance: GDPR, HIPAA, SOC2
2026-07-15 09:23 → rahul@[Link] → SELECT * FROM customers
2026-07-15 09:45 → john@[Link] → UPDATE transactions SET...

Lineage — Where did data come from

→ Tracks entire chain from source to model predictions
→ Automatic — no code needed
→ Impact analysis: what breaks if I change this table?

Data Quality — Is data accurate

→ Schema enforcement → wrong data types rejected
→ NOT NULL constraints → missing values rejected
→ Consistent definitions across teams

2. What is Unity Catalog

Unity Catalog is Databricks' centralized governance system — one place to govern all data assets
across all workspaces and clusters.

Before Unity Catalog:

→ Each workspace had its own Hive metastore
 → Tables not visible across workspaces
→ No central access control
→ No audit trail
→ No data lineage
→ PII data exposed — anyone could read anything

With Unity Catalog:

→ One governance layer across ALL workspaces
→ Central access control via GRANT/REVOKE
→ Automatic audit logging
→ Automatic lineage tracking
→ Governs tables, views, volumes, models, functions

3. The 3-Level Namespace

Everything in Unity Catalog lives in a 3-level hierarchy:
CATALOG
└── SCHEMA (also called database)
└── TABLE / VIEW / VOLUME / FUNCTION / MODEL

Written as:
[Link]

example: [Link]
↑ ↑ ↑
catalog schema table

What lives at each level

Catalog — top level container
→ One per business domain or environment
→ Examples: finance_prod, marketing_dev, ml_learning
→ Special built-in catalogs: system, samples, hive_metastore

Schema — middle level container

→ Groups related tables together
→ Like a database inside a catalog
→ Can hold unlimited tables, views, volumes, functions

Table/View/Volume — bottom level objects

→ TABLE → structured data (rows and columns)
→ VIEW → virtual table (saved SQL query, no data stored)
→ VOLUME → unstructured files (images, PDFs, CSVs, models)
→ MODEL → MLflow registered models
 → FUNCTION → reusable SQL/Python functions

Special built-in catalogs

Catalog Purpose Access
system Databricks internal metadata, Read only
billing, audit logs
samples Sample datasets (nyctaxi, tpch Read only
etc.)
hive_metastore Legacy catalog before Unity No UC governance
Catalog
workspace Default workspace catalog Full access

USE CATALOG and USE SCHEMA

-- Set default catalog
USE CATALOG ml_learning;

-- Set default schema

USE SCHEMA spark_models;

-- Now query without full path

SELECT * FROM wine_results;
-- same as: SELECT * FROM ml_learning.spark_models.wine_results

4. Access Control — GRANT and REVOKE

The 3-Door Rule
This is the most important exam concept in Unity Catalog:

⚠ EXAM TRAP: Even if user has SELECT on a table, they ALSO need USE CATALOG and
USE SCHEMA. Missing either one → access denied.

Complete access pattern — always follow this:

-- Door 1: catalog access (ALWAYS required)
GRANT USE CATALOG ON CATALOG learning_catalog
TO `user@[Link]`;

-- Door 2: schema access (ALWAYS required)

GRANT USE SCHEMA ON SCHEMA learning_catalog.raw_data
TO `user@[Link]`;

-- Door 3: object access (specific privilege)

GRANT SELECT ON TABLE learning_catalog.raw_data.employees
TO `user@[Link]`;
This rule applies for EVERY operation:
→ READ data → USE CATALOG + USE SCHEMA + SELECT
→ WRITE data → USE CATALOG + USE SCHEMA + MODIFY
→ CREATE table → USE CATALOG + USE SCHEMA + CREATE TABLE
→ READ files → USE CATALOG + USE SCHEMA + READ VOLUME
→ WRITE files → USE CATALOG + USE SCHEMA + WRITE VOLUME

All Privilege Types

Privilege What it allows Applied on
SELECT Read data Table, View
MODIFY Insert, Update, Delete Table
CREATE TABLE Create tables Schema
CREATE SCHEMA Create schemas Catalog
CREATE VOLUME Create volumes Schema
CREATE MODEL Register ML models Schema
USE CATALOG Navigate into catalog Catalog
USE SCHEMA Navigate into schema Schema
READ VOLUME Read files from volume Volume
WRITE VOLUME Write files to volume Volume
EXECUTE Run a function Function
ALL PRIVILEGES Everything Any level

Privilege Inheritance — how access flows down

→ Grant at catalog level → access to everything inside
→ Grant at schema level → access to everything in that schema only
→ Grant at table level → access to that table only
-- Catalog level — broadest
GRANT SELECT ON CATALOG learning_catalog TO `user@[Link]`;
→ can SELECT all tables in all schemas

-- Schema level — medium

GRANT SELECT ON SCHEMA learning_catalog.raw_data TO `user@[Link]`;
→ can SELECT all tables in raw_data only

-- Table level — most specific

GRANT SELECT ON TABLE learning_catalog.raw_data.employees TO `user@[Link]`;
→ can SELECT employees table only

Owner vs Granted Privileges

Owner Granted User
How obtained Created the object Explicit GRANT
Privileges ALL automatically Only what was granted
SHOW GRANTS output Empty (implicit) Shows explicitly
Can GRANT to others Yes Only if WITH GRANT OPTION
Principle of Least Privilege
Give users ONLY what they need — nothing more.
-- Wrong — too broad
GRANT ALL PRIVILEGES ON CATALOG finance TO `analyst@[Link]`
→ can delete tables, drop schemas, modify salary data

-- Correct — least privilege

GRANT USE CATALOG ON CATALOG finance TO `analyst@[Link]`
GRANT USE SCHEMA ON SCHEMA [Link] TO `analyst@[Link]`
GRANT SELECT ON TABLE [Link].monthly_summary TO `analyst@[Link]`
→ can only read one specific table

REVOKE — Remove access

REVOKE SELECT ON TABLE learning_catalog.raw_data.employees
FROM `user@[Link]`;

5. Managed vs External Tables

The core difference — who owns the data:
Managed → Databricks owns BOTH metadata AND data
External → Databricks owns metadata ONLY, YOU own data

Managed Table
CREATE TABLE learning_catalog.raw_data.employees (
emp_id INT,
name STRING,
salary DOUBLE
) USING DELTA;

→ Databricks decides WHERE to store data

→ No LOCATION clause needed
→ DROP TABLE → metadata deleted AND data files deleted PERMANENTLY
→ No recovery possible after DROP
→ is_managed_location = true

External Table
CREATE TABLE learning_catalog.raw_data.employees_ext
USING DELTA
LOCATION 's3://your-bucket/data/employees';

→ YOU decide where data is stored

→ LOCATION clause required
 → DROP TABLE → only metadata deleted, data STAYS in S3
→ Can recreate table pointing to same location
→ is_managed_location = false

Managed vs External — comparison

Managed External
Data location Databricks managed storage Your S3/ADLS/GCS path
Who controls files Databricks You
LOCATION clause Not needed Required
DROP TABLE effect Deletes everything permanently Only metadata deleted, data stays
is_managed_location true false
Use case New data, default choice Legacy data, shared data, external
systems

When to use which

Use Managed when:
→ Creating new tables from scratch
→ Data belongs entirely to Databricks
→ Simple setup, no external storage to manage
→ ML feature tables, processed tables, inference logs

Use External when:

→ Data already exists in S3/ADLS/GCS
→ Data shared with non-Databricks systems
→ Data must survive even if table is dropped
→ Multiple teams/tools access same files

6. Volumes
Volumes store UNSTRUCTURED data — files that don't fit in tables:
→ Raw CSV files before loading into tables
→ ML model files (.pkl, .pt, .h5)
→ Images for computer vision
→ PDFs, audio files, JSON configs

Volume path structure

/Volumes/catalog/schema/volume_name/[Link]

example:
/Volumes/ml_learning/spark_models/mlflow_tmp/[Link]
↑ ↑ ↑ ↑
catalog schema volume file
Two types of Volumes
Managed Volume:
CREATE VOLUME ml_learning.spark_models.my_volume;
→ Databricks manages storage location
→ DROP VOLUME → files deleted permanently

External Volume:
CREATE EXTERNAL VOLUME ml_learning.spark_models.my_ext_volume
LOCATION 's3://your-bucket/path';
→ You control storage location
→ DROP VOLUME → only metadata deleted, files stay in S3

Volume Privileges
GRANT READ VOLUME ON VOLUME learning_catalog.raw_data.my_files
TO `user@[Link]`;

GRANT WRITE VOLUME ON VOLUME learning_catalog.raw_data.my_files

TO `user@[Link]`;

⚠ EXAM TRAP: Volume access is all-or-nothing at volume level. File-level access control is
NOT supported. To restrict access to specific files, put them in separate volumes.

Working with Volume files

# Write file to volume
[Link]('/Volumes/catalog/schema/volume/[Link]', data)

# Read file from volume into Spark

df = [Link]('csv')\
.option('header', 'true')\
.load('/Volumes/catalog/schema/volume/[Link]')

# List files in volume

[Link]('/Volumes/catalog/schema/volume/')

7. Data Lineage
Lineage tracks the entire journey of data:
→ Where did this data come from? (Upstream)
→ What tables were used to create this table?
→ Which notebooks/jobs processed this data?
→ Which downstream tables depend on this table?
Upstream vs Downstream
raw_sales (source)
↓
monthly_summary
↓
top_products (end)

From monthly_summary perspective:

→ Upstream → raw_sales (where data came FROM)
→ Downstream → top_products (where data GOES TO)

How lineage is tracked

→ Completely automatic — no code needed
→ Unity Catalog monitors every query
→ Records: tables read, tables written, notebook, user, timestamp
→ Visible in Catalog UI → table → Lineage tab
→ Shows both table lineage AND notebook lineage

Why lineage matters

Data quality issue:
→ Model predictions wrong → trace lineage → find bug in raw data
→ Fix raw data → know exactly which downstream tables to rebuild

Compliance (GDPR):
→ Delete customer data → find ALL tables containing that data via lineage
→ Without lineage → might miss tables → compliance violation

Impact analysis:
→ Want to change schema → check downstream lineage first
→ See which tables, notebooks, jobs will break
→ Fix those first, then make the change

8. information_schema
Every catalog has an information_schema — a special schema containing metadata about all objects
in the catalog.
-- List all tables in a catalog
SELECT table_name, table_type, comment
FROM learning_catalog.information_schema.tables
 WHERE table_schema = 'raw_data';

-- List all schemas in a catalog

SELECT schema_name
FROM learning_catalog.information_schema.schemata;

-- List all columns in a table

SELECT column_name, data_type
FROM learning_catalog.information_schema.columns
WHERE table_name = 'employees';

9. Exam Questions & Answers

Q: What is Unity Catalog?
A: Databricks' centralized governance system — one place to govern all data assets (tables, views,
volumes, models, functions) across all workspaces with access control, audit logging, and lineage
tracking.
Q: What is the 3-level namespace in Unity Catalog?
A: [Link] — Catalog is the top level, Schema is the middle level (like a database),
and Table/View/Volume/Function/Model are at the bottom level.
Q: A user has SELECT on a table but gets an error when querying it. What might be wrong?
A: Missing USE CATALOG or USE SCHEMA privilege. All three are required: USE CATALOG
on catalog + USE SCHEMA on schema + SELECT on table.
Q: What is the minimum privilege needed to read a table?
A: USE CATALOG on the catalog + USE SCHEMA on the schema + SELECT on the table. All
three required — not just SELECT.
Q: What privilege allows inserting and deleting rows?
A: MODIFY privilege on the table.
Q: What is the difference between a managed and external table?
A: Managed: Databricks owns both metadata and data. DROP TABLE deletes everything
permanently. External: Databricks owns metadata only, you own data in S3/ADLS. DROP TABLE
only removes metadata, data stays.
Q: A data engineer accidentally drops a managed table. What happens to the data?
A: Data is permanently deleted — both metadata and data files are gone. No recovery possible.
Q: A data engineer drops an external table. What happens to the data?
A: Data is safe — only metadata is removed from Unity Catalog. Data files remain in
S3/ADLS/GCS. Table can be recreated pointing to same location.
Q: What are Volumes used for?
A: Storing unstructured files (images, PDFs, CSVs, ML model files) in a Unity Catalog governed
path under /Volumes/catalog/schema/volume_name/.
Q: Can you grant access to a specific file inside a Volume?
A: No — Volume access is all-or-nothing at the volume level. READ VOLUME or WRITE
VOLUME grants access to ALL files. To restrict access to specific files, put them in separate
volumes.
Q: What privilege is needed to read files from a Volume?
 A: USE CATALOG + USE SCHEMA + READ VOLUME on the volume. Same 3-door rule
applies.
Q: What is data lineage?
A: Automatic tracking of where data came from (upstream) and where it goes (downstream). Unity
Catalog tracks this automatically — no code needed. Visible in Catalog UI under Lineage tab.
Q: What is upstream vs downstream in lineage?
A: Upstream = where data came FROM (source tables, earlier in the pipeline). Downstream =
where data GOES TO (derived tables, later in the pipeline).
Q: What is the principle of least privilege?
A: Give users only the minimum privileges they need — nothing more. Prevents accidental or
malicious data access, modification, or deletion.
Q: What does SHOW GRANTS return for the table owner?
A: Empty result — owners have all privileges implicitly without explicit GRANT. Only explicitly
granted privileges appear in SHOW GRANTS.
Q: What special built-in catalog contains audit logs and billing data?
A: The system catalog — specifically [Link] for audit logs and [Link]
for billing data. Read only.
Q: What is information_schema?
A: A special schema automatically created in every catalog. Contains metadata about all objects —
tables, schemas, columns, privileges. Used to query catalog metadata with SQL.
Q: What is hive_metastore?
A: The legacy catalog from before Unity Catalog. Tables created before Unity Catalog live here.
No Unity Catalog governance features like lineage or fine-grained access control.

10. Quick Reference — All Commands

Command Purpose
SHOW CATALOGS List all catalogs
SHOW SCHEMAS IN catalog List schemas in a catalog
SHOW TABLES IN [Link] List tables in a schema
SHOW VOLUMES IN [Link] List volumes in a schema
SHOW GRANTS ON TABLE ... See privileges on a table
USE CATALOG catalog_name Set default catalog
USE SCHEMA schema_name Set default schema
CREATE CATALOG IF NOT EXISTS name Create new catalog
CREATE SCHEMA IF NOT EXISTS Create new schema
[Link]
CREATE TABLE ... USING DELTA Create managed Delta table
CREATE TABLE ... LOCATION 's3://...' Create external table
CREATE VOLUME [Link] Create managed volume
GRANT SELECT ON TABLE ... TO `user` Give read access
GRANT MODIFY ON TABLE ... TO `user` Give write access
GRANT ALL PRIVILEGES ON CATALOG ... TO Give all access
`user`
REVOKE SELECT ON TABLE ... FROM `user` Remove read access
DESCRIBE TABLE EXTENDED See full table details
[Link]
SELECT * FROM Query catalog metadata
catalog.information_schema.tables
11. The 3-Door Rule — Always Remember
USE CATALOG + USE SCHEMA + Object Privilege = Access Granted
Missing ANY ONE of these three = Access Denied

Databricks ML Associate Exam Prep — Module 3: Unity Catalog