Case Study: Extract from AWS S3 & MySQL, Merge, and Store into
Databricks Volume (Databricks Free Edition)
Overview
This case study demonstrates a step-by-step solution (with runnable Python code) for the
Databricks New Free Edition environment. The pipeline:
1) Extract data from an AWS S3 bucket and from a MySQL database using Python in a
Databricks notebook.
2) Merge/join the datasets using Spark (PySpark) APIs.
3) Write the merged output into a Databricks Unity Catalog Volume.
Objectives
- Show practical code cells that can be pasted into a Databricks notebook (Python).
- Use secure best-practices (Databricks secrets) for credentials where appropriate.
- Provide alternatives (S3 access-keys vs IAM roles) and notes on JDBC driver setup for
MySQL.
Prerequisites
1. Databricks Free Edition workspace and a running cluster attached to your notebook. Free
Edition may have some feature limitations; if Unity Catalog or volumes are not available in
your workspace, follow the 'Volumes' section in Databricks docs or use DBFS paths as a
fallback.
2. AWS S3 bucket with object(s) (CSV/Parquet) you want to read. Access to the S3 bucket
via either an IAM role/instance profile or access keys.
3. MySQL database (host, port, dbname) reachable from the Databricks workspace or via a
network gateway/VPN. Your cluster must have network access.
4. MySQL JDBC driver attached to the cluster (Maven coordinate example:
[Link]:mysql-connector-j:9.4.0). You can also upload the connector jar to the cluster.
Best practices / Security notes
- DO NOT hard-code credentials in notebooks. Use Databricks Secret Scopes or cluster-
scoped init scripts.
- Prefer IAM instance profiles or Unity Catalog external locations for S3 access over
embedding AWS keys.
- Ensure the MySQL JDBC driver is available on the cluster (add as a library).
Cluster library: MySQL JDBC driver (brief)
Install the MySQL JDBC connector on the cluster. Example Maven coordinate to add as a
library: [Link]:mysql-connector-j:9.4.0. Alternatively, upload the mysql-connector-java
jar to the cluster and restart.
Troubleshooting & Tips
- If you get ClassNotFoundException for the JDBC driver, the connector jar is not attached to
the cluster.
- Network/timeouts: ensure security groups / VPC peering or public access for MySQL is
configured.
- Use Secret Scopes for credentials and avoid printing secrets in notebooks.
- For large S3 reads, prefer Spark native reads (s3a) over boto3 to leverage distributed read.
Conclusion
This case study provides runnable notebook cells and clear instructions to extract data from
S3 and MySQL, merge using PySpark, and store outputs into Databricks Unity Catalog
Volumes. Adjust paths, credentials, and resource names for your environment.
Appendix: Quick checklist before you run
1. Cluster has network access to MySQL host.
2. MySQL JDBC driver is installed on the cluster (Maven or uploaded jar).
3. If using S3 access keys, use Databricks Secret Scopes and never store keys inline.
4. You have the required Unity Catalog permissions to create or write to volumes.
5. Replace placeholder names throughout the notebook cells.