Scribd
Upload
4 views5 pages

Databricks Spark Performance Tuning Guide

The document discusses the architecture of Databricks/Apache Spark and identifies performance bottlenecks that can affect efficiency. It highlights that tuning options differ based on the platform used, such as Delta Lake versus non-Delta Lake. The content is presented in a lesson format by Bryan Cafferky, focusing on key constraints and data distribution strategies.

Uploaded by

LearnMSBI First
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
4 views5 pages

Databricks Spark Performance Tuning Guide

The document discusses the architecture of Databricks/Apache Spark and identifies performance bottlenecks that can affect efficiency. It highlights that tuning options differ based on the platform used, such as Delta Lake versus non-Delta Lake. The content is presented in a lesson format by Bryan Cafferky, focusing on key constraints and data distribution strategies.

Uploaded by

LearnMSBI First
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Understanding Databricks/Spark

Performance Tuning
Lesson 01: The Spark Architecture & Bottlenecks
by Bryan Cafferky from my YouTube channel
Where We're Going?
• Databricks/Apache Spark Architecture
• Performance Bottlenecks
• Tuning Options Vary By Platform
• Wrap Up
Spark Cluster
External Storage
Worker Nodes

Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core
Disk Disk Disk Disk Disk Disk Disk Disk Disk Disk Disk Disk Disk Disk Disk Disk Disk Disk
Do Count Do Count Do Count Do Count Do Count Do Count Do Count Do Count Do Count

Send Back Result Partition Data by City


and Copy One City to Each Node Executor

Constraints Distributing the Data Over the Cluster


Driver Node
Core Core
➢ Hardware/Resources Disk Disk
➢ Software (Spark/DBR)
➢ Environment Configurations SELECT City, Count(*) FROM PhoneBook
Group By City
➢ Your Code/Application Order By City
➢ Data Source & Format
➢ Data Distribution

Phone Book
Options Vary By Platform
Delta Lake vs. Non Delta Lake

Open-Source Spark

Databricks
Wrapping Up
• Databricks/Apache Spark Architecture
• Performance
ThankBottleneck
You!
• Tuning Options Vary By Platform

You might also like