Data Processing In
Databricks
Introduction to Advanced Data Processing
in Databricks
Incremental Data Processing
• Efficiently Process New Data
Without Re-Processing Entire
Dataset
• Bullet points:
• Reduce processing time
• Improve data freshness
Auto
loader
• Autoloader in Databricks is a
feature that allows you to
automatically load data from a
variety of sources, such as
cloud storage, messaging
queues, and databases, into
Databricks for processing and
analysis.
• Once you have configured
Autoloader, you can start
loading data into Databricks.
Here's an example of how to
load data from AWS S3 using
Autoloader:
Streaming Data Processing
• Real-Time Data Processing with
Structured Streaming
• Bullet points:
• Real-time data processing
• Event-driven architecture
• Code example:
Optimizing Data
Processing in
Databricks
• Techniques for Efficient Data
Processing
• Bullet points:
• Caching
• Broadcasting
• Predicate pushdown
• Data Compression
• Optimized Data Storage
Databricks workflow jobs (dbt tasks)
• We can use Databricks job to orchestrate our data processing, machine learning, or
data analytics pipelines on the Databricks platform. Databricks Jobs support a number
of workload types, including notebooks, scripts, Delta Live Tables pipelines, Databricks
SQL queries, and dbt projects.
Use notebooks or Python code maintained in a central repository
• A common way to manage version control and collaboration for production artifacts is to use a central
repository such as GitHub. Databricks Jobs supports creating and running jobs using notebooks or Python code
imported from a repository, including GitHub or Databricks Git folders
Orchestrate your jobs with Apache Airflow
• Databricks recommends using Databricks Jobs to orchestrate your workflows. However, Apache Airflow is
commonly used as a workflow orchestration system and provides native support for Databricks Jobs.
Use dbt transformations in a job
• Use the dbt task type if you are doing data transformation with a dbt core project and want to integrate that
project into a Databricks job, or you want to create new dbt transformations and run those transformations in a
job
Use Code Package in JAR
• We can use Databricks job to orchestrate our data processing, machine learning, or data analytics pipelines on
the Databricks platform. Databricks Jobs support a number of workload types, including notebooks, scripts,
Delta Live Tables pipelines, Databricks SQL queries, and dbt projects.
Use Python Package in job
• We can use Databricks job to orchestrate our data processing, machine learning, or data analytics pipelines on
the Databricks platform. Databricks Jobs support a number of workload types, including notebooks, scripts,
Delta Live Tables pipelines, Databricks SQL queries, and dbt projects.
Python wheel file
Advanced Data Processing Techniques
• Handling Data Skew and Partitioning for Efficient Data Processing
• Bullet points:
• Data skew handling
• Data partitioning
• Cashing
• Data locality
• Optimize joins.(Broadcast hash join)