PYSPARK
PRESENTED BY
Naresh Kumar
• Databricks Workspace Walkthrough
• Magic Commands in Databricks Overview
• Databricks DBUtils Commands
• Different Ways of Creating RDDs
• What is Lineage & Directed Acyclic Graph (DAG) in Spark
• Creating Data Frames With Automatic and Manual Schemas
• Data Frame Read and Write Operations with different file formats (CSV, JSON, Parquet)
AGENDA
• Spark SQL Operations
• Creating SQL Databases and Tables
• Creating a SQL table out of a DataFrame
WEEK-4
• Writing SQL query result into a Directory
• Analyzing the files data without creating the DataFrame
• Working on temporary views and global views
• Creating a DataFrame from a SQL table
• Use Case: Query to Find the Highest Salary in Each Department
• Managed/Internal Vs Unmanaged/External Tables
• Delta Tables and its Key Features
• What is Delta Lake
• What is Data Warehouse
• What is Data Lake
• What is Lakehouse
• Data Warehouse Vs Data Lake Vs Lakehouse
• Databricks Workspace Walkthrough
• Magic Commands in Databricks Overview
Explained in separate databricks notebooks
• Databricks DBUtils Commands
• Different Ways of Creating RDDs
• What is Lineage & Directed Acyclic Graph (DAG) in Spark
• Creating Data Frames With Automatic and Manual Schemas
• Data Frame Read and Write Operations with different file formats (CSV, JSON, Parquet)
• Spark SQL Operations
AGENDA • Creating SQL Databases and Tables
• Creating a SQL table out of a DataFrame
• Writing SQL query result into a Directory
WEEK-4 • Analyzing the files data without creating the DataFrame
• Working on temporary views and global views
• Creating a DataFrame from a SQL table
• Use Case: Query to Find the Highest Salary in Each Department
• Managed/Internal Vs Unmanaged/External Tables
• Delta Tables and its Key Features
• What is Delta Lake
• What is Data Warehouse
Will be covered in ‘WEEK-5’
• What is Data Lake
• What is Lakehouse
• Data Warehouse Vs Data Lake Vs Lakehouse
Thank You