Scribd
Upload
6 views16 pages

DP 600 Python

The document discusses techniques for optimizing data performance in Spark, including partitioning data files and using Delta Lake for efficient data storage. It highlights the benefits of V-Order for faster reads and the Optimize Write feature for improved reliability. Additionally, it mentions the use of temporary views and SQL magic in notebooks for querying data in the Spark catalog.

Uploaded by

getusama
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
6 views16 pages

DP 600 Python

The document discusses techniques for optimizing data performance in Spark, including partitioning data files and using Delta Lake for efficient data storage. It highlights the benefits of V-Order for faster reads and the Optimize Write feature for improved reliability. Additionally, it mentions the use of temporary views and SQL magic in notebooks for querying data in the Spark catalog.

Uploaded by

getusama
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Better compression and performance

Bronze, Silver, Gold


Underlying Files can be partitioned using Data Pilelines in Lakehouse Table

Another example:
One of the simplest ways to make data in a dataframe available for querying in the Spark catalog is to create a temporary view,
as shown in the following code example:

Microsoft Learn Spark Exercise Link


In a notebook, you can also use the %%sql magic to run SQL code that queries objects in the catalog, like
this:
Add a new cell with the following code; which saves the dataframe, partitioning the data by Year and
Month:

Partitioning data files is a common way to optimize performance when dealing with large volumes of data.
This technique can significant improve performance and make it easier to filter data.
Delta Lake is an open-source storage layer that adds relational database semantics to Spark-based data
lake processing. Tables in Microsoft Fabric lakehouses are Delta tables, which is signified by the triangular
Delta (▴) icon on tables in the lakehouse user interface.

Delta tables are schema abstractions over data files that are stored in Delta format. For each table, the
lakehouse stores a folder containing Parquet data files and a _delta_Log folder in which transaction details
are logged in JSON format.
V-Order enables faster and more efficient reads by various compute engines, such as Power BI, SQL, and
Spark. V-order applies special sorting, distribution, encoding, and compression on parquet files at
write-time.

Optimize write improves the performance and reliability by reducing the number of files written and
increasing their size. It's useful for scenarios where the Delta tables have suboptimal or nonstandard file
sizes, or where the extra write latency is tolerable.
Explicit Schema Read with Wildcard

The line at the beginning used to specify the language is called “magic”:
%%pyspark - ​ Python
%%spark - ​ Scala​
%%sql - ​ SQL

You might also like