Scribd
Upload
1 views2 pages

PDF2 Read Parquet Using PySpark

This document is a practical guide on reading Parquet data using PySpark, detailing how to start Spark, read Parquet files and directories, select columns, filter data, and utilize partitioned Parquet. It emphasizes the benefits of Parquet's columnar storage format, including efficient reads and schema information. Key points include the use of Spark's read method, the advantages of column selection, and performance improvements through partition pruning and predicate pushdown.

Uploaded by

demo18887
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
1 views2 pages

PDF2 Read Parquet Using PySpark

This document is a practical guide on reading Parquet data using PySpark, detailing how to start Spark, read Parquet files and directories, select columns, filter data, and utilize partitioned Parquet. It emphasizes the benefits of Parquet's columnar storage format, including efficient reads and schema information. Key points include the use of Spark's read method, the advantages of column selection, and performance improvements through partition pruning and predicate pushdown.

Uploaded by

demo18887
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Read Parquet Data Using PySpark

Practical guide: reading Parquet, schema inspection, partitioned data, filtering, and column selection.

1. Objective
Apache PySpark is the Python API for Apache Spark. Spark DataFrames provide a distributed, schema-aware way to
process structured data. The examples in this guide use SparkSession, which is the entry point for DataFrame
operations.

Parquet is a columnar storage format commonly used in data engineering because it supports efficient column reads,
compression, and schema information.

2. Start Spark
from [Link] import SparkSession

spark = [Link] \
.appName("Read Parquet Data") \
.getOrCreate()

3. Read a Parquet file


df = [Link]("data/[Link]")

[Link]()
[Link]()

Spark reads the schema stored with the Parquet data, so explicit schema inference is normally unnecessary.

4. Read a Parquet directory


df = [Link]("data/parquet/")
[Link](truncate=False)

A directory can contain many Parquet part files. Spark treats the directory as a logical dataset.

5. Read selected columns


df = [Link]("data/[Link]")

[Link]("id", "name", "country").show()

Because Parquet is columnar, selecting only required columns can reduce the amount of data Spark needs to read.

6. Filter Parquet data


df = [Link]("data/[Link]")

[Link](df["age"] > 30).show()


[Link](df["country"] == "India").show()

Filters can be pushed down toward the data source in suitable cases, improving read efficiency.

7. Partitioned Parquet
df = [Link]("data/customer/country=India/")
[Link]()
A common data-lake layout partitions data by business columns such as year, month, day, or country. Reading only
relevant partitions can reduce work.

8. Useful options
df = (
[Link]
.option("mergeSchema", "true")
.parquet("data/parquet/")
)

mergeSchema can be useful when compatible Parquet files have evolved schemas. It can add overhead, so it should
not be enabled unnecessarily.

9. Complete example
from [Link] import SparkSession

spark = [Link]("ParquetExample").getOrCreate()

df = [Link]("data/[Link]")

[Link]()
[Link](10, truncate=False)

[Link]("id", "name", "country").show()


[Link](df["age"] >= 18).show()

10. Key points


Use [Link](path).

Parquet stores schema information with the data.

Column selection can benefit from Parquet columnar storage.

Partition pruning and predicate pushdown can improve performance.

You might also like