Scribd
Upload
1K views1 page

Read and Write JSON in PySpark

This Python code uses Spark SQL to read employee data from a JSON file into a DataFrame. It then filters the DataFrame to only rows where the stream is "JAVA" and writes the filtered DataFrame to a new Parquet file. It first reads the JSON, displays the DataFrame, coalesces and writes it to a Parquet file. Then it reads the Parquet, filters for "JAVA" stream, displays and writes the filtered DataFrame to a new Parquet file.

Uploaded by

Arpita Das
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
1K views1 page

Read and Write JSON in PySpark

This Python code uses Spark SQL to read employee data from a JSON file into a DataFrame. It then filters the DataFrame to only rows where the stream is "JAVA" and writes the filtered DataFrame to a new Parquet file. It first reads the JSON, displays the DataFrame, coalesces and writes it to a Parquet file. Then it reads the Parquet, filters for "JAVA" stream, displays and writes the filtered DataFrame to a new Parquet file.

Uploaded by

Arpita Das
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd

#Put your code here

from [Link] import SparkSession


spark = SparkSession \
.builder \
.appName("Data Frame EMPLOYEE") \
.config("[Link]", "some-value") \
.getOrCreate()
df = [Link]("[Link]")
[Link]()
[Link](1).[Link]("Employees")
pf = [Link]("Employees")
dfNew = [Link]([Link]=='JAVA')
[Link]()
[Link](1).[Link]("JavaEmployees")

You might also like