Scribd
Upload
177 views1 page

Pyspark DataFrame Operations Guide

This Python code creates a SparkSession and defines a Row schema to represent personal data with fields for ID, Name, Age, and Area of Interest. It then creates sample Row objects and collects them in a list to create a DataFrame. The code calculates descriptive statistics on the Age column, writes it to Parquet format, sorts the DataFrame by Name in descending order and displays the results, and writes it to Parquet format.

Uploaded by

Arpita Das
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
177 views1 page

Pyspark DataFrame Operations Guide

This Python code creates a SparkSession and defines a Row schema to represent personal data with fields for ID, Name, Age, and Area of Interest. It then creates sample Row objects and collects them in a list to create a DataFrame. The code calculates descriptive statistics on the Age column, writes it to Parquet format, sorts the DataFrame by Name in descending order and displays the results, and writes it to Parquet format.

Uploaded by

Arpita Das
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd

# Put your code here

from [Link] import SparkSession


spark = SparkSession \
.builder \
.appName("Data Frame Personal") \
.config("[Link]", "some-value") \
.getOrCreate()
from [Link] import *
Personal = Row("ID","Name","Age","Area of Interest")
data1 = Personal("1","Jack", "22", "Data Science")
data2 = Personal("2","Luke", "21", "Data Analytics")
data3 = Personal("3","Leo", "24", "Micro Services")
data4 = Personal("4","Mark", "21", "Data Analytics")

PersonalData=[data1,data2,data3,data4]
df = [Link](PersonalData)
Result1 = [Link]('Age')
[Link](1).[Link]("Age")
Result2 = [Link]('ID','Name','Age').orderBy('Name',ascending=False)
[Link]()
[Link](1).[Link]("NameSorted")
#[Link]()

You might also like