Scribd
Upload
6 views1 page

Assignment Questions

The document outlines two assignments focused on Apache Spark and PySpark, covering topics such as Spark architecture, SparkSession, CSV file handling, lazy evaluation, and big data problems. It includes practical coding tasks for data manipulation, aggregation, and joining data frames. Additionally, it addresses key functions like fillna(), dropna(), rank(), dense_rank(), and the differences between union and unionByName.

Uploaded by

shivam9762247088
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
6 views1 page

Assignment Questions

The document outlines two assignments focused on Apache Spark and PySpark, covering topics such as Spark architecture, SparkSession, CSV file handling, lazy evaluation, and big data problems. It includes practical coding tasks for data manipulation, aggregation, and joining data frames. Additionally, it addresses key functions like fillna(), dropna(), rank(), dense_rank(), and the differences between union and unionByName.

Uploaded by

shivam9762247088
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

Assignment - 1

1. What is Spark and its benefits?


2. Explain Apache Spark architecture with aa neat diagram.
3. What is the importance of SparkSession? Create a spark session.
4. How to read a csv file in spark. Explain the importance of
inferSchema and header.
5. Explain Lazy Evaluation in Spark.
6. What is big data problem, How Spark resolves it?
7. Write a PySpark code to
a) Select only name, department, and marks.
b) Add a new column ‘bonus_marks’ and add 5 marks to every
student.
c) Rename marks to student_marks using withColumnRenamed()
8. Write a Spark code to create a new column named “college” and
give the value “ARMIET” to everyone.

Assignment - 2
1. Write a PySpark code to groupBy department and perform these
aggregations
a. Calculate Count of students (column name should be
std_count)
b. Calculate Sum of Marks (column name should be total_marks)
c. Calculate Average of Marks (column name should be
avg_marks)
2. Distinguish between fillna() and dropna() with example
3. What is the difference between rank() and dense_rank() with
example
4. Explain sorting and write spark code to sort the students by marks.
5. What is inner join, Perform inner join between two data frames.
6. What is left outer join, right outer join and full outer join with
examples.
7. What is union? Explain with example code.
8. What is the difference between union and unionByName.

You might also like