Scribd
Upload
3 views10 pages

Spark Data Abstractions Explained

The document provides an overview of key concepts in Big Data and Spark, including the differences between DataFrame, Dataset, and RDD, as well as their usage examples. It also explains the concept of sockets in network communication, the purpose of ADHOC files in Spark, and the structure of input data for linear regression using RDDs. Additionally, it outlines the levels of data abstraction in Spark, highlighting RDD, DataFrame, and Dataset.

Uploaded by

Mbarek Jeweher
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
3 views10 pages

Spark Data Abstractions Explained

The document provides an overview of key concepts in Big Data and Spark, including the differences between DataFrame, Dataset, and RDD, as well as their usage examples. It also explains the concept of sockets in network communication, the purpose of ADHOC files in Spark, and the structure of input data for linear regression using RDDs. Additionally, it outlines the levels of data abstraction in Spark, highlighting RDD, DataFrame, and Dataset.

Uploaded by

Mbarek Jeweher
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

F.

Big Data

Questions & Answers

Compte Rendu

Master 2 : Data sc Etudiante :


2025/2026 Mbarek Jeweher

Enseignante : Dr . Rania Mkhinini


[Link] Data

Q1: What is the difference between


DataFrame, Dataset, and RDD in Spark?
When and how do we use them?

When to use each?


[Link] Data

How to use in Spark?


RDD Example (Python)
First open terminal and PySpark Shell :

Test RDD Example

rdd = [Link]([1, 2, 3, 4])


rdd_squared = [Link](lambda x: x * x)
rdd_squared.collect()
[Link] Data

How to use in Spark?


Test DataFrame Example
No CSV needed — we will create a simple DataFrame:

df = [Link]([(1, "Alice"),
(2, "Bob")], ["id", "name"])
[Link]()

Test Dataset example ( with scala )


open spark shell

case class Person(name: String, age: Int)


val ds = Seq(Person("Alex", 30)).toDS()
[Link]()
[Link] Data

Q2: What is a socket and when is it used?

A socket is a communication endpoint between two programs


over a network.
It allows sending and receiving data between:
Two different machines (via Internet or LAN)
Or two processes on the same machine
A socket usually works with TCP or UDP protocols.

Use cases:
Client-server applications (web browser → web server)
Instant messaging apps ( WhatsApp, Telegram)
Online games and streaming
Distributed services and clusters (like Spark, Hadoop)

case class Person(name: String, age: Int)


val ds = Seq(Person("Alex", 30)).toDS()
[Link]()
[Link] Data

Q3 : ADHOC File in spark : Actual Contents

An ADHOC file in Spark or a Big Data cluster contains


instructions and/or data for a one-time, interactive query.
Specifically

case class Person(name: String, age: Int)


val ds = Seq(Person("Alex", 30)).toDS()
[Link]()
[Link] Data

Q4 : Given an RDD x used to perform linear regression in


Spark, what should be the structure of the input data
stored in this RDD? Explain what each part of the input
represents.

Step 1 — Import and create the RDD

from [Link] import Vectors

data = [
(3.0, [Link]([1.0])),
(7.0, [Link]([2.0])),
(11.0, [Link]([3.0]))
]

rdd = [Link](data)
[Link]()

case class Person(name: String, age: Int)


val ds = Seq(Person("Alex", 30)).toDS()
[Link]()
[Link] Data

Step 2 — Convert to DataFrame (required for ML)

df = [Link](["label", "features"])
[Link]()

Étape 3 — Entraîner le modèle de régression linéaire

Step 4 — Display the model results


[Link] Data

The model found is: R² = 1.0


Y = 4X − 1 The model perfectly explains the data

Meaning:
For each increase of 1 unit in X, Y increases by 4
When X = 0, the predicted value of Y is −1

Step 5 — Make a prediction

>>> predictions = [Link](df)


>>> [Link]()
[Link] Data

Q5 : Spark provides multiple levels of data abstraction:


RDD (Level 1): Distributed data collection with transformations and
actions.
DataFrame (Level 2): Tabular data with schema and query
optimization.
Dataset (Level 3): Type-safe and optimized data abstraction
(Scala/Java).

You might also like