Real Time Applications

Apache Spark: Ecosystem, components of the Spark context, Spark stage, Spark executor

Apache Spark is an open-source distributed computing system designed to process large

volumes of data quickly and efficiently. It was developed to overcome the limitations of
Hadoop MapReduce, primarily its slower processing speed for iterative tasks and interactive
queries. Spark achieves high performance through in-memory computation and provides a
unified framework for both batch and real-time data processing.

Spark supports multiple programming languages including Java, Scala, Python, and R,
enabling a wide range of users: data engineers, data scientists, and analysts to leverage its
capabilities. Spark can process data stored in various sources like HDFS, Apache Cassandra,
Apache HBase, Amazon S3 and many more.

A key strength of Spark is its ability to handle iterative algorithms and interactive queries
efficiently by keeping intermediate data in memory, thereby avoiding repeated disk I/O. This
makes Spark up to 100 times faster than Hadoop MapReduce for certain workloads.
Apache Spark Ecosystem Components

The Apache Spark ecosystem is built around the Spark Core, which serves as the foundation
for all Spark functionalities. On top of Spark Core, several specialized libraries enable a wide
variety of data processing tasks. The major components of the Spark ecosystem are:

1. Spark Core: The foundation of Apache Spark, providing essential services like - Task
scheduling, In-memory computation, I/O functionalities and Fault tolerance.
Introduces Resilient Distributed Datasets (RDDs), which are immutable, partitioned
collections of objects distributed across the cluster.

• Partitioning and parallelism of data across nodes

• Task dispatching and cluster monitoring
• Transformation and Action operations on RDDs
o Transformation: Creates a new RDD from an existing one
(e.g., map(), filter())
o Action: Triggers computation and returns results (e.g., collect(), count())

2. Spark SQL: Module for structured data processing, supporting SQL queries and
DataFrame/Dataset APIs. Enables optimizations through Catalyst optimizer and allows
integration with existing Hive tables and various data sources like JSON, Parquet, ORC and
JDBC. Useful for both interactive analytics and batch processing of structured/semi-structured
data.

3. Spark Streaming: Supports real-time data processing from sources such as Kafka, Flume,
Kinesis, TCP sockets and HDFS. Uses micro-batching, converting live data streams into small
batches, which are then processed by the Spark engine. Provides a high-level abstraction
called DStream (Discretized Stream), representing continuous streams as sequences of RDDs.

4. MLlib (Machine Learning Library): A scalable library for machine learning in Spark.
Provides implementations of algorithms for classification, regression, clustering,
collaborative filtering and utilities for feature extraction, transformation, and dimensionality
reduction. The newer DataFrame-based API (from Spark 2.0) is preferred over the older
RDD-based API for user friendliness and integration with Catalyst and Tungsten optimizations.

5. GraphX: API for graph processing and analysis. Provides a directed multigraph
abstraction, extending RDDs to support graph-parallel computation. Supports operations such
as subgraph(), joinVertices(), aggregateMessages(), and algorithms like PageRank,
Connected Components and Triangle Counting.

6. SparkR: R package that allows R users to interact with Spark. Provides SparkR
DataFrames, scalable across multiple cores and nodes. Enables integration with Spark SQL
data sources and benefits from in-memory computation and distributed processing.

Core Spark Concepts

1. Resilient Distributed Datasets (RDDs): Fundamental data structure in Spark.

Immutable and distributed, allowing parallel processing. Fault tolerance is maintained
via lineage, tracking transformations for recomputation in case of node failure.

2. Partitions: RDDs are split into multiple partitions, each processed independently on
worker nodes. Partitioning enables parallelism and efficient resource utilization.

3. Transformations and Actions:

• Transformations: Lazy operations that define new RDDs from existing ones
(map(), filter(), reduceByKey()).
 • Actions: Trigger the execution of transformations and return results
(collect(), count(), saveAsTextFile()).

4. Directed Acyclic Graph (DAG): Spark builds a DAG of stages based on RDD
transformations to optimize execution. DAG ensures fault tolerance and efficient scheduling,
rerunning only affected stages on failure.

5. DataFrames and Datasets: High-level structured APIs over RDDs.

• DataFrames: Untyped, row/column-based representation

• Datasets: Strongly-typed, JVM objects providing compile-time type safety

Spark Cluster Architecture

1. Driver Program: The central coordinator of a Spark application. It is responsible for

o Converting user code into tasks

o Scheduling tasks on executors via the cluster manager
o Monitoring job execution and collecting metrics

2. SparkContext / SparkSession: Entry point for Spark applications (SparkContext in older

versions, SparkSession in newer versions). Manages cluster connections, job scheduling and
resource allocation. Creates RDDs, DataFrames and Datasets and maintains Spark UI for
monitoring.

3. Executors: Processes running on worker nodes that perform the actual computation.
Executors remain active for the application’s lifetime and handle multiple tasks in parallel.

o Execute assigned tasks

o Store intermediate data in memory or disk
o Report status and results back to the driver

4. Spark Stage: A physical unit of execution within a Spark job. Contains a set of tasks that
can run in parallel. Defined by the DAGScheduler based on RDD dependencies. Stages are
separated by operations requiring data shuffling (e.g., groupByKey).
5. Tasks: Units of work executed by executors. Each task operates on a single partition of an
RDD or DataFrame.

6. Cluster Manager: External service responsible for allocating resources to Spark

applications. Supported types:

o Standalone: Spark’s built-in manager

o YARN: Hadoop resource manager
o Mesos: General purpose cluster manager
o Kubernetes: Container orchestration platform

7. Worker Nodes: Physical machines where executors run. Manage task execution, memory
storage, and communication with the driver.

How Spark Works: Execution Flow

1. Spark Submit: User submits a job using spark-submit.

2. Driver Launch: Spark driver starts and requests resources from the cluster manager.
3. SparkContext Initialization: Parallel transformations and actions are recorded as
DAG lineage.
4. Action Trigger: When an action is called, a job is created.
5. Job Scheduling: DAG is split into stages, and each stage is divided into tasks.
6. Task Execution: Tasks are scheduled on executors on worker nodes.
7. Result Aggregation: Executors execute tasks, store intermediate results in memory and
return results to the driver.

Example: Filtering words containing "spark" from a list and counting them:

from pyspark import SparkContext

sc = SparkContext("local", "count app")

words = [Link](["scala", "java", "hadoop", "spark", "akka", "spark vs hadoop",
"pyspark", "pyspark and spark"])
words_filter = [Link](lambda x: 'spark' in x)
counts = words_filter.count()
 print("Number of elements filtered ->", counts)

Here, filter() is a transformation, and count() is an action.

Best Practices for Spark Applications

• Optimize partitioning: Avoid data skew and improve parallelism.

• Leverage in-memory computation: Use cache() or persist() for frequently accessed
data.
• Choose efficient data formats: Columnar formats like Parquet or ORC for analytics
workloads.
• Minimize shuffles: Apply narrow transformations before wide transformations.
• Use broadcast variables: Efficiently distribute read-only datasets to all executors.
• Monitor and tune: Use Spark UI and metrics to identify bottlenecks and optimize
resources.

SDD Architecture

The SDD (Spark Distributed Data) Architecture describes how Apache Spark efficiently
stores, manages and processes large-scale data across distributed computing environments. It
ensures fault tolerance, scalability and high performance through an integrated design that
combines storage, data abstraction and distributed execution.

1. Storage Layer

• This is the physical layer where the actual data resides.

• Spark interacts seamlessly with various distributed storage systems such as:
o HDFS (Hadoop Distributed File System)
o Amazon S3, Apache Cassandra, HBase or local file systems
• Data is stored in partitions, allowing Spark to process chunks of data in parallel.
• Replication within distributed file systems ensures fault tolerance and data
availability.
• Spark reads data lazily and in partitions instead of loading entire datasets into
memory, enabling large-scale, memory-efficient computation.

2. Data Layer
 • The data layer defines Spark’s logical data abstractions:
o RDD (Resilient Distributed Dataset) – Immutable, distributed collections of
data elements.
o DataFrame – Distributed collections of data with schema information (column
names and types).
o Dataset – Type-safe, object-oriented abstraction combining RDD and
DataFrame benefits.
• These abstractions manage:
o Partitioning – dividing data for parallel processing
o Caching – storing frequently accessed data in memory
o Fault tolerance – using lineage information to recompute lost partitions
• Schema awareness in DataFrames and Datasets allows Catalyst Optimizer to
improve query execution efficiency.

3. Driver Layer

• The Driver Program acts as the central coordinator of the Spark [Link]:
o Initializes SparkContext (for RDD-based API) or SparkSession (for unified
API access).
o Converts user-defined transformations and actions into a Directed Acyclic
Graph (DAG) of stages and tasks.
o Communicates with the Cluster Manager (YARN, Mesos, Kubernetes, or
Standalone) to allocate resources.
o Collects and returns results from worker nodes.
• The Driver also manages job scheduling, task distribution and failure recovery.

4. Execution Flow: The Spark execution flow under the SDD architecture follows these steps:

1. Data Loading: Data resides in external storage (HDFS/S3/etc.) and is loaded into
RDDs or DataFrames.
2. Logical Plan: Transformations and actions applied to data are converted into a logical
DAG by the driver.
3. Physical Plan: The DAG Scheduler splits the computation into stages and each stage
is divided into multiple tasks based on partitions.
4. Execution:
 o Tasks are sent to executors on worker nodes.
o Executors perform the assigned computation, store results in memory (if
cached), and write final results back to storage.
5. Result Collection: The driver gathers outputs or writes them to persistent storage.

5. Benefits of SDD Architecture

Feature Description
Data and tasks are distributed across multiple nodes,
Distributed Computation
enabling parallel processing.
Spark uses RDD lineage to recompute lost partitions
Fault Tolerance
automatically.
Intermediate results can be cached in memory, improving
In-memory Computation
iterative processing speeds.
The driver and cluster manager optimize resource use for
Dynamic Resource Allocation
scalability and efficiency.
Supports a wide range of data sources and formats, scaling
Scalability and Flexibility
from a laptop to a cluster.

RDD and RDD Operations – RDD Features and limitations - RDD persistence - Caching
mechanism - DAG

RDD (Resilient Distributed Dataset): An RDD is the fundamental data abstraction in Apache
Spark. It is a distributed collection of immutable objects that can be processed in parallel
across a cluster.

• Immutable: Once created, RDDs cannot be changed.

• Distributed: Data is split across nodes in a cluster.
• Fault-tolerant: Can recover from node failures using lineage information.
• Lazy evaluation: Transformations are computed only when an action is called.
Creating an RDD: RDDs can be created in two main ways:

1. From existing data in your program (parallelized collection): Use [Link]() to

create an RDD from a Python list or collection.

data = [1, 2, 3, 4, 5]
rdd = [Link](data)
print([Link]()) # Output: [1, 2, 3, 4, 5]
rdd = [Link](data, numSlices=3) (we can specify the number of partitions)

2. From external storage (text file, HDFS, S3, etc.): Use [Link]() to read data from
storage into an RDD.

rdd = [Link]("[Link]") # Read [Link] into an RDD

print([Link](5)) # Print first 5 lines
Supports reading from distributed storage:

rdd = [Link]("hdfs://namenode:9000/path/[Link]")

RDD Operations: RDD operations are divided into Transformations and Actions:

A. Transformations (lazy operations): Transformations create a new RDD from an existing

one.

• map(func) → applies a function to each element.

 • filter(func) → keeps only elements satisfying a condition.
• flatMap(func) → maps and flattens elements.
• groupByKey() → groups values by key.
• reduceByKey(func) → combines values by key using a function.

B. Actions (trigger computation): Actions return a result to the driver program or write data
to storage.

• collect() → returns all elements to the driver.

• count() → returns the number of elements.
• take(n) → returns the first n elements.
• saveAsTextFile(path) → saves RDD to HDFS or local filesystem.

RDD Features

1. Fault tolerance through lineage graphs.

2. Lazy evaluation improves performance by minimizing unnecessary computations.
3. In-memory computation: Speeds up iterative algorithms.
4. Partitioning: Data is split across nodes for parallelism.
5. Supports functional operations like map, filter, reduce.

RDD Limitations

1. No support for schema (unlike DataFrames/Datasets).

2. Cannot handle structured queries efficiently (SQL-like operations are better in
DataFrames).
3. Less optimized than DataFrames for certain operations (Catalyst optimizer is absent).
4. Manual memory management: Requires explicit caching for repeated access.
5. No automatic optimization for complex workflows.

RDD Persistence: Persistence allows you to store RDDs in memory or on disk for reuse,
improving performance in iterative operations.

Persistence Levels

1. MEMORY_ONLY – Stores RDD in memory (fastest).

 2. MEMORY_AND_DISK – Keeps in memory; spills to disk if memory is full.
3. DISK_ONLY – Stores only on disk.
4. MEMORY_ONLY_SER / MEMORY_AND_DISK_SER – Stores serialized RDDs
to reduce memory footprint.
5. OFF_HEAP – Uses off-heap memory for storage (experimental).

Syntax:

[Link](StorageLevel.MEMORY_AND_DISK)
[Link]() # remove from cache

Caching Mechanism

• Caching is a shortcut for persistence using the default storage level

(MEMORY_ONLY).
• Useful when you access the same RDD multiple times in a program.
• Improves performance by avoiding recomputation from lineage every time.

rdd = [Link]("[Link]")
rdd2 = [Link](lambda x: [Link]())
[Link]() # cache the RDD
print([Link]()) # triggers computation

DAG (Directed Acyclic Graph): When transformations is done, Spark does not execute them
immediately. It builds a DAG of stages. DAG represents logical execution plan for RDD
operations. When an action is called, Spark optimizes the DAG and schedules tasks across the
cluster. DAG ensures fault tolerance, because if a partition is lost, Spark can recompute it
from its lineage.

• DAG = Task execution plan for a Spark job.

• Each node = RDD, edge = transformation.
• Stages = sets of transformations without shuffle; Shuffles = boundary between stages.
 Example Workflow:
rdd = [Link]("[Link]") → Load data
rdd2 = [Link](lambda x: "error" in x) → Transformation
[Link]() → Action triggers DAG execution

Concept Description
RDD Immutable, distributed collection of objects
Transformations Lazy operations creating new RDDs (map, filter)
Actions Trigger computation (collect, count)
Features Fault-tolerant, in-memory, partitioned, lazy
Limitations No schema, less optimized, manual caching needed
Persistence Store RDD in memory/disk to reuse (persist)
Caching Shortcut for persistence, default MEMORY_ONLY
DAG Logical execution plan; supports fault tolerance & optimization

Spark cluster management

Apache Spark is a powerful engine for big data processing. When running Spark on a
distributed cluster, cluster management is critical to allocate resources, schedule tasks, and
monitor execution. Spark supports pluggable cluster managers, allowing flexibility
depending on infrastructure and use cases.

The Cluster Manager orchestrates Spark applications across a cluster:

• Launches the Driver: Coordinates the application.

• Allocates Resources: Provides CPU and memory to Spark Executors.
• Manages Executors: Starts and monitors executors on worker nodes.
• Handles Failures: Recovers tasks from failed nodes/executors.

Spark applications run as independent processes (executors) on a cluster, and

the SparkContext in the driver program communicates with the cluster manager to coordinate
tasks.

Types of Spark Cluster Managers

a) Standalone Cluster Manager: Simple, built-in cluster manager. Suitable for small-to-
medium clusters.

• Setup: Easy, no additional dependencies, works on Linux, Windows, Mac.

• Resource Allocation: Based on CPU cores and memory configured per worker.
• High Availability: Supported via ZooKeeper quorum or manual recovery.
• Security: Authentication via shared secret; SSL/SASL encryption supported.
• Monitoring: Web UI shows jobs, tasks, and executor statistics.
• Pros: Easy setup, resilient, access to HDFS.
• Cons: Limited scalability, basic resource management.

b) Hadoop YARN (Yet Another Resource Negotiator): Resource manager for the Hadoop
ecosystem. Enables running Spark alongside other Hadoop applications.
o ResourceManager: Global scheduler and application monitor.
o NodeManager: Runs containers for tasks, reports metrics.
o Application Master: Requests resources and executes tasks per application.
• High Availability: Manual via CLI; automatic via ZooKeeper ActiveStandbyElector.
• Security: Kerberos authentication, SSL for data and Web console communication,
ACLs for service access.
 • Monitoring: Web UI for ResourceManager and NodeManager metrics.
• Pros: High scalability, robust scheduling, Hadoop integration.
• Cons: Complex setup, overhead from Hadoop ecosystem.

c) Apache Mesos: Distributed cluster manager for multiple frameworks (Spark, Hadoop,
Chronos, Marathon, etc.).

o Mesos Master: Assigns resources; multiple masters for fault tolerance.

o Mesos Slave: Provides resources to tasks.
o Frameworks: Applications request resources from Mesos.
• High Availability: Automatic recovery via ZooKeeper; running tasks continue during
failover.
• Security: Authentication for slaves, frameworks, and operators; optional SSL/TLS;
ACLs for services.
• Monitoring: Metrics for master and slave nodes available via URLs.
• Pros: High scalability, supports multiple frameworks, fine-grained resource sharing.
• Cons: Complex setup, requires additional components.

d) Kubernetes

• Description: Container orchestration platform increasingly used for Spark

applications.
• Features: Advanced resource management, scaling, fault tolerance, containerized
deployment.
• Pros: High scalability, containerization, fault tolerance.
• Cons: Requires containerization knowledge; setup is more complex.

Comparison of Cluster Managers

Feature Standalone YARN Mesos Kubernetes

Setup Simple Complex Complex Complex
Scalability Low-Medium High High High
Built-in (via
High Availability ZooKeeper/manual ZooKeeper/CLI ZooKeeper
pods)
 Shared secret, Kerberos, SSL, ACL, optional
Security RBAC, TLS
SSL/SASL ACL SSL
Kubernetes
Web UI (RM & Web UI /
Monitoring Web UI dashboard &
NM) metrics
logs
Hadoop Multi- Cloud-native,
Dev & small
Use Case integration, framework, containerized
clusters
large clusters large clusters apps

Performance tuning
Spark Performance Tuning is the process of adjusting system settings, resource allocations, and
application code to ensure optimal execution of Spark jobs, avoid bottlenecks, improve
execution time and efficiently utilize resources (CPU, memory, I/O). Spark’s in-memory
computation can lead to resource contention (CPU, memory) if not tuned properly.
Misconceptions about Spark:

• Default configurations are always sufficient – False.

• Spark is low-code with drag and drop UI – False, requires optimized code.
• Spark has a storage layer – False, data is only cached/persisted during job execution.

Spark Performance Optimization Categories

1. Code-level optimizations – Efficient Spark programming practices.
2. Storage-level optimizations – Data formats, serialization, compression.
3. Configuration-level optimizations – Cluster and JVM tuning, memory allocation.

Code-Level Optimizations
a. Use DataFrames/Datasets instead of RDDs: Built-in Catalyst optimizer and Tungsten
project enhance performance.
Catalyst Optimizer: Generates efficient execution plans using predicate pushdown, column
pruning and code generation.
Tungsten: Off-heap memory management, binary encoding, and bytecode generation.

b. Caching and Persistence: Avoid recomputation of RDDs/DataFrames in iterative

transformations.
cache() – stores in memory (MEMORY_ONLY by default).
persist(StorageLevel) – allows memory+disk, memory-only serialized or disk-only options.

c. Avoid expensive groupByKey operations: Prefer reduceByKey or aggregateByKey –

reduces shuffles and memory usage.

d. Broadcast Variables: Efficiently share small, read-only datasets across all worker nodes.
Reduces shuffling and network overhead.
[Link](broadcast(dataset2))

e. Minimize collect() and count() usage: These bring all data to the driver. Use take() or
show() to inspect small samples.

f. Prefilter irrelevant data: Push filters early using predicate pushdown to reduce processing.

g. Reduce UDF usage: UDFs bypass Catalyst optimization and introduce serialization
overhead. Prefer built-in Spark functions or Pandas UDFs when vectorized processing is
needed.

h. Partitioning and Parallelism: Choose appropriate number of partitions to avoid skewed

workloads. Recommended partition size: ~128 MB.
repartition() – triggers full shuffle for balanced partitions.
coalesce() – reduces number of partitions, may avoid shuffle.
Tune [Link] and [Link].

Storage-Level Optimizations
a. Data Serialization: Converts in-memory objects to byte streams for network transfer or
storage.
Java Serialization – Flexible but slow and produces large serialized objects.
Kryo Serialization – Compact, faster, requires class registration:
[Link]("[Link]", "[Link]")
[Link]([MyClass])

b. Compression: Reduces disk and network I/O overhead.

Libraries: Snappy, LZ4, Zstd, LZF
c. File Formats:
Structured Data: Parquet, ORC (columnar, supports predicate pushdown).
Semi-structured: JSON, Avro (Row based formats).
Unstructured: Text, Binary (images, files).

Configuration-Level Optimizations
a. Memory Management:
Execution memory: Used for shuffles, joins, aggregations.
Storage memory: Used for caching/persisting data. Unified memory region with dynamic
sharing between execution and storage.
Key parameters:
[Link] (default 0.6) – fraction of JVM heap for Spark memory.
[Link] (default 0.5) – fraction of Spark memory reserved for storage.

b. Garbage Collection Tuning:

Monitor GC logs (-verbose:gc -XX:+PrintGCDetails) to avoid full GCs.
Reduce memory churn by using compact data structures or serialized RDDs.

c. Dynamic Resource Allocation: Spark can scale executors based on workload:

[Link] = true
[Link] = true

d. Shuffle Optimization: Use map-side combiners to reduce shuffle size. Broadcast joins for
small datasets. Tune [Link] based on cluster resources.

e. Parallelism & Level of Concurrency: Ensure high enough parallelism to utilize all cores.
Adjust partitions and cores to balance workload and prevent idle resources.

Monitoring Spark Performance

Tools: Spark Web UI, Ganglia, custom logs.
Key metrics: executor/driver memory, task duration, CPU/I/O usage.
Tune based on observations, e.g., adjust memory, shuffle partitions, caching strategy.

Advanced Optimization
Adaptive Query Execution (AQE): Dynamically optimize joins, shuffles and skewed data.
Speculative Execution: Mitigates slow tasks by running redundant tasks on other nodes.
Custom partition coalescing: Reduces overhead in stages with narrow transformations.
Predicate pushdown & vectorized reads: For better query performance.

Data frames and Dataset

Apache Spark provides multiple abstractions for distributed data processing - RDDs,
DataFrames and Datasets: each evolving to offer higher abstraction, better performance, and
ease of use. These APIs coexist and complement each other, and understanding their
differences helps in selecting the right tool for the job.

1. Resilient Distributed Datasets (RDDs): RDDs are Spark’s fundamental data structure,
introduced in Spark’s first release. They represent an immutable, distributed collection of
objects that can be processed in parallel across a cluster.

Features:
• Immutable and partitioned across nodes.
• Provides low-level control with transformations (map, filter, reduceByKey) and actions
(collect, count, saveAsTextFile).
• Fault-tolerant using lineage information—if data is lost, RDDs can recompute it.
• Supports unstructured and semi-structured data.

Use Cases:

• When fine-grained control of data processing is required.

• Iterative algorithms (e.g., K-means, PageRank).
• ETL pipelines with custom logic.
• Real-time data with Spark Streaming.
• Graph data processing with GraphX.
 Example (Python):

from pyspark import SparkContext

sc = SparkContext("local", "RDD Example")

# Create RDD
data = [1, 2, 3, 4, 5]
rdd = [Link](data)
# Transform and filter
rdd2 = [Link](lambda x: x * 2).filter(lambda x: x > 5)

# Collect results
print([Link]())

Limitations:
• No schema enforcement (harder for structured data).
• No built-in query optimization.
• Verbose and lower-level API.

2. DataFrames: Introduced in Spark 1.3, a DataFrame is a distributed collection of data

organized into named columns, conceptually similar to a table in SQL or a data frame in
R/Python. Internally, it is built on top of RDDs but adds schema awareness and optimizations.

Features:
• Provides SQL-like operations (select, filter, groupBy, join).
• Schema enforcement at runtime.
• Available in Scala, Java, Python, and R.
• Optimized by Catalyst optimizer (logical/physical plan rewriting, predicate
pushdown).
• Executes efficiently via Tungsten engine (code generation, memory management).

Use Cases:
• Working with structured/semi-structured data (CSV, JSON, Parquet, Hive tables).
• SQL-style analytics.
 • BI integration (Tableau, Power BI).
• ETL workflows.
• Streaming with Structured Streaming.

Example (Scala):
import [Link]

val spark = [Link]("DataFrame Example").getOrCreate()

val data = Seq(("John", 25), ("Jane", 30), ("Bob", 45))
val df = [Link](data).toDF("name", "age")

[Link]()

3. Datasets: Introduced in Spark 1.6, Datasets provide the benefits of RDDs (strong typing,
functional transformations) combined with DataFrame optimizations. They are only available
in Scala and Java.

Features:
• Strongly typed: each row maps to a JVM object (case class in Scala, class in Java).
• Provides compile-time type safety (errors caught before runtime).
• Uses Encoders for efficient serialization.
• Supports both functional transformations (map, flatMap) and SQL-like operations.

Use Cases:
• When working with domain-specific objects.
• Applications requiring compile-time type safety.
• Complex ETL and ML pipelines in Scala/Java.

Example (Scala):
case class Person(name: String, age: Int)
val data = Seq(Person("John", 25), Person("Jane", 30))
val ds = [Link](data)
[Link]()

4. Spark SQL: Spark SQL is the engine powering DataFrames and Datasets.

• Unifies APIs (SQL queries, DataFrames, Datasets).

• Developers can switch seamlessly between SQL and DataFrame/Dataset APIs.
• Optimizations via Catalyst (query optimization) and Tungsten (execution engine).
• Supports integration with Hive, JDBC/ODBC, BI tools.

Example:
 [Link]("people")
result = [Link]("SELECT name, age FROM people WHERE age > 30")
[Link]()

5. RDD vs DataFrame vs Dataset:

Feature RDD DataFrame Dataset (Scala/Java)

Mid-high, typed +
Abstraction Level Low-level, functional High-level, tabular
tabular
Type Safety No No Yes (compile-time)
Schema Support None Runtime schema Compile-time schema
Catalyst + Tungsten +
Optimization None Catalyst + Tungsten
Encoders
Moderate, needs case
Ease of Use Verbose, complex Easy, SQL-like
classes
Lowest (no High + efficient
Performance High
optimizations) serialization
Language Scala, Java, Python,
Scala, Java, Python, R Scala, Java only
Support R
Unstructured data, Structured/semi- Typed data, domain
Use Cases
ML, ETL structured, SQL objects

When to Use What?

Use RDDs: When you need low-level control, unstructured data processing, or iterative
ML/graph algorithms.

Use DataFrames: For most structured and semi-structured workloads, SQL queries, and
ETL pipelines where ease and performance matter.

Use Datasets: When you want type safety and domain-specific object mapping (Scala/Java
only).
 In-memory distributed processing using Apache Spark

Big data analytics requires systems that can process massive volumes of data efficiently.
Traditional systems like Hadoop MapReduce rely heavily on disk-based computation,
leading to slower performance due to repeated disk I/O. Apache Spark overcomes this
limitation by enabling in-memory distributed processing, where data is kept in RAM and
processed in parallel across a cluster. This approach accelerates computation, making Spark
especially powerful for iterative algorithms, machine learning, and real-time analytics.

What is Spark In-Memory Computing? In-memory computing means keeping data in RAM
instead of disk drives, enabling faster access and processing.

Key pillars:
• RAM Storage – avoids expensive disk I/O.
• Parallel Distributed Processing – data is partitioned and processed simultaneously
across multiple nodes.

Benefits:
• Faster execution of iterative and interactive queries.
• Economical, as RAM has become more affordable.
• Highly suitable for machine learning, graph processing, and micro-batch stream
processing.

Core Components Enabling In-Memory Distributed Processing

1. Resilient Distributed Datasets (RDDs): Fundamental data structure in Spark. Immutable,

fault-tolerant, and distributed collections of objects. Can be cached in memory across
nodes → avoids recomputation. Supports lineage-based fault tolerance → lost partitions can
be recomputed.
2. Caching and Persistence

• cache(): Stores RDD/DataFrame in memory (MEMORY_ONLY by default).

• persist(): Allows fine-grained control with multiple storage levels.

Storage Levels in persist()

1. MEMORY_ONLY – RDD as deserialized Java objects in memory.

2. MEMORY_AND_DISK – Spill to disk if memory is insufficient.
3. MEMORY_ONLY_SER – Serialized objects in memory (space-efficient).
4. MEMORY_AND_DISK_SER – Serialized objects in memory + spill to disk.
5. DISK_ONLY – Store only on disk.
6. MEMORY_ONLY_2 / MEMORY_AND_DISK_2 – Replicates partitions on 2
nodes for higher fault tolerance.

3. DAG Scheduler: Spark represents transformations as a Directed Acyclic Graph (DAG).

Optimizes job execution by minimizing data shuffling and recomputation. Ensures
efficient task scheduling and fault recovery.

4. Fault Tolerance: Even with in-memory storage, Spark ensures resilience. Uses RDD
lineage to recompute lost data partitions. No explicit replication required unless specified.

5. Parallel Processing: Tasks are distributed across cluster nodes. Multiple partitions
processed concurrently. Combines with in-memory caching for massive performance gains.

Example: Iterative Machine Learning in Spark

In machine learning, iterative computations (e.g., gradient descent, clustering) repeatedly

access the same dataset.

• In Hadoop MapReduce → every iteration reads data from disk.

• In Spark → data cached in RAM → dramatically reduces execution time.

rdd = [Link]("[Link]")
[Link]()
# Run multiple transformations without reloading from disk
result = [Link](lambda x: [Link](",")).filter(lambda x: int(x[1]) > 50).count()
Advantages of In-Memory Distributed Processing in Spark

• High Speed: Eliminates repeated disk reads/writes.

• Efficient for Iterative Jobs: ML, graph analytics, streaming.
• Low Latency: Supports real-time risk management and fraud detection.
• Reduced I/O Overhead: Intermediate results stay in RAM.
• Scalable & Fault-Tolerant: Handles petabyte-scale workloads with resilience.
• Economic: Falling RAM costs make in-memory feasible for enterprises.

Applications

• Machine Learning & AI – faster training of models.

• Graph Processing – iterative graph algorithms (PageRank).
• Real-time Analytics – fraud detection, anomaly detection.
• Interactive Queries – faster response for BI dashboards.
• Streaming Data Processing – with Spark Streaming & Structured Streaming.

Spark shell commands

Spark Shell provides an interactive environment to experiment with Apache Spark. It is very
useful for learning the Spark API, exploring data interactively and testing transformations
and actions before writing full applications. It supports multiple languages:

1. spark-shell → Scala shell (default, since Spark is written in Scala).

2. pyspark → Python shell (popular for data science users).
3. sparkR → R shell (for statisticians and R users).

Starting the Spark Shell

• Scala: spark-shell
• Python (PySpark): pyspark
• R: sparkR

Once started, Spark automatically provides:

• SparkContext (sc) → for RDD-based operations.

• SparkSession (spark) → for DataFrames, Datasets, and SQL queries.
Basic Commands in Spark Shell

Check Spark Context and Session

[Link] // Spark version

[Link] // Application name
[Link] // Cluster manager (local/YARN/etc.)
[Link] // SparkSession version

Stop Spark Context

[Link]()

Working with RDDs - Creating RDDs

From a collection:
val data = Array(1, 2, 3, 4, 5)
val rdd = [Link](data)
From a file:
val textFile = [Link]("path/to/[Link]")

Basic Actions
• Count elements: [Link]()
• Collect results: [Link]()
• First element: [Link]()
• Take first n elements: [Link](3)

Basic Transformations

• Map: val mapRDD = [Link](x => x + 10)

• Filter: val filterRDD = [Link](x => x != 2)
• FlatMap (split into multiple elements): val words = [Link](line => [Link](" "))

DataFrames & Spark SQL in Shell

• Read file into DataFrame: val df = [Link]("path/to/[Link]")

• Create temp view & run SQL:
[Link]("my_view")
[Link]("SELECT * FROM my_view LIMIT 5").show()

Intermediate Spark Shell Commands

 • Save results to a file: [Link]("/output/path")
• Count partitions: [Link]
• Join RDDs: [Link](rdd2)
• Chain operations: [Link](_.contains("yes")).count()

Caching & Persistence: Caching improves performance by storing data in memory.

• Cache: [Link]()
• Persist with storage levels:
import [Link]
[Link](StorageLevel.MEMORY_AND_DISK)
• Unpersist (remove from memory): [Link]()

Advanced Spark Shell Commands

1. Broadcast Variables: Distribute a read-only variable efficiently across nodes.

val bVar = [Link](Array(1, 2, 3))
[Link]

2. Accumulators: Used as counters (for sums, metrics, etc.).

val accum = [Link]("My Accumulator")
[Link](x => [Link](x))
[Link]

3. Coalesce: Reduce the number of partitions without full shuffle.

val rdd2 = [Link](2)

Example: Word Count in Spark Shell

val text = [Link]("[Link]")

val counts = [Link](line => [Link](" "))
.map(word => (word, 1))
.reduceByKey(_ + _)
[Link]()

• Use pyspark if you are comfortable with Python.

 • Monitor execution in Spark UI (default: [Link]
• Always check the number of partitions before big jobs.
• Use cache/persist when data is reused multiple times.
• Avoid too much use of collect() (can overwhelm the driver).

Spark MLlib for Machine Learning

Spark MLlib is Apache Spark’s scalable machine learning library, designed to provide high
performance ML algorithms and tools for large datasets. It makes practical machine
learning scalable, distributed, and easy to use. Works seamlessly with Spark Core, SQL,
Streaming and GraphX, ensuring unified data processing and ML in one platform. Available
in Java, Scala, Python (PySpark) and R. Compared to traditional Hadoop/MapReduce
approaches, Spark MLlib is up to 100x faster for iterative ML tasks due to Spark’s in-memory
computation.

Key Features of MLlib

1. ML Algorithms (built-in):
§ Classification: Logistic Regression, Naive Bayes, Decision Trees, Random Forest,
Gradient Boosted Trees, SVMs.
§ Regression: Linear Regression, Generalized Linear Models, Survival Regression.
§ Clustering: K-Means, Gaussian Mixture Models (GMM), Bisecting K-Means.
§ Recommendation: Alternating Least Squares (ALS).
§ Topic Modeling & Mining: Latent Dirichlet Allocation (LDA), frequent itemsets,
association rules, sequential pattern mining.
§ Dimensionality Reduction: PCA, SVD.

2. Feature Engineering:
§ Extraction: TF-IDF, Word2Vec.
§ Transformation: Scaling, normalization, one-hot encoding, hashing.
§ Selection: Chi-squared selector, variance threshold.

3. Pipelines API ([Link]):

§ Inspired by scikit-learn pipelines.
§ Supports chaining of stages → feature transformations → training → evaluation.
 § Built on DataFrames for easier integration.

4. Utilities:
§ Statistics: summary stats, correlations, hypothesis testing.
§ Linear Algebra: dense/sparse vectors & matrices.
§ Model Evaluation: metrics for classification, regression, clustering.

5. Persistence: Save/load trained models and pipelines for reuse in batch or streaming.

MLlib Architecture
§ Spark Core → Provides RDDs, memory management, and distributed execution.
§ MLlib → Machine learning algorithms and tools.
§ Data Sources → Works with HDFS, Cassandra, Hive, HBase, local files, or cloud
storage.
§ APIs:
o [Link] → RDD-based API (now in maintenance mode).
o [Link] → DataFrame-based API (primary API since Spark 2.0).

The shift to DataFrame-based API allows:

• Catalyst & Tungsten optimizations.
• Uniform APIs across languages.
• Easier feature engineering and pipelines.

ML Workflow in Spark MLlib

1. Data Ingestion: df = [Link]("libsvm").load("hdfs://path/data")

2. Data Preprocessing: Handle categorical/numeric features. Use StringIndexer,
OneHotEncoder, VectorAssembler.
3. Model Selection & Training

from [Link] import KMeans

model = KMeans(k=10).fit(df)

4. Prediction: predictions = [Link](df)

5. Evaluation
 from [Link] import ClusteringEvaluator
evaluator = ClusteringEvaluator()
print("Silhouette Score:", [Link](predictions))

6. Deployment

Save model: [Link]("/models/kmeans_model")

Load for reuse: from [Link] import KMeansModel

model = [Link]("/models/kmeans_model")

Example: Classification Pipeline in PySpark

from [Link] import SparkSession

from [Link] import StringIndexer, OneHotEncoder, VectorAssembler
from [Link] import LogisticRegression
from [Link] import Pipeline

spark = [Link]("MLlibExample").getOrCreate()

# Load dataset
df = [Link]("loan_bank.csv", header=True, inferSchema=True)

# Categorical encoding
categoricalCols = ["job", "marital", "education"]
stages = []
for col in categoricalCols:
indexer = StringIndexer(inputCol=col, outputCol=col+"Index")
encoder = OneHotEncoder(inputCols=[[Link]()],
outputCols=[col+"Vec"])
stages += [indexer, encoder]

# Feature assembler
assemblerInputs = [c+"Vec" for c in categoricalCols] + ["age", "balance", "duration"]
assembler = VectorAssembler(inputCols=assemblerInputs, outputCol="features")
stages += [assembler]

# Model
lr = LogisticRegression(featuresCol="features", labelCol="deposit")
stages += [lr]

# Build pipeline
pipeline = Pipeline(stages=stages)

# Train
pipelineModel = [Link](df)
 # Predict
predictions = [Link](df)
[Link]("deposit", "prediction", "probability").show(5)

Advantages of MLlib
§ Scalable: Handles terabytes of data via distributed execution.
§ Fast: Iterative computation in memory (much faster than MapReduce).
§ Integrated: Works with Spark SQL, Streaming, GraphX.
§ Language Flexibility: Java, Scala, Python, R.
§ End-to-End ML: Covers ingestion → preprocessing → training → evaluation →
deployment.

MLlib in Practice
§ Recommendation Engines → Netflix-style ALS recommender.
§ Fraud Detection → Classification on transaction data.
§ Customer Segmentation → Clustering (K-Means/GMM).
§ Predictive Analytics → Regression for demand forecasting.
§ Streaming ML → Use trained models with Structured Streaming for real-time
predictions.