What is Spark Configuration?

• Spark configuration refers to the settings that

customize the behavior of a Spark
application.
• These settings, also known as Spark
properties, control various aspects of Spark's
execution, including memory allocation,
parallelism, and network behavior.
• They can be set through a SparkConf object,
environment variables, or through
configuration files.
 What is Spark Configuration?
• 1. What Spark Configuration Controls:
• Resource Allocation:
• Spark configurations determine how much memory and CPU resources
are allocated to the driver and executors, influencing performance and
resource utilization.
• Parallelism:
• Configurations like [Link] control the number of
tasks that can run concurrently, impacting the speed of computations.
• Network Settings:
• Spark configurations can manage network parameters, such as the
number of connections and buffer sizes, affecting data transfer
efficiency.
• Event Logging:
• Spark configurations define how events are logged, which is crucial for
debugging and monitoring.
• Storage and Caching:
• Configurations like [Link] influence how Spark
utilizes memory for caching data and intermediate results, impacting
performance on iterative algorithms.
 What is Spark Configuration?
• 2. How to Configure Spark:
• SparkConf:
• The SparkConf class in the Java API provides a way to set
configuration parameters programmatically.
• Environment Variables:
• Environment variables can be used to set machine-specific
settings, like the IP address, through the conf/[Link] script
on each node.
• Configuration Files:
• Spark supports configuration files (e.g., [Link]) that
allow you to define default settings for your Spark applications.
• Command-line Arguments:
• Some configurations can be passed as command-line arguments
when submitting a Spark job.
 What is Spark Configuration?
• 3. Best Practices:
• Avoid Over-Configuration:
• Databricks recommends against configuring most Spark
properties, especially when upgrading or migrating.
• Understand the Impact:
• Carefully consider the implications of each configuration
setting before modifying them, as incorrect
configurations can negatively impact performance or
stability.
• Use Units:
• When specifying time or byte sizes, always use the
correct units (e.g., ms, s, m, mb, gb).
 What is Spark Configuration?
• 4. Viewing Configurations:
• You can view the current Spark configurations
using
the [Link]().getAll() me
thod in the Spark shell or in your Spark
application.
• You can also use the [Link]() method to
retrieve specific configuration values.
• By understanding and properly configuring Spark,
you can optimize your applications for better
performance, resource utilization, and stability
 What is Spark Context?
• In Apache Spark, a SparkContext serves as
the entry point for interacting with the Spark
cluster and its core functionalities.
• It represents the connection to the Spark
cluster and is responsible for tasks like
creating Resilient Distributed Datasets
(RDDs), accumulators, and broadcast
variables.
• SparkContext is crucial for setting up the
execution environment and managing
resources for your Spark application.
 What is Spark Context?
• Entry Point:
• SparkContext is the initial connection point for any Spark application,
especially in older versions (before Spark 2.0).
• Cluster Connection:
• It establishes a connection to the Spark cluster, enabling your
application to utilize its resources for distributed computations.
• Resource Management:
• SparkContext manages the resources required for executing tasks
across the cluster, ensuring efficient utilization.
• RDD Creation:
• It provides the ability to create RDDs, which are the fundamental data
structures in Spark for distributed data processing.
• Low-Level API:
• SparkContext acts as a gateway to Spark's lower-level API, allowing for
more fine-grained control over the execution environment.
• Single Active Context:
• Only one SparkContext should be active per Java Virtual Machine (JVM),
and if you need a new one, you must first stop the existing one
 What is Spark Session?
• A SparkSession is the entry point for
interacting with Spark functionality and
represents a single point of connection to
the Spark engine.
• It was introduced in Spark 2.0 and
combines the functionalities of
SparkContext, SQLContext, and
HiveContext.
• It's used for creating DataFrames,
executing SQL queries, and interacting
with Spark's data sources.
 What is Spark Session?
• Unified Entry Point:
• It provides a single point of access to all Spark
functionalities, simplifying interactions with Spark.
• DataFrame Creation:
• SparkSession is used to create DataFrames, which are
distributed collections of data, from various sources like
RDDs, files, or databases.
• SQL Queries:
• It enables executing SQL queries on registered
DataFrames, allowing for structured data manipulation.
• Configuration:
• It allows for setting up Spark configurations, like
application name, and other parameters that customize
the Spark environment.
 What is Spark Session?
• Access to SparkContext:
• SparkSession provides access to SparkContext, which
is still useful for low-level operations and working with
RDDs.
• Multiple Sessions:
• SparkSession allows for creating multiple sessions, which
can be useful for isolating different workloads or users.
• Builder Pattern:
• SparkSession is typically created using
the [Link] pattern, providing a fluent API
for configuring the session.
• In essence, SparkSession provides a more user-friendly
and unified way to interact with Spark, making it easier
to perform data analysis and processing tasks.
 What is RDD in Spark?
• RDD stands for Resilient Distributed Dataset.
• - Backbone and fundamental data structure of
Apache Spark
• - Immutable collection of objects
• - Computed on different nodes of the cluster
• - Logically partitioned across servers
 Decomposing the Name RDD
• Resilient: Fault-tolerant via RDD DAG lineage graph,
and so it is able to recompute missing or
damaged partitions due to node failures.
• Distributed: Data resides on multiple nodes
• Dataset: represents records of the data you
work with and the user can load the data
set externally which can be either JSON
file, CSV file, text file, or database via
JDBC with no specific data structure.
 Features of Spark RDD
• 1. In-memory Computation: Spark RDDs
have a provision of in-memory
computation. It stores intermediate
results in distributed memory(RAM)
instead of stable storage(disk).
 Features of Spark RDD
• 2. Lazy Evaluations: By using lazy operations we can
implement Transformations in RDDs. In lazy
evaluation, we can’t compute results
immediately. The generation of results will be
depending upon the trigger of action. Thus, the
performance of the program is increased.
transformations in Apache Spark are lazy, in that
they do not compute their results right away.
Instead, they just remember the transformations
applied to some base data set.
• Spark computes transformations when an action
requires a result for the driver program.
 Features of Spark RDD
• 3. Fault Tolerance: Spark RDDs are fault-
tolerant as they track data lineage
information to rebuild lost data
automatically on failure. They rebuild
lost data on failure using lineage,
each RDD remembers how it was
created from other datasets (by
transformations like a map, join, or
groupBy) to recreate itself.
 Features of Spark RDD
• 4. Immutability: The important fact about RDD
is, it is immutable. You cannot change the
state of RDD. If you want to change the state
of RDD, you need to create a copy of the
existing RDD and perform your required
operations. Hence, the required RDD can be
retrieved at any time.
 Features of Spark RDD
• 5. Partitioning: Data items in RDDs are usually
huge. This data is partitioned and sent
across different nodes for distributed
computing.
• 6. Persistence: Intermediate results generated
by RDD are stored to make the computation
easy. It makes the process optimized.
• 7. Coarse-grained Operations: It applies to all
elements in datasets through maps or filter
or group by operation.
 RDD Operations in Spark
• Two types of operations:
• - Transformations: Create new RDDs
• - Actions: Return final results to the driver
program
 RDD Transformations
• These are functions that accept the
existing RDDs as input and output
one or more RDDs. However, the
data in the existing RDD in Spark
does not change as it is immutable.
• Some of the transformation
operations are provided in the table
next.
 RDD Transformations

These transformations are executed when they are invoked or called. Every
time transformations are applied, a new RDD is created.
 RDD Actions
• Actions in Spark are functions that return
the end result of RDD computations.
• It uses a lineage graph to load data onto
the RDD in a particular order.
• After all of the transformations are done,
actions return the final result to the Spark
Driver.
• Actions are operations that provide non-
RDD values. Some of the common actions
used in Spark are given next.
RDD Actions
 When to Use RDDs
• RDD is preferred to use when you want to
apply low-level transformations and actions.
It gives you a greater handle and control
over your data. We can use RDD when the
data is highly unstructured such as media
or text streams. These are used when you
want to add functional programming
constructs rather than domain-specific
expressions. RDDs are used in situations
where the schema is not applied.
 Conclusion
• We explored:
• - What is RDD
• - Features of Spark RDD
• - RDD Operations
• - When to use RDDs
 Using Mllib
• So let's do a little more complicated example here that actually
drives home the power of spark.
• And what we're going to do is actually use the Movielens dataset
to recommend movies for a given user.
• So what we're going to do is actually look at the entire 100,000
ratings data set and use that to actually predict the rating for
every movie for a given user based on the other stuff they they
watched.
• So just like when you go into Netflix or something, it shows you
recommendations for movies you might like.
• Or maybe if you go to Amazon and you see things like
recommended products for you.
• Same idea here.
• And what we're going to do is use something called Mllib, the
machine learning library that's part of spark built on top of Spark
Core.
• And it contains Machine Learning Model called ALS, which is a