Scribd
Upload
17 views8 pages

Hadoop and Spark Overview

The document provides lecture notes on Hadoop and Spark, focusing on their architecture, functionalities, and differences. Hadoop is described as a reliable, fault-tolerant framework for processing large datasets using MapReduce, while Spark is presented as a more efficient alternative that enables in-memory computations and interactive analysis. Additionally, the notes cover data preparation in Spark and its components, including Spark SQL, Spark Streaming, MLLib, and GraphX.

Uploaded by

abhishekhagowda11
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
17 views8 pages

Hadoop and Spark Overview

The document provides lecture notes on Hadoop and Spark, focusing on their architecture, functionalities, and differences. Hadoop is described as a reliable, fault-tolerant framework for processing large datasets using MapReduce, while Spark is presented as a more efficient alternative that enables in-memory computations and interactive analysis. Additionally, the notes cover data preparation in Spark and its components, including Spark SQL, Spark Streaming, MLLib, and GraphX.

Uploaded by

abhishekhagowda11
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

PES Institute of Technology and Management

NH-206, Sagar Road,Shivamogga-577204

Department of Computer Science and Engineering

Affiliated to

VISVESVARAYA TECHNOLOGICAL UNIVERSITY


Jnana Sangama, Belagavi, Karnataka –590018c

Lecture Notes
on

Module 5
HADOOP AND SPARK
(21CS754)
2021 Scheme

Prepared By,
Mrs. Prathibha S ,
Assistant Professor,
Department of CSE,PESITM

MODULE -5
Hadoop: a framework for storing and processing large data sets.

Apache Hadoop is a framework that simplifies working with a cluster of computers. It


aims to be all of the following things and more:
■ Reliable—By automatically creating multiple copies of the data and redeploying
processing logic in case of failure.
■ Fault tolerant —It detects faults and applies automatic recovery.
■ Scalable—Data and its processing are distributed over clusters of computers
(horizontal scaling).
■ Portable—Installable on all kinds of hardware and operating systems.

Hadoop Architecture
Hadoop: a framework for storing and processing large data sets.

At the heart of Hadoop we find


■ A distributed file system (HDFS)
■ A method to execute programs on a massive scale (MapReduce)
■ A system to manage the cluster resources (YARN)
MAPREDUCE: HOW HADOOP ACHIEVES PARALLELISM
Hadoop uses a programming method called MapReduce to achieve parallelism.

A MapReduce algorithm splits up the data, processes it in parallel, and then sorts,
combines,and aggregates the results back together. However, the MapReduce algorithm
isn’t well suited for interactive analysis or iterative programs because it writes the data
to a disk in between each computational step. This is expensive when working with
large data sets.

Map Reduce example: (simplified example of a MapReduce flow for counting the
colors in input texts)

MapReduce would work on a small fictitious example. You’re the director of a toy company.
Every toy has two colors, and when a client orders a toy from the web page, the web page
puts an order file on Hadoop with the colors of the
toy. Your task is to find out how many color units you need to prepare. You’ll use a
MapReduce-style algorithm to count the colors. First let’s look at a simplified version.
As the name suggests, the process roughly boils down to two big phases:
■ Mapping phase—The documents are split up into key-value pairs. Until we
reduce, we can have many duplicates.

■ Reduce phase—It’s not unlike a SQL “group by.” The different unique occurrences
are grouped together, and depending on the reducing function, a different
result can be created. Here we wanted a count per color, so that’s what the
reduce function returns.

The whole process is described in the following six steps and depicted in figure 5.4
.
1. Reading the input files.
2. Passing each line to a mapper job.
3. The mapper job parses the colors (keys) out of the file and outputs a file for each
color with the number of times it has been encountered (value). Or more technically
said, it maps a key (the color) to a value (the number of occurrences).
4. The keys get shuffled and sorted to facilitate the aggregation.
5. The reduce phase sums the number of occurrences per color and outputs one
file per key with the total number of occurrences for each color.
6. The keys are collected in an output file.
Spark: replacing MapReduce for better performance

What is Spark?

Spark is a cluster computing framework similar to MapReduce. Spark, however, doesn’t


handle the storage of files on the (distributed) file system itself, nor does it handle the
resource management. For this it relies on systems such as the Hadoop File System, YARN, or
Apache Mesos. Hadoop and Spark are thus complementary [Link] testing and
development, you can even run Spark on your local system.

HOW DOES SPARK SOLVE THE PROBLEMS OF MAPREDUCE?


Spark creates a kind of shared RAM memory between the computers of your cluster. This
allows the different workers to share variables (and their state) and thus eliminates the need
to write the intermediate results to disk. More technically and more correctly if you’re into
that: Spark uses Resilient Distributed Datasets (RDD), which are a distributed memory
abstraction that lets programmers perform in-memory computations on large clusters in a
faulttolerant way.1 Because it’s an in-memory system, it avoids costly disk operations.

THE DIFFERENT COMPONENTS OF THE SPARK ECOSYSTEM


Spark core provides a NoSQL environment well suited for interactive, exploratory nalysis.
Spark can be run in batch and interactive mode and supports Python. Spark has four other
large components, as listed below and depicted in figure 5.5.
1 Spark streaming is a tool for real-time analysis.
2 Spark SQL provides a SQL interface to work with Spark.
3 MLLib is a tool for machine learning inside the Spark framework.
4 GraphX is a graph database for Spark. We’ll go deeper into graph databases

DATA PREPARATION IN SPARK


Cleaning data is often an interactive exercise, because you spot a problem and fix the
problem, and you’ll likely do this a couple of times before you have clean and crisp data. An
example of dirty data would be a string such as “UsA”, which is improperly capitalized. At this
point, we no longer work in [Link] but use the PySpark command
line interface to interact directly with Spark.

Spark is well suited for this type of interactive analysis because it doesn’t need to
save the data after each step and has a much better model than Hadoop for sharing
data between servers (a kind of distributed memory).
The transformation consists of four parts:
1 Start up PySpark (should still be open from section 5.2.2) and load the Spark
and Hive context.
2 Read and parse the .CSV file.
3 Split the header line from the data.
4 Clean the data.
Listing 5.3 Connecting to Apache Spark
Step 1: Starting up Spark in interactive mode and loading the context
The Spark context import isn’t required in the PySpark console because a context is readily
available as variable sc. You might have noticed this is also mentioned when
opening PySpark; in case you overlooked it. We then load a Hive context to enable us to work
interactively with Hive. If you work interactively with Spark, the Spark and Hive contexts are
loaded automatically, but if you want to use it in batch mode you need to load it manually.

Step 2: Reading and parsing the .CSV file


Next we read the file from the Hadoop file system and split it at every comma we encounter.
In our code the first line reads the .CSV file from the Hadoop file system. The second line splits
every line when it encounters a comma. Our .CSV parser is naïve by design because we’re
learning about Spark, but you can also use the .CSV
package to help you parse a line more correctly.

Step 3: Split the header line from the data


To separate the header from the data, we read in the first line and retain every line
that’s not similar to the header line.

Step 4: Clean the data


In this step we perform basic cleaning to enhance the data quality. This allows us to
build a better report.
After the second step, our data consists of arrays. We’ll treat every input for a lambda function
as an array now and return an array. To ease this task, we build a helper function that cleans.
Our cleaning consists of reformatting an input such as “10,4%” to 0.104 and encoding every
string as utf-8, as well as replacing underscores with spaces and lowercasing all the strings.
SAVE THE DATA IN HIVE
To store data in Hive we need to complete two steps:
1 Create and register metadata.
2 Execute SQL statements to save data in Hive.

Common questions

Powered by AI

Both YARN in Hadoop and Mesos in Spark are used for resource management, but they play these roles differently aligned to their respective systems' architecture. YARN acts as an operating system for Hadoop, managing resources and scheduling tasks for various Hadoop applications, ensuring high availability and scalability through resource allocation decisions . Mesos, on the other hand, provides a more general resource management solution suitable for a wider range of distributed systems beyond Hadoop alone. While Spark can run on YARN, using Mesos integrates its capabilities directly with Spark's cluster management, offering a more fine-grained level of control and flexibility in resource scheduling across multiple clusters .

Spark's SQL component provides several advantages over traditional MapReduce. Firstly, it simplifies data manipulation using a familiar SQL syntax, allowing users to perform complex queries more efficiently and with less code compared to writing custom MapReduce jobs . Furthermore, Spark SQL optimizes query execution plans and utilizes Spark's in-memory processing capabilities, significantly enhancing performance for interactive queries. It supports a range of data sources and formats seamlessly, enabling more flexible and robust data integration and analysis .

Resilient Distributed Datasets (RDDs) contribute to Spark's fault tolerance by providing a robust system for data handling across a cluster. Each RDD represents a distributed collection of data, partitioned across nodes in the cluster and capable of recomputation if a node fails, relying on lineage information to rebuild lost data . RDDs thus eliminate the need for costly disk writes after each computation step and handle failures through recomputation, ensuring data processing continuity and efficiency in large-scale applications .

Data preparation in Spark differs significantly from Hadoop due to Spark’s in-memory processing capabilities. Unlike Hadoop, which writes intermediary data to disk after each MapReduce step, Spark processes data in-memory, avoiding costly disk operations and speeding up the data preparation process . This is especially advantageous in iterative and exploratory data analysis where data needs to be cleaned and transformed in multiple short cycles. Spark enables such interactivity effectively using PySpark, which supports dynamic data manipulation without needing to persist changes to disk after every operation, thus greatly enhancing processing speed and efficiency .

MapReduce's chief limitation in the context of iterative data processes lies in its dependence on storing intermediary data to the disk, which is resource-intensive and slows down the processing . The model involves splitting data, processing it in parallel, and aggregating results post each computational step, but this disk I/O is costly for iterative tasks where outputs of one stage are repeatedly reused as inputs for subsequent stages. This poses a challenge for applications requiring iterative processing or interactive analysis where performance and speed are crucial .

Hadoop achieves fault tolerance through several mechanisms. It creates multiple copies of data automatically and can redeploy processing logic if a failure occurs . The Hadoop Distributed File System (HDFS) is central to its data storage, ensuring reliability and scalability by distributing data across clusters and handling node failures without loss of data integrity . Additionally, Hadoop's ecosystem includes YARN for resource management, which helps manage system restarts and rescheduling of tasks efficiently in case of node failures. These features make Hadoop suitable for processing large data sets by maintaining high availability which is crucial for long-running batch processes typical in large-scale data operations .

Apache Spark addresses the inefficiencies of MapReduce primarily by leveraging in-memory computing. Unlike MapReduce, Spark uses Resilient Distributed Datasets (RDDs) for data abstraction, allowing in-memory computations which significantly reduce disk I/O operations . This leads to faster processing speeds as intermediate results are not written to disks but retained in memory. Additionally, Spark's shared RAM memory model allows clusters to share variables across nodes, further enhancing speed and efficiency. These features make Spark particularly well-suited for iterative and interactive tasks where latency can be minimized .

Spark can be run locally for testing and development by using the PySpark shell, which comes with a built-in Spark context (sc) allowing users to test and develop applications on a single system before deploying them on a cluster . This approach benefits developers by providing a simplified environment where they can rapidly prototype and experiment with Spark functionalities without needing an actual cluster. It reduces the overhead associated with cluster management during initial development phases and enables iterative testing and debugging efficiently .

Key steps in the data cleaning process within Spark include starting up Spark to load the Spark and Hive contexts, reading and parsing CSV files, splitting header lines from data, and performing basic data cleaning . These steps are crucial as they ensure the data's integrity and prepare it for accurate analysis. For instance, improperly formatted data such as “10,4%” needs transformation for accurate numerical analysis, while strings must be uniformly capitalized and encoded to avoid inconsistencies . Such cleaning allows for more reliable data analysis and enhances the quality of insights drawn from the processed data.

The Spark ecosystem enriches its functionality by offering several components that allow it to handle a broader range of tasks beyond traditional MapReduce operations. Spark Streaming is used for real-time data processing, enabling applications to process live streams of data. Spark SQL provides a SQL-like interface for Spark, allowing structured data manipulation. MLLib offers machine learning capabilities directly within the Spark framework, simplifying the application of advanced analytics on large datasets. Lastly, GraphX supports graph processing, further expanding Spark’s capability to handle complex data relationships within large datasets .

You might also like