134 Internet of Things

4.1 Big Data : Hadoop

Notes

e
Hadoop is an open-source framework built on Java that facilitates the management
of vast amounts of data for various applications. It adopts a distributed storage and

in
parallel processing approach to handle extensive data and analytical tasks. This
involves breaking down larger workloads into smaller ones, enabling simultaneous
processing. Functioning under the Apache open-source banner, Hadoop serves as

nl
a platform for storing, processing and analysing massive datasets. Operating primarily
in batch or offline processing modes, it does not support online analytical processing
(OLAP). Prominent companies like Facebook, Yahoo, Google, Twitter and LinkedIn
leverage Hadoop for its capabilities. An interesting historical note is that Hadoop was

O
founded in 2002 by Doug Cutting and Mike Cafarella, with its origins rooted in Google’s
release of the Google File System article. As a system, Hadoop can be easily scaled by
incorporating additional nodes into the cluster.

ity
In the next steps, we will look into Hadoop’s history:
™™ Doug Cutting and Mike Cafarella began working on Apache Nutch in 2002. It is
a web crawler software project that is open source.
™™ They were dealing with large amounts of data while working on Apache Nutch.

rs
They have to spend a lot of money to store that data, which is the result of
the project. This challenge becomes one of the primary reasons for the
development of Hadoop.
™™ Google introduced the GFS (Google file system) file system in 2003. It is a
ve
proprietary distributed file system designed to allow fast data access.
™™ Google published a white paper on Map Reduce in 2004. This method
streamlines data processing on huge clusters.
™™ Doug Cutting and Mike Cafarella introduced the NDFS (Nutch Distributed File
ni

System) file system in 2005. Map reduction is also included in this file system.
™™ Doug Cutting left Google in 2006 to join Yahoo. Dough Cutting introduces a new
project Hadoop with a file system known as HDFS (Hadoop Distributed File
U

System) based on the Nutch project. This year saw the introduction of Hadoop’s
first version, 0.1.0.
™™ Doug Cutting gave named his project Hadoop after his son’s toy elephant.
ity

™™ In 2007, Yahoo runs two clusters of 1000 machines.

™™ Hadoop became the fastest system in 2008, sorting 1 terabyte of data on a 900
node cluster in 209 seconds.
™™ In 2013, Hadoop 2.2 was released.
m

™™ In 2017, Hadoop 3.0 was released.

4.1.1 Big Data Introduction

)A

Big data is a term that refers to massive amounts of organised and unstructured
data that organisations collect on a regular basis. The study of this data can result in
operational insights that allow for better business decisions. One of the primary
developments pushing the adoption of big data analytics tools will be IoT. Big Data in
IoT is a vast and rapidly evolving field in which many different methodologies and
(c

strategies can play a role. A dynamic turnaround of methodologies and technologies

can be witnessed as a result of quick progress in Machine Learning and new hardware
improvements.
Amity Directorate of Distance & Online Education
 Internet of Things 135

Massive data collection incorporates storing data generated by various technology

Notes

e
nodes. The operation of IoT networks is dependent on data analysis. The network
generates many sorts of data, noise and some redundant data. Because IoT sensors
and devices degrade with time, Big Data organisations use machine learning to limit

in
the risk of errors and preserve correct decision making. There are numerous examples
in the field of Industrial IoT (IIoT) where they encountered difficulties in discovering,
analysing and fixing errors. There is a requirement for automatic data collecting and error

nl
correction.
Big data has traditionally involved four dimensions, sometimes known as the Four
V’s. They are as follows:

O
1) Volume: amount of data
2) Variety: different types of structured and unstructured data
3) Velocity: processing speed of the data

ity
4) Veracity: truthness of the data
Certain researchers categorise the challenges of dealing with Big Data using the
concept of the “3Vs,” while others may extend this classification by adding dimensions
like Value and Validity. Let’s explore these aspects:
A.

rs
Volume: Internet of Things (IoT) devices generate substantial data, encompassing a
wide range of information such as personal records, stock details, invoices, purchase
histories, card particulars and location data, along with supplementary metadata that
ve
adds context to the information. Organisation s often invest in advanced databases,
data management solutions, distributed systems and cloud storage to accommodate
the sheer volume of digital data. The need to measure, store and transmit all generated
data is paramount, posing a challenge due to the escalation of data volumes beyond
ni

conventional database capabilities.

B. Variety: Big data involves the simultaneous amalgamation of data from diverse
sources. IoT data entails sensor-derived information as well as non-numeric content
like audio and video files, radio signals and more. Managing such diverse data types
U

is complex. Maintaining metadata in alignment with collected data, enabling automatic

correlation for future data sets and addressing software adaptability to evolving sensor
formats or video qualities are essential concerns.
ity

C. Velocity: IoT devices produce data at a rapid pace, often from sensors and other
inputs. Handling this high-velocity data flow becomes challenging, demanding swift
processing to accommodate the continuous inflow. The data rate is also dynamic,
shifting over time due to scenarios like promotional offers leading to increased
sales. Adequate preparation involving planning, processing capabilities and storage
m

is necessary to avoid data loss or system disruptions. While resource allocation

for processing might entail costs, it is a strategic investment for enhancing overall
organisational revenue.
)A

D. Veracity: IoT sensors lack the precision margins found in conventional measuring
instruments. Wireless sensors can experience communication glitches or hardware
malfunctions due to environmental factors, animals, or other interferences.
Consequently, data accuracy, completeness and integrity are crucial. Business
decisions often hinge on the credibility of data. Accurate identification of trustworthy
(c

and unreliable data is pivotal for informed decision-making, helping to delineate

between data that can be relied upon and data that cannot be.

Amity Directorate of Distance & Online Education

 136 Internet of Things

What are the types of Big Data?

Notes

e
™™ Structured
™™ Semi-Structured

in
™™ Unstructured
A) Structured Data
Structured Data is a standardised format that has a well-defined structure. Structured

nl
data is organised in a table with column and row relationships. Structure data can
be found in Excel files or SQL databases, for example. Structured data requires the
existence of a data model — an idea of how data is stored, accessed and processed.

O
Each field is different and can be examined separately or in conjunction with data from
other regions.
B) Semi-Structured Data

ity
Semi-structured data is defined as information that cannot be organised in Relational
Databases or that lacks a definite functional structure but has certain structural
properties. Semi-structured data is information that has been organised by topic or
that fits into a hierarchical programming language. It is a type of structured data that
lacks the tabular format of Data Models associated with Relational Databases. Semi-

rs
structured data is commonly found in object-oriented databases, which comprise XML
documents, HTML files and tables. Semi-structured data has the advantage of being
readily available and capable of yielding in-depth insights.
ve
C) Unstructured Data
Because it has no predetermined shape or organisation, unstructured data is
categorised as Qualitative Data. Every day, businesses get large amounts of
Unstructured data - video, audio, text and more - which is used to train massive
ni

Deep Learning models to address complex real-world problems. However, extracting

insights from unstructured data is tough and needs a significant amount of processing
power.
U

4.1.2 Hadoop – HDFS

Before delving into HDFS (Hadoop Distributed File System), it’s essential to grasp
the concept of a file system. A file system serves as a fundamental data structure or
ity

mechanism within operating systems, facilitating the organisation and management of

files on disk storage. In essence, it empowers users to effectively control and retrieve
data stored on local disks.
In the realm of Windows operating systems, two prominent examples of file systems
m

are NTFS (New Technology File System) and FAT32 (File Allocation Table 32). While
FAT32 was employed in earlier Windows versions, it maintains compatibility with all
iterations of Windows XP. Similarly, in the Linux ecosystem, file systems such as ext3 and
ext4 are pivotal components of the operating system’s structure.
)A

DFS: What is it?

DFS stands for distributed file system, a concept that involves the distribution of
files across multiple nodes. This approach creates the illusion of a unified and extensive
system, whose storage capacity corresponds to the collective storage of all nodes within
(c

a cluster.
To elucidate, consider an example where a DFS comprises four distinct machines,

Amity Directorate of Distance & Online Education

 Internet of Things 137

each with a storage capacity of 10TB. In this scenario, the DFS affords the ability to store
Notes

e
a total of 30TB due to the collective storage capacity of 40TB across the machines. This
allocation of 30TB data is achieved through the partitioning of data into blocks, distributed
among these individual nodes.

in
Why We Need DFS?
You might be questioning the necessity of a Distributed File System (DFS) when

nl
it’s possible to store a 30TB file on a single machine. However, the capacity of a
system’s disk has its limits. Relying solely on a single system for data storage can lead
to processing bottlenecks, as managing extensive datasets on a single workstation is

O
inefficient.
An analogy can help clarify this point. Imagine you have a 40TB file that needs
processing. Running this task on a single system would take around 4 hours. Now,
consider employing a Distributed File System (DFS). In the scenario depicted in the

ity
diagram below, the 40TB file is partitioned across four nodes within a cluster, with each
node accommodating a 10TB segment. Thanks to the simultaneous efforts of these
nodes, the entire process can be completed in just 1 hour. This example underscores the
significance of DFS in optimising processing efficiency.

rs
Local File System Processing
ve
ni
U

[Link]
[Link]
ity

Distributed File System Processing

m
)A
(c

Amity Directorate of Distance & Online Education

 138 Internet of Things

Notes

e
in
nl
O
[Link]
jpg?cb=1487396296

ity
Hadoop Architecture
The purpose of building Hadoop is to create a low-cost, dependable and scalable
framework for storing and analysing the growing amount of large data. Hadoop uses a
master-slave design to effectively store and handle massive amounts of data. The slave
nodes are assigned responsibilities by the master nodes.

rs
The slave nodes are in charge of storing the data and conducting the computation/
processing. The master nodes are in charge of storing the metadata and administering
the cluster’s resources. Slave nodes hold actual business data, while master nodes keep
ve
metadata.
The Hadoop architecture is divided into three tiers. They are as follows:
1. Storage layer (HDFS)
ni

2. Resource Management layer (YARN)

3. Processing layer (MapReduce)
U
ity
m

Fig: The Hadoop Framework’s primary components are HDFS, YARN and MapReduce.
[Link]
)A

Key Characteristics of HDFS (Hadoop Distributed File System)

●● Accessing files within HDFS is straightforward.
●● HDFS ensures high availability and resilience in the face of failures.
●● It offers scalability, allowing for node expansion or reduction based on needs.
(c

●● Data is distributed across multiple Data nodes for storage.

●● HDFS incorporates data replication, mitigating the risk of data loss.

Amity Directorate of Distance & Online Education

 Internet of Things 139

●● It delivers exceptional reliability, accommodating vast data volumes spanning

Notes

e
Petabytes.
●● HDFS boasts built-in servers in both Name nodes and Data Nodes, facilitating

in
seamless cluster information retrieval.
●● It achieves remarkable throughput levels.

HDFS Storage Daemon’s

nl
In HDFS, there are several daemons that play different roles in the storage and
management of data:
1. NameNode: The NameNode is the master daemon in HDFS. It maintains the metadata

O
about the file system hierarchy, such as file and directory information, permissions
and block locations. It keeps track of where each file’s data blocks are stored across
the cluster. The NameNode ‘s health and proper functioning are critical to the overall
stability of the HDFS cluster.

ity
2. DataNode: DataNodes are slave daemons that store the actual data blocks of files.
They are responsible for serving read and write requests from clients and they
periodically report their health and block status to the NameNode . DataNodes provide
fault tolerance by replicating data blocks across multiple nodes in the cluster.
3.

rs
Secondary NameNode: Despite its name, the Secondary NameNode is not a backup or
secondary instance of the NameNode. Instead, it periodically merges the namespace
and edits from the NameNode , creating a new checkpoint. This checkpoint helps to
ve
prevent data loss in case the NameNode fails and needs to be restored.
These daemons work together to ensure data reliability, fault tolerance and efficient
data distribution in a HDFS cluster. The NameNode is crucial for managing the file
system’s metadata, while DataNodes store the actual data blocks and replicate them for
ni

redundancy. The Secondary NameNode assists in creating periodic checkpoints to aid

recovery in case of a NameNode failure.

4.1.3 Hadoop - Map Reduce and Yarn

U

Apache Hadoop MapReduce

ity
m
)A

[Link]
6%2F07%[Link]&tbnid=MGVKAaLfGL_ABM&vet=1&imgrefurl=https%3A%2
F%[Link]%2F%2Ftutorial%2Fbig-data-and-hadoop2Fhadoop-mapreduce-key-features-
(c

highlights%2F&docid=J_QeUkKZy5uwpM&w=621&h=265&source=sh%2Fx%2Fim%2F0
Hadoop’s data processing layer is referred to as MapReduce, a software framework
designed to develop applications capable of handling massive data volumes (ranging

Amity Directorate of Distance & Online Education

 140 Internet of Things

from terabytes to petabytes) in parallel across commodity hardware clusters. The

Notes

e
MapReduce framework operates based on the concept of key-value pairs. Each
MapReduce job represents a specific unit of work that a client intends to perform. This
task consists of input data, the MapReduce program and associated configuration details.

in
Hadoop executes MapReduce jobs by breaking them down into two fundamental types of
tasks: map tasks and reduce tasks. These tasks are managed and scheduled by Hadoop
YARN and they are executed on the nodes within the cluster.

nl
In cases where tasks encounter fail due to unfavourable conditions, automatic
rescheduling is initiated on alternative nodes. The user defines both the map function and
the reduce function for the MapReduce task. The map function takes in a key-value pair

O
as input and generates an intermediate set of key-value pairs as output. These outputs
then serve as the input for the reduce task, where grouping and aggregation operations
are performed.
The MapReduce task is a multi-stage process consisting of the following steps:

ity
1. Map Phase
a. RecordReader: Hadoop divides the input of the MapReduce job into fixed-sise
portions called input splits or splits. The RecordReader transforms these splits
into records and organises the data into records, without parsing the records

b.
rs
themselves. It provides the mapper function with key-value pairs containing data.
Map: In the map phase, a single map task is created for each input split. This
task executes a user-defined function known as the map function on each
ve
record within the split. The map function generates zero or more intermediate
key-value pairs as its output. The map task saves this intermediate output to
local storage. These intermediate results are later processed by the reduce
tasks.
ni

c. Combiner: The output from all map tasks is directed to a single reduce job.
A combiner function, if defined by the user, can be used to process the map
outputs. It organises the map function’s outputs, which then serve as input to
U

the reduction function in the map phase. The combiner is used to optimise the
amount of data transferred to reducers.
d. Partitioner: When there are multiple reducers, map tasks divide their output into
partitions, each intended for a specific reducer. A partition can contain multiple
ity

keys and their associated values, but records for a particular key are grouped
within a single partition. Users can influence partitioning by supplying a custom
partitioning function. By default, a hash function is used for partitioning.
2. Reduce Phase
m

a. Sort and Shuffle: The reducer phase starts with a sort and shuffle process. The
objective is to gather all equivalent keys together. Data written by the partitioner
is shuffled and sorted to the reducer’s node. Each data component is sorted
)A

and grouped into a substantial data list. This sorting and shuffling are handled
automatically by the MapReduce framework, with the option for developers to
customise sorting and grouping using a comparator.
b. Reduce: For each key grouping, the reducer, or user-defined reduce function,
is executed. The reducer processes, filters, aggregates and combines data in
(c

various ways. Once the reduction process is complete, the reducer outputs zero
or more key-value pairs to the OutputFormat. The reduction task’s output is
then stored in Hadoop HDFS.
Amity Directorate of Distance & Online Education
 Internet of Things 141

c. OutputFormat: The OutputFormat employs the RecordWriter to store the

Notes

e
reduction output in an HDFS file. By default, key-value pairs are separated by
tabs and records by newlines.

in
Hadoop Mapreduce

nl
O
ity
rs
[Link]
ve
YARN
YARN, which stands for Yet Another Resource Negotiator, serves as Hadoop’s
resource management layer. It was introduced in Hadoop 2 with the purpose of
ni

decoupling job scheduling and resource management functions into distinct daemons.
The fundamental concept involves a global ResourceManager and an application Master
for each application, whether it’s a single job or a sequence of jobs in a DAG.
U

YARN comprises three main components: ResourceManager, NodeManager and

ApplicationMasters, each associated with individual applications.

Apache Hadoop Yarn

ity
m
)A
(c

[Link]

Amity Directorate of Distance & Online Education

 142 Internet of Things

1. ResourceManager
Notes

e
The ResourceManager plays a pivotal role in allocating resources across the entire
cluster to various applications. It is comprised of two main components: the Scheduler

in
and the ApplicationManager.
a. Scheduler:
 The Scheduler is tasked with distributing resources among multiple

nl
applications running within the cluster, factoring in elements like capacity
and queues.
 However, the Scheduler’s responsibility is limited to allocation and doesn’t

O
involve tracking application statuses.
 In case of task failures due to application or hardware issues, the Scheduler
does not guarantee automatic restart.

ity
 It makes allocation decisions based on the resource requirements of the
applications.
b. ApplicationManager:
 ApplicationManagers are responsible for handling job submissions from

rs
users.
 They negotiate the initial container for the ApplicationMaster, which is
specific to each application.
ve
 ApplicationManagers facilitate the restarting of ApplicationMaster containers
in the event of failures.
 These managers collaborate with the Scheduler for container negotiation,
tracking their status and scaling as needed.
ni

2. NodeManager
The NodeManager operates on slave nodes and is responsible for managing
containers. It also monitors resource utilisation on the machine, encompassing
U

metrics like CPU, memory, disk and network usage. This information is reported to the
ResourceManager or Scheduler for resource management and allocation.
3. ApplicationMaster
ity

The ApplicationMaster is specific to each application framework and orchestrates

interactions with the ResourceManager. It collaborates with NodeManagers to handle
task execution and monitoring within the containers.
Hadoop follows a master-slave architecture. The Hadoop Distributed File System
(HDFS) serves as the distributed storage system for managing vast datasets. The
m

MapReduce framework handles distributed data processing within the Hadoop cluster.
YARN, or Yet Another Resource Negotiator, is responsible for efficiently allocating resources
among various applications in the cluster. The NameNode and ResourceManager daemons
)A

run on the master node, while DataNode and NodeManager daemons operate on slave
nodes. The coexistence of HDFS and the MapReduce framework on the same set of
servers ensures high aggregate bandwidth across the cluster.

4.1.4 Hadoop - Open Source Libraries

(c

Using straightforward programming concepts, Hadoop is an open-source platform

that enables the storage and processing of large amounts of data in a distributed setting

Amity Directorate of Distance & Online Education

 Internet of Things 143

across computer clusters. From a single server to thousands of devices, each providing
Notes

e
local computing and storage, it is intended to scale up.
Hadoop, an open-source framework developed by Apache in Java, facilitates the

in
distributed processing of extensive datasets across clusters of computers through simple
programming paradigms. The Hadoop framework functions within an environment that
provides both distributed computing and storage capabilities across computer clusters.
Designed to seamlessly scale from a single server to numerous workstations, Hadoop

nl
leverages the resources of individual machines for both computation and storage
purposes.
Large datasets can be stored on open-source platforms that house clusters of

O
related software. Hadoop is an Apache project that is utilised by a variety of users and
supported by a sizable community for code contributions. Licence 2.0 governs this work.
Hadoop that is available for free is ineffective since it costs more to operate and maintain
than it does to deploy.

ity
Hadoop Architecture
Hadoop is composed of two main levels, namely
●● MapReduce computation layer, and
●● Hadoop Distributed File System’s storage layer.

MapReduce rs
ve
Google pioneered the parallel programming paradigm called MapReduce to
effectively handle extensive data processing tasks involving multi-terabyte datasets.
This approach operates seamlessly on expansive clusters composed of thousands of
commodity hardware nodes, offering both reliability and fault tolerance. Apache’s open-
ni

source platform, Hadoop, is built upon the principles of MapReduce, empowering its
implementation and execution.

Hadoop Distributed File System

U

Derived from the design of Google File System (GFS), the Hadoop Distributed
File System (HDFS) is a distributed file system built to operate effectively on standard
hardware. While it bears resemblances to existing distributed file systems, it sets itself
apart through several notable differentiators. Specifically engineered for deployment on
ity

cost-effective hardware, HDFS boasts exceptional fault tolerance. Its capabilities extend
to accommodating applications dealing with immense datasets while delivering efficient
access to application data with high throughput.
The following two modules are also part of the Hadoop framework, in addition to the
m

aforementioned two basic components:

™™ Hadoop Common: Other Hadoop modules depend on these Java libraries and
tools.
)A

™™ Hadoop YARN: It is a framework for managing cluster resources and scheduling

jobs.

How Does Hadoop Work?

Creating more powerful servers with intricate setups to manage extensive
(c

processing demands can be financially burdensome. However, an alternative approach

involves linking numerous cost-effective single-CPU machines to establish an efficient

Amity Directorate of Distance & Online Education

 144 Internet of Things

distributed system. In practical terms, these clustered units can collectively process
Notes

e
the dataset concurrently, resulting in significantly improved throughput. Notably, this
approach proves to be more cost-effective than investing in a single high-end server. This
cost-efficiency stands as a central motivation for the adoption of Hadoop, which thrives

in
on the utilisation of economical and interconnected server clusters.
Hadoop executes programmes on a group of computers. the core operations that
Hadoop carries out are as follows:

nl
™™ Files and folders are used to organise data at first. Files are separated into
uniform chunks of 128M and 64M size (with 128M being preferred).

O
™™ These files are divided across different cluster nodes for additional processing.
™™ HDFS manages the processing because it sits on top of the local file system.
™™ Replicated blocks are used to handle hardware failure.
™™ Verifying the code’s successful execution.

ity
™™ Carrying out the sort that comes after the map and before the decrease phases.
™™ Delivering the data after sorting to a certain computer.
™™ Keeping track of every job’s debugging logs.

Advantages of Hadoop
●●
rs
The Hadoop framework enables users to create and test distributed systems
quickly. It uses the underlying parallelism of the CPU cores to divide the data and
ve
work efficiently and automatically across the machines.
●● Hadoop’s fault-tolerance and high availability (FTHA) capabilities are not dependent
on hardware; rather, the Hadoop library was built with the capacity to recognise and
address application-layer problems.
ni

●● Servers can be dynamically added to or deleted from the cluster and Hadoop keeps
running uninterrupted.
●● Another significant benefit of Hadoop is that it is Java-based and, in addition to
U

being open source, works on all platforms.

4.1.5 Apache Hadoop

Apache Hadoop stands as an open-source software platform, designed to handle
ity

the storage and analysis of extensive data volumes on clusters of commodity hardware.
This high-level project under the Apache umbrella is nurtured by a global community
of contributors and users, functioning within the framework of the Apache License 2.0.
Essentially, Apache Hadoop furnishes an open platform to manage and process large
m

datasets, ranging from gigabytes to petabytes, in a distributed manner. Rather than

relying on a single monolithic machine, Hadoop leverages the power of interconnected
computers to parallelise the processing of massive datasets, achieving enhanced speed
and efficiency.
)A

Comprising four key modules, Apache Hadoop offers a comprehensive solution:

●● Hadoop Distributed File System (HDFS): A distributed file system tailored to run on
common or cost-effective hardware. HDFS outperforms traditional file systems in
terms of data performance, while also boasting fault tolerance and native support for
(c

large datasets.
●● Yet Another Resource Negotiator (YARN): This component is responsible for the

Amity Directorate of Distance & Online Education

 Internet of Things 145

management and tracking of cluster nodes and resource consumption. It efficiently

Notes

e
organises jobs and tasks.
●● MapReduce: An integral framework designed to facilitate parallel processing of data.

in
The map task transforms input data into manageable key-value pairs, which are
subsequently processed by reduce tasks to generate the desired outcome.
●● Hadoop Common: Serving as a repository of Java libraries shared among all the
modules.

nl
4.2 Big Data Analytics: Spark

O
The Apache Software Foundation proposed Spark to accelerate the Hadoop
computational computing software process. Spark incorporates cluster management, but
Hadoop is merely one of several ways to implement Spark. Spark makes use of Hadoop
in two ways. The first is storage and the second is processing. As a result, Spark contains

ity
compute for cluster management while using Hadoop only for storage.
Apache Spark is an open-source distributed processing system. It is used for ‘Big
data’ tasks. For fast queries across any scale of data, Spark uses optimised query
execution and in-memory caching. It is essentially a broad and fast engine for large-scale
data processing. It is much faster to implement with Big Data than earlier concepts such

faster processing than disc drivers.

rs
as traditional MapReduce. Spark is faster since it runs on RAM/memory and allows for

Spark is simple since it can be used for multiple tasks such as working with data
ve
streams or graphs, Machine Learning techniques, ingesting data into databases,
establishing data pipelines, performing distributed SQL and so on.
Spark is one of Hadoop’s most important sub-projects. Matei Zaharia created it in the
APMLab at UC Berkeley in 2009. It became open-source in 2010 under the BSD licence.
ni

Spark was donated to the Apache Software Foundation in 2013. Since February 2014,
Apache Spark has been a top-level project of Apache.
U

4.2.1 Spark Introduction

Apache Spark stands as a widely embraced data processing technology, renowned
for its prowess in facilitating parallel programming across entire clusters while introducing
ity

the concept of Resilient Distributed Datasets (RDDs). Its popularity spans both the
academic community and the industrial landscape, with dedicated research and
development efforts directed towards harnessing its capabilities for big data analytics.
One of Apache Spark’s notable achievements is its ability to handle real-time streams
with minimal latency. This technology has found application in diverse fields, such
m

as constructing a more efficient pipeline for functional magnetic resonance imaging

(fMRI) using PySpark—a data analysis and pipeline language that exposes Spark’s
programming model to Python. This newly designed pipeline showcases a fourfold
reduction in processing time when compared to its Python counterpart. It achieves this by
)A

parallelising in-memory data processing, converting data to Resilient Distributed Datasets

and storing results in various formats.
A comparative analysis between Apache Spark and Apache Flink delves into the
machine learning libraries for batch data processing within these frameworks, leveraging
(c

techniques like Support Vector Machines and Linear Regression. This investigation
revolves around assessing the utility and convenience of employing distributed data
flow-oriented platforms such as Apache Hadoop MapReduce, Apache Spark and

Amity Directorate of Distance & Online Education

 146 Internet of Things

Apache Flink. Notably, Spark emerges as a robust platform for processing vast datasets,
Notes

e
embracing a hybrid structure that accommodates both batch and stream processing.
While sharing certain concepts with Hadoop’s MapReduce engine, Spark outperforms
it significantly in terms of performance. Consequently, Spark can be employed either

in
independently or in conjunction with Hadoop, offering an efficient alternative to the
MapReduce engine.
1) Spark Batch Processing Model

nl
The most notable advantage of Spark over MapReduce lies in its in-memory
computation capability. Spark’s interaction with disk storage is limited to loading data
into memory and saving the final results. Intermediate results are stored entirely in

O
memory. This in-memory processing characteristic of Spark sets it apart from its
batch processing competitor, Hadoop. Furthermore, Spark’s extensive optimisation
contributes to its remarkable speed by enabling a comprehensive analysis of the entire
workflow beforehand. This is achieved through the creation of Directed Acyclic Graphs

ity
(DAGs), which provide a representation of all tasks, data and their interdependencies.
To support its in-memory computing prowess, Spark employs Resilient Distributed
Datasets (RDDs). RDDs are read-only data structures that reside in memory, allowing
Spark to maintain fault tolerance without the need to write to disk after every operation.

rs
2) Spark Stream Processing Model
Apart from its prowess in batch processing, Spark also extends its capabilities to
encompass stream processing by employing micro-batches. The concept of micro-
batching involves treating data streams as minuscule collections of batches, each of
ve
which is processed as a conventional task within the Spark batch engine. While this
micro-batching technique proves to be efficient, it’s worth noting that it might introduce
certain variations in performance when compared to dedicated stream processing
frameworks.
ni

Features
Apache Spark boasts a range of distinctive attributes, some of which include:
U

●● Swiftness: When compared to Apache Hadoop, Apache Spark exhibits remarkable

speed, being approximately 100 times faster. Its efficiency also surpasses traditional
disk access by a factor of ten. This performance enhancement is achieved through
the Resilient Distributed Dataset (RDD) concept, which facilitates precise in-memory
ity

data storage.
●● User-Friendliness: Spark empowers users to swiftly develop applications using
various programming languages, such as Java, Scala, R and Python. This flexibility
allows developers to work in languages they are well-versed in, simplifying the
m

creation of parallel applications.

●● Advanced Analytics: In addition to basic map and reduce operations, Spark
facilitates advanced analytics including SQL queries, data streaming and complex
tasks like machine learning and graph algorithms.
)A

●● Versatile Deployment: Apache Spark can be deployed on diverse platforms like

Apache Hadoop YARN, Mesos, EC2, Kubernetes and various cloud environments
via the standalone cluster mode. It can also extract data from sources like HDFS,
Cassandra, HBase and other databases.
(c

●● In-Memory Computing: The in-memory cluster computing of Spark empowers rapid

execution of iterative machine learning methods, bidirectional queries and real-time
data streaming analysis. Data is stored in the server’s RAM, ensuring swift retrieval.
Amity Directorate of Distance & Online Education
 Internet of Things 147

●● Real-Time Stream Processing: Spark streaming not only supports real-time stream
Notes

e
processing but also various alternative setups. This design makes spark streaming
straightforward, resilient and unidirectional.

in
Use Cases of Apache Spark
●● Healthcare: Spark is employed in the healthcare industry since it allows for a full
study of patient records as well as past medical data. This assists in identifying

nl
which patients are likely to experience health issues in the near future, hence
avoiding hospital re-admission and lowering costs for hospitals and patients alike,
as it is now possible to deploy home care for the identified patient. Additionally,

O
Spark is employed in genomic sequencing because it may minimise the processing
time required to analyse genome data, which previously would take several weeks
to organise all of the chemical compounds with genes. MyFitnessPal is a firm that
makes use of Spark.

ity
●● Finance: Apache Spark provides insights that aid in making sound decisions
in areas such as customer segmentation, credit risk assessment and targeted
advertising. Financial organisations frequently employ big data to determine
the exact time and place of the fraud so that it can be halted. Various models for
detecting bogus transactions are already available, with the majority of them

●● rs
deployed in a batch environment. Financial organisations may detect false
transactions in real-time using Apache Spark on Hadoop and past fraud footprints.
E-commerce: Spark is used in the e-commerce business to find information about
ve
real-time transactions that are then fed to a streaming clustering algorithm such as
the K means clustering algorithm or the alternating least squares method. It also
improves client recommendations based on current patterns. Spark is used in
e-commerce by organisations such as Alibaba and eBay.
ni

●● Entertainment: In the gaming sector, Apache Spark aids in the recognition of

patterns from real-time in-game events and then responding to them to generate
profitable business opportunities such as selective advertising, player retention, or
the automatic change of gameplay levels based on difficulty. Spark and MongoDB
U

are widely employed in video sharing services such as Pinterest, Netflix and Yahoo.
These websites display relevant adverts to its visitors depending on the videos they
have viewed, shared and explored.
ity

4.2.2 Spark Ecosystem -1

m
)A
(c

[Link]
Figure: Spark ecosystem

Amity Directorate of Distance & Online Education

 148 Internet of Things

The Spark ecosystem revolves around Apache Spark, an open-source, distributed

Notes

e
computing framework designed for processing large datasets efficiently and quickly. At its
core, Spark provides a unified platform for various data processing tasks, including batch
processing, real-time streaming, machine learning and graph processing. The ecosystem

in
comprises several key components and tools that enhance Spark’s capabilities.
1. Apache Spark Core: At the heart of the ecosystem is the Apache Spark core, which
provides the foundation for distributed data processing. It includes the Resilient

nl
Distributed Dataset (RDD) abstraction, enabling efficient data manipulation across
clusters.
2. Spark SQL: This component allows users to perform SQL-like queries on structured

O
data, bridging the gap between traditional relational databases and distributed big
data processing. It supports various data formats and sources, making it easier to
integrate with existing data infrastructure.

ity
3. Spark Streaming: Designed for real-time processing, Spark Streaming enables the
ingestion and processing of data streams in mini-batches. This makes it suitable for
applications like monitoring social media feeds, real-time analytics and fraud detection.
4. MLlib (Machine Learning Library): MLlib equips Spark with a rich set of machine
learning algorithms and tools. This enables data scientists and engineers to develop

5.
faster training and prediction.
rs
and deploy machine learning models at scale, making use of distributed computing for

GraphX: Spark’s GraphX module facilitates graph processing and analytics, which
ve
is crucial for applications like social network analysis, recommendation systems and
network optimisation . It provides a graph computation API and optimised graph
algorithms.
6. SparkR: SparkR serves as an interface between Spark and the R programming
ni

language, enabling R users to leverage Spark’s distributed computing capabilities for

large-scale data analysis and machine learning tasks.
7. External Libraries and Integrations: The Spark ecosystem extends beyond its core
U

components. External libraries like Spark Streaming, Spark MLlib (for machine
learning) and Spark GraphX (for graph processing) offer specialised functionalities.
Additionally, Spark integrates with other data processing frameworks like Hadoop,
allowing users to leverage existing data sources and infrastructure.
ity

8. Cluster Managers: While not a direct part of the Spark ecosystem, it’s important to
note that Spark can be deployed on various cluster managers like Apache Hadoop
YARN, Apache Mesos and Kubernetes. These managers handle resource allocation
and scheduling of Spark applications on clusters.
m

9. Broad Applicability: The Spark ecosystem caters to a wide range of data processing
needs, from batch processing and real-time analytics to machine learning and graph
analysis. Its versatility and unified approach make it appealing to businesses and
)A

researchers dealing with diverse data challenges.

Machine Learning
The machine learning capabilities within Spark are split into two core components:
MLlib and KeystoneML.
(c

1. MLlib
Serving as the primary machine learning library, MLlib encompasses a wide range

Amity Directorate of Distance & Online Education

 Internet of Things 149

of algorithms and specialised data structures. It offers machine learning solutions for
Notes

e
tasks like clustering, regression, classification and collaborative filtering. Within MLlib,
there are also sophisticated data structures such as sparse and dense matrices and
vectors. Additionally, supervised learning structures are available that function akin

in
to vectors but encapsulate the dataset’s attributes based on their labels. This design
simplifies the process of inputting data into machine learning algorithms, negating the
necessity for extensive code to dictate data organisation within the algorithm.

nl
2. KeystoneML
Drawing an analogy from the efficient oil pipelines, KeystoneML is engineered to
facilitate the assembly of machine learning pipelines. These pipelines play a pivotal

O
role in multiple facets of the machine learning process. They aid in data preparation
for model training, iterative model construction and testing and the optimisation of
model parameters to achieve peak performance and capabilities.

ity
Text Processing
Numerous highly capable natural language processing (NLP) technologies
are available; however, their comprehensive implementation lags behind the more
numerically focused machine learning methods. MLlib encompasses embedded

rs
functionalities tailored to models like Latent Dirichlet Allocation topic modelling, Naive
Bayes modelling and Term Frequency-Inverse Document Frequency feature engineering.
Leveraging the power of the Spark engine, these functions can seamlessly process
ve
extensive datasets spanning millions of elements in parallel, eliminating the need for
additional code. This holds true for various parallelisable NLP processes like stopword
removal, stemming/lemmatisation , or term frequency filtering.

Graph Analytics
ni

Spark introduces a library known as GraphX, which is presently exclusively

interactive when used within the Scala programming language. GraphX is a top-tier
network graph analytics engine and data object repository that excels in executing
U

numerous prevalent graph analytics tasks. These tasks encompass activities such as
clustering, categorisation , traversal, searching and pathfinding.

Real-Time Stream Processing

ity

Leveraging Spark’s rapid utilisation of pooled server memory, new data can be
integrated into a batch almost in real-time, introducing a near-real-time aspect to the
operations discussed so far, which have predominantly revolved around batch processing
with static data. Termed micro-batching, this approach serves as Spark’s mechanism for
m

processing real-time data streams. Moreover, the engine guarantees end-to-end fault
tolerance to prevent data loss within the pipeline during processing. It also ensures an
exactly-once guarantee, ensuring that a specific portion of data in the stream is never
processed more than once.
)A

Apache Spark attains its high efficacy for both extensive and limited data processing
tasks not by reinventing the wheel, but by enhancing existing tools essential for efficient
analysis. By coupling Spark’s exceptional scalability on commodity hardware and
its remarkable performance advantages when compared to other widely recognised
(c

Big Data processing engines, the software holds the potential to revolutionise data
consumption on a global scale.

Amity Directorate of Distance & Online Education

 150 Internet of Things

4.2.3 Spark Ecosystem -2

Notes

e
In the ever-evolving landscape of big data, the Spark ecosystem has emerged as
a versatile and powerful framework, revolutionising how large-scale data processing

in
and analytics are conducted. In this continuation, we delve deeper into the various
components and tools that constitute the Spark ecosystem, showcasing its remarkable
capabilities.

nl
1. Spark Streaming: Real-Time Data Processing
Spark Streaming addresses the demand for real-time data processing. Traditional
approaches involve collecting and processing data in discrete batches, but Spark

O
Streaming revolutionises this by enabling continuous data ingestion and processing. It
divides incoming data streams into micro-batches and processes them using Spark’s
core processing engine. This approach ensures high throughput and low latency,
making it suitable for applications such as monitoring social media sentiment, real-

ity
time fraud detection and live dashboard updates.
2. MLlib (Machine Learning Library): Democratising Machine Learning
Machine learning (ML) is at the heart of data-driven decision-making and Spark’s
MLlib democratises the use of machine learning at scale. MLlib offers a rich set of

rs
algorithms and tools that cover a wide spectrum of tasks, from classification and
regression to clustering and collaborative filtering. By leveraging Spark’s distributed
computing architecture, MLlib accelerates model training and prediction on large
datasets. This empowers data scientists and engineers to build sophisticated models
ve
for various applications, including recommendation systems, predictive maintenance
and sentiment analysis.
3. GraphX: Tapping into Graph Analytics
ni

Graph analytics has gained significance with the rise of social networks,
recommendation engines and other interconnected data structures. Spark’s GraphX
module provides a unified platform for graph processing and analytics. It introduces the
Resilient Distributed Property Graph (RDGP) abstraction, allowing users to perform
U

operations on graphs with billions of vertices and edges. GraphX also offers a range of
optimised graph algorithms, such as PageRank and connected components, enabling
users to gain insights from complex network structures.
ity

4. SparkR: Bridging Spark and R

Data analysts and scientists often rely on R for statistical analysis and data visualisation.
SparkR bridges the gap between R and Spark, enabling users to leverage Spark’s
distributed computing capabilities while staying within the familiar R environment.
This integration enhances the scalability of R-based analyses, making it feasible to
m

work with larger datasets and complex computations. SparkR is particularly useful for
researchers who are proficient in R and wish to harness the power of Spark for their
analyses.
)A

5. External Libraries and Integrations: Expanding Capabilities

The Spark ecosystem extends beyond its core components, with various external
libraries and integrations contributing to its richness. Spark Streaming, mentioned
earlier, is an essential external library designed explicitly for real-time data processing.
Additionally, Spark SQL enables seamless interaction with structured data, offering
(c

support for various data formats, external data sources and query optimisation
techniques. The Spark ecosystem’s integration with Hadoop, Apache Hive and other

Amity Directorate of Distance & Online Education

 Internet of Things 151

data processing frameworks allows users to leverage their existing investments in

Notes

e
data infrastructure.
6. Cluster Managers: Efficient Resource Utilisation

in
While Spark provides a powerful framework for distributed computing, efficient
resource utilisation is vital. Spark can be deployed on cluster managers such as
Apache Hadoop YARN, Apache Mesos and Kubernetes. These cluster managers

nl
handle resource allocation, scheduling and monitoring of Spark applications across
clusters. This integration ensures that Spark applications run smoothly and coexist
with other processes on the same cluster, optimising resource utilisation and overall
performance.

O
7. Broad Applicability: A Unified Solution
The beauty of the Spark ecosystem lies in its broad applicability. It serves diverse data
processing needs, from handling massive batch processing tasks to real-time data

ity
analysis and machine learning. This unified platform eliminates the need for separate
tools for different tasks, reducing complexity and integration challenges. Organisation
s can streamline their data processing workflows, enabling data engineers, analysts
and scientists to collaborate effectively on a single platform.

4.3 Introduction of RDD

rs
RDD (Resilient Distributed Dataset) is the fundamental data structure of Apache
ve
Spark, consisting of an immutable collection of objects that compute on each node of the
cluster. Each dataset in Spark RDD is logically partitioned among several servers so that
it can be computed on different cluster nodes. It is Apache Spark’s primary data structure.
In Apache Spark, an RDD is an immutable collection of objects that compute on each
ni

node of the cluster.

RDD is deconstructed as follows:
●● Resilient, i.e., fault-tolerant using RDD lineage graph (DAG) and hence capable of
U

recompiling missing or damaged partitions owing to node failures.

●● Distributed, since Data resides on multiple nodes.
●● Dataset represents recordings of the data with which you work. The data set can be
ity

loaded externally as a JSON file, CSV file, text file, or database via JDBC with no
required data structure.
As a result, each dataset in RDD is logically partitioned among several servers so
that it can be computed on different cluster nodes. RDDs are fault tolerant, which means
they can self-recover in the event of a failure.
m

In Spark, there are three ways to construct RDDs: data in stable storage, other
RDDs and parallelising an already existing collection in the driver programme. Spark
RDDs can also be used in tandem with a low-level API that provides transformations
)A

and actions. Spark RDD can also be cached and partitioned manually. Caching is useful
when we use RDD several times. Manual partitioning is also required to properly balance
partitions. Smaller partitions, in general, allow more executors to distribute RDD data
more evenly. As a result, fewer partitions make the job easier.
(c

Programmers can also use the persist method to specify which RDDs should be
reused in future operations. Spark retains persistent RDDs in memory by default, but if
there is insufficient RAM, it can spill them to disc. Through flags to persist, users can

Amity Directorate of Distance & Online Education

 152 Internet of Things

also request additional persistence strategies, such as saving the RDD only on disc or
Notes

e
replicating data between computers.

4.3.1 RDD Fundamentals -1

in
Resilient Distributed Datasets, often known as RDDs, are a cornerstone of the
Apache Spark framework, which is used to process big data sets in a distributed

nl
computing setting. Distributed data processing can be done effectively and fault-tolerantly
with the help of RDDs.
The fundamental units of Apache Spark’s data processing are RDDs. They stand for

O
a distributed, immutable collection of data elements that may be processed concurrently
by a cluster of computers. RDDs are essential to Spark’s processing architecture
because they possess a number of crucial qualities.
1. Immutable and Partitioned: RDDs are read-only and cannot be modified once created.

ity
They are conceptually divided among the cluster, enabling Spark to efficiently distribute
the data and compute.
2. Resilience: RDDs’ fault-tolerance is referred to by the word “resilient” in this sentence.
In the cluster, RDDs automatically recover from node failures. Spark can recompute a

rs
partition of an RDD if it is lost due to a node failure by leveraging lineage information
(knowledge about the data transformations).
3. Lazy Evaluation: RDDs enable lazy evaluation, which delays the execution of
ve
transformations on RDDs. Instead, transformations are only carried out when an
action is carried out and are stored in a directed acyclic graph (DAG). Spark is able to
optimise the execution plan thanks to this.
4. Transformation and Action: There are two different sorts of RDD operations:
ni

transformations and actions. A new RDD is created from an existing one by

transformations (such as “map,” “filter,” and “reduce”), whilst computation is started by
actions (such as “count,” “collect,” and “saveAsTextFile”), which then return the results
U

to the driver programme or write data to an external system.

5. Data Lineage: RDDs save details regarding the order in which transformations were
applied to their parent RDDs. For fault tolerance, this lineage information is essential
since it enables Spark to rebuild lost data.
ity

Higher-level abstractions like DataFrames and Datasets in Spark are built on top
of RDDs. For specific kinds of processing jobs and optimisations, they offer a level of
control and fine-grained manipulation over data.
Let us use a straightforward example to demonstrate the basics of RDD:
m

Assume that we have a sizable text file with data about customer orders. The
following operations are what we wish to carry out with RDD transformations and actions:
)A

1. Create an RDD of lines after reading the text file.

2. Remove lines that are missing the phrase “electronics.”
3. To obtain the order total, map each line.
4. Determine the total of each order.
(c

Each step in this scenario transforms or takes an action on the RDD (‘map’, ‘filter’, or
‘reduce’) and Spark optimises the execution plan based on lineage information.

Amity Directorate of Distance & Online Education

 Internet of Things 153

Remember that RDDs are the cornerstone of Spark’s data processing and that using
Notes

e
the Spark framework to handle massive amounts of data in a distributed setting requires
a solid understanding of RDDs.

in
4.3.2 RDD Fundamentals -2
RDDs (Resilient Distributed Datasets) are a key abstraction used in Apache Spark
that enables efficient distributed data processing over a cluster of computers. RDDs offer

nl
a structured and fault-tolerant mechanism to describe and modify data, allowing for the
concurrent execution of complicated tasks. Let us explore the principles of RDD in more
detail by building on the prior concepts:

O
RDDs are partitioned into more manageable logical pieces known as partitions. A
portion of the data is stored in each partition. The ability to process several partitions
simultaneously on various nodes thanks to partitioning. The type of data and the

ity
modifications used determine the partitioning scheme. RDDs guarantee effective cluster
resource utilisation and faster processing speed by partitioning data.
RDD transformations: RDD transformations are the many operations that are used
to convert an existing RDD into a new one. Transformations are carried out lazily, which
means they are not immediately carried out after being invoked. As an alternative, they

rs
develop a lineage of dependencies that explain how the new RDD is related to its parent
RDD(s). ‘map’, ‘filter’, ‘reduceByKey’ and other commonly used transformations are listed
here. With the aid of transformations, you can adjust and computations to RDD data
ve
without carrying out the real computations or modifications until an action is triggered.
Actions: RDD actions are operations that start transformations, return outcomes,
or have unintended consequences. Actions, as opposed to transformations,
eagerly examine and start the processing of input. ‘Count’, ‘Collect’, ‘Reduce’ and
ni

‘saveAsTextFile’ are a few examples of activities. Actions make it possible to retrieve

computed outcomes from the RDD, including final datasets or aggregated values.
Lineage and Fault Tolerance: RDDs keep track of the series of transformations that
U

were used to build the RDD in the form of a lineage graph. In the event of node failure,
this lineage information enables Spark to retrieve lost data. When a node crashes and a
partition is lost, Spark can recreate the partition using the lineage data and the original
data source. Even in the face of failures, this fault tolerance system guarantees data
ity

dependability and consistency.

Caching: For quicker access, RDDs can be cached in memory. When several
operations must be carried out on a single RDD or when iterative methods are used,
caching is especially helpful. You can significantly enhance performance by avoiding
m

having to repeatedly compute transformations by caching intermediate or frequently used

RDDs.
Persistence: The method for caching RDDs in memory or on disc is persistence.
)A

Different storage levels, including MEMORY_ONLY, MEMORY_AND_DISK and DISK_

ONLY, can be used to persist RDDs. Considering variables like memory availability and
performance requirements can help you choose the right storage level.
Partitioning, transformations, actions, lineage, fault tolerance, caching and
(c

persistence are covered in depth in RDD foundations - 2. Together, these ideas form
the basis for Spark’s distributed data processing, making it possible to handle massive
datasets in an effective and dependable manner. RDDs are an essential part of Spark’s

Amity Directorate of Distance & Online Education

 154 Internet of Things

capabilities since they enable programmers and data scientists to express intricate data
Notes

e
processing jobs succinctly and scalable.

Examples and Use Cases of RDD Fundamentals

in
Example 1: Word Count
The word count issue is a well-known illustration. You can build an RDD from a
lengthy text document in which each element corresponds to a line of text. Then, you

nl
may use transformations to divide each line into words, remove frequently occurring
terms (stop words) and determine the number of times each unique word appears.

O
Example 2: Log Analysis
Let us say you have a sizable collection of log files from different servers. The log
entries can be represented using an RDD. You may then filter and analyse the logs using
RDD transformations to find trends, mistakes, or abnormalities throughout the entire

ity
dataset.

Example 3: Sensor Data Processing

Imagine that you have an IoT sensor network that is gathering information on the
temperature, humidity and air quality in various areas.

rs
You can use transformations to compute averages, spot unexpected shifts, or
examine patterns over time by generating an RDD for the sensor data.
ve
Example 4: Social Media Analysis
Let us say you wish to research social media sentiment trends. An RDD can be
made to represent user comments or posts. To track changing attitudes over time or
across various user groups, transformations can be used to tokenise the text, identify
ni

sentiment and aggregate sentiment scores.

Example 5: Graph Processing

U

Social network analysis and recommendation systems often use graph algorithms
like PageRank or community discovery. Large-scale graph data can be represented and
processed using RDDs. Transformations can be used to navigate around the network,
determine node centrality, or find clusters there.
ity

Example 6: Image Processing

You can represent images as RDDs containing pixel data for use in image
processing activities. The application of transformations can be used to carry out feature
extraction, filters and image enhancement. You can process and analyse big collections
m

of photos concurrently using RDDs.

Use Case 1: Large-Scale Data Processing

)A

Processing huge datasets that will not fit in a single machine’s RAM is best done via
RDDs. RDDs provide for effective data processing and analysis by distributing the data
across a cluster of computers and processing it in parallel.

Use Case 2: Iterative Algorithms

(c

For iterative methods used in machine learning and graph processing, RDDs are
ideally suited. The RDD is transformed and its content is updated during each loop.
During iterative computations, fault tolerance is ensured via RDD lineage.

Amity Directorate of Distance & Online Education

 Internet of Things 155

Use Case 3: Data Cleaning and Transformation

Notes

e
Unorganised, dirty data can be cleaned up and transformed using RDDs. To create
clean datasets for additional research, you can remove faulty records, change data

in
formats and aggregate data.

Use Case 4: Processing Data in Real-Time

Real-time data streams can be handled using RDDs. You may gain insights and

nl
make decisions in real time by continuously processing incoming data and performing
transformations.
The principles of RDD serve as the building blocks of Apache Spark’s distributed

O
data processing. They are excellent for a variety of data processing, analysis and
machine learning activities because they provide effective, fault-tolerant and parallel data
transformations.

ity
Use Case 5: Batch Data ETL (Extract, Transform, Load)
For ETL procedures, where data is extracted from many sources, changed to suit
your needs and then loaded into the destination system, RDDs are useful. Prior to
importing data into databases or data warehouses, RDD transformations can help clean,

rs
enrich and reshape it.

Use Case 6: Analysing Streaming Data

RDDs can be used in circumstances involving stream processing. Real-time
ve
analytics, aggregations and pattern recognition can be carried out by continuously
consuming and processing streaming data. RDD transforms make it possible to process
and update data as it comes in.
ni

Use Case 7: Natural Language Processing (NLP)

Textual data must be processed and analysed in order to complete NLP tasks like
text classification and language translation. Text documents can be represented and
U

processed using RDDs. Text can be tokenised, stop words can be eliminated and word
frequencies can be calculated.

Use Case 8: Financial Analysis

ity

Working with huge amounts of time-series data is frequently required for financial
data analysis. Through a variety of transformations, RDDs can assist in processing
stock market data, computing moving averages, spotting trends and performing risk
assessments.
m

Use Case 9: Distributed Machine Learning

An essential component of distributed machine learning frameworks is RDDs.
Iterative calculations over huge datasets are used in algorithms like gradient descent
)A

and decision tree training. Due to RDDs’ ability to parallelise these operations, distributed
machine learning is now practical.

Use Case 10: Geospatial Analysis

RDDs can represent spatial data points for geographic data processing.
(c

Transformations can be used to analyse spatial relationships, calculate distances

between points and find clusters or patterns in the data.

Amity Directorate of Distance & Online Education

 156 Internet of Things

A flexible framework for a variety of data processing, analysis and transformation

Notes

e
operations is offered by RDD principles. The use cases and examples presented here
demonstrate how RDDs may be used and used in a variety of fields, including social
media analysis, image processing, stream processing and machine learning.

in
4.4 Spark Execution Model

nl
Runtime concepts like as driver, executor, task, job and stage are used in Spark
application execution. Understanding these notions is critical for building Spark
programmes that are both quick and resource efficient.

O
A Spark application corresponds to a singular driver process along with a collection
of executor processes that are dynamically distributed across the hosts within a cluster
during runtime.

ity
The driver process takes charge of supervising the overall job flow and tasks, being
present throughout the application’s execution. While frequently the driver process aligns
with the client process that initiated the task, on YARN, the driver can function within the
cluster. In interactive mode, the shell serves as the driver process.

rs
The executors are responsible for executing tasks and storing cached data. The
executor’s lifespan is influenced by the presence of dynamic allocation. Each executor
encompasses multiple task slots and can concurrently execute several tasks throughout
ve
its operational period.
ni
U
ity

[Link]

Invoking an action within a Spark application causes a job to be launched to

complete it. Spark analyses the dataset on which the action is based and devises an
m

execution strategy. The execution plan categorises the dataset transformations. A stage
is a group of jobs that each run the same code on a different subset of the data.
)A

4.4.1 Spark Overview

Apache Spark is an open-source, distributed computing platform made for quickly,
flexibly and fault-tolerantly processing and analysing massive amounts of data. For
batch processing, interactive queries, real-time analytics, machine learning and
graph processing, it offers a unified platform. In order to overcome the drawbacks of
(c

conventional MapReduce-based frameworks, Spark was created. It offers notable

performance gains and a more flexible programming style.

Amity Directorate of Distance & Online Education

 Internet of Things 157

Important Elements of Apache Spark

Notes

e
1. In-Memory Computing: Spark uses in-memory processing to keep temporary data
in memory rather than often accessing the disc, which speeds up data processing

in
operations.
2. Unified Computing Model: Spark provides a uniform API that supports a range of data
processing activities, such as batch processing, interactive queries, streaming and

nl
machine learning, making it appropriate for a variety of application cases.
3. Resilient Distributed Datasets (RDDs): RDDs are Spark’s primary abstraction. They
offer distributed data processing fault tolerance, parallelism and usability. RDDs can

O
be modified using a variety of methods without changing the original data because
they are immutable.
4. Lazy Evaluation: Spark uses lazy evaluation, which delays the execution of
transformations on RDDs. The execution plan is instead improved by Spark and

ity
transformations are only computed when an action is requested. Through this
optimisation, efficiency is increased and pointless computations are decreased.
5. Extensive Range of Library: Spark provides a wide selection of libraries for a variety
of workloads, including Spark Streaming for real-time data processing, MLlib for

rs
machine learning, GraphX for processing graphs and Spark SQL for structured data
processing.
6. Fault Tolerance: Spark uses lineage information to achieve fault tolerance. Spark can
ve
recompute a lost data partition based on the series of transformations that produced it.
7. Ease of Use: Spark offers user-friendly features and boasts APIs across various
programming languages, including Scala, Java, Python and R, ensuring accessibility for
a diverse range of developers. Additionally, it facilitates interactive shell environments
ni

for effortless exploration.

8. Scalability: Spark can distribute data and calculations easily across a cluster of
devices by scaling horizontally. It can be used with different cluster managers, such
U

as standalone mode and Apache Mesos and Apache Hadoop YARN.

4.4.2 Spark Execution Model

The Spark execution model lies at the heart of the Apache Spark framework, defining
ity

how computations are organised and executed in a distributed and efficient manner across
clusters. This model underpins Spark’s ability to process vast amounts of data while
maintaining performance and fault tolerance. Understanding the Spark execution model is
crucial for harnessing the full potential of Spark for various data processing tasks.
m

At its core, the Spark execution model operates around the concept of Resilient
Distributed Datasets (RDDs), which are immutable distributed collections of data. RDDs
serve as the fundamental building blocks of Spark computations. They enable parallel
processing by breaking data into partitions, which are then distributed across nodes in
)A

a cluster. RDDs are created through deterministic operations applied to data sources
or existing RDDs, ensuring that computations can be traced and recomputed in case of
node failures.

DAG and Transformations

(c

The execution of Spark tasks follows a Directed Acyclic Graph (DAG) structure,
known as the “lineage graph.” This graph represents the sequence of transformations

Amity Directorate of Distance & Online Education

 158 Internet of Things

applied to the original data to derive the final output. Transformations in Spark are
Notes

e
categorised into two types: narrow and wide transformations.
1. Narrow Transformations: Narrow transformations maintain a one-to-one mapping

in
between input and output partitions. Examples include filtering and mapping
operations. These transformations are executed within individual partitions, promoting
parallelism and efficiency.

nl
2. Wide Transformations: Wide transformations involve shuffling data across partitions,
such as grouping or joining operations. These transformations require data to be
redistributed and reorganised, often leading to a performance overhead due to data
movement.

O
Actions
Actions are operations that trigger the actual computation and result in data being
processed or returned to the driver program. Unlike transformations, actions materialise

ity
the data and might trigger a sequence of transformations leading up to the desired
output. Examples of actions include `count()`, `collect()` and `saveAsTextFile()`.

Lazy Evaluation

rs
A key feature of the Spark execution model is its lazy evaluation strategy.
Transformations are not immediately executed when called but are recorded in the
DAG. This allows Spark to optimise computations by fusing together compatible
transformations and minimising data movement. Only when an action is invoked does
ve
Spark evaluate the DAG and execute the necessary transformations in the most efficient
order.

Stages and Tasks

ni

The Spark execution model divides the computation process into stages. A stage is
a sequence of transformations that can be executed together in a single pass, without
requiring data shuffling. Stages are separated by wide transformations that necessitate
U

data exchange. Each stage is further divided into tasks, which are the smallest units of
work that can be executed independently on individual partitions of the data.

Data Locality and Task Scheduling

ity

Efficient data processing hinges on minimising data movement across the network.
Spark optimises this by employing data locality-aware task scheduling. When scheduling
tasks, Spark tries to place tasks on nodes where the data they need is already stored,
reducing the need for costly data transfers.
m

Fault Tolerance
Spark’s execution model also incorporates fault tolerance mechanisms. RDDs
enable lineage tracking, allowing Spark to recompute lost partitions by tracing the
)A

sequence of transformations that led to their creation. If a node fails, the lost tasks can be
rerun on other nodes using the lineage information, ensuring that computations proceed
without interruption.

4.4.3 Scala
(c

The acronym Scala stands for “Scalable Language,” a cutting-edge, statically typed
programming language that combines the concepts of both functional and object-oriented
programming (OOP). It was created to provide a clear and expressive syntax while
Amity Directorate of Distance & Online Education
 Internet of Things 159

getting around some of the drawbacks of conventional programming languages. Because

Notes

e
Scala is executed on the Java Virtual Machine (JVM), it can easily be integrated with
Java tools and libraries.

in
Scala’s Features and Advantage
Scala is a strong and adaptable programming language with a wide range of
capabilities, including:

nl
1. Concise Syntax: Scala’s expressive syntax enables you to write clear, readable code,
cutting down on boilerplate and enhancing code maintainability.
2. Object-Oriented and Functional: Scala integrates OOP and FP concepts in a seamless

O
manner, allowing programmers to create orderly and modular code. For encapsulation
and reuse, you can use objects and classes. For robust and scalable programmes,
you can also employ functional principles like immutability and higher-order functions.

ity
3. Type Inference: The type inference system in Scala makes use of automatic variable
type inference to reduce the requirement for explicit type annotations and improve
code readability.
4. Immutable Data Structures: Scala promotes the usage of immutable data structures,
fostering concurrent programming that is safer and more predictable.
5.
rs
Pattern Matching: With Scala’s robust pattern matching, you can match complicated
data structures and take different actions based on patterns, which makes code more
expressive and simpler to comprehend.
ve
6. Higher-Order Functions: Scala recognises functions as first-class citizens and permits
the creation of anonymous functions (lambdas), the passing of functions as arguments
and the return of functions from other functions.
ni

7. Concurrency: Scala offers tools for creating concurrent and distributed applications,
such as Akka, enabling you to properly handle concurrency.
8. Interoperability: Scala seamlessly interoperates with Java. Scala code can be included
U

into Java projects and Scala libraries and frameworks can be utilised straight from
Scala.

Scala’s Role in Data Processing (e.g., with Apache Spark)

ity

Scala is important for data processing, especially when used with Apache Spark:
1. Apache Spark Integration: Scala is natively supported by Apache Spark, a well-liked
large data processing tool. Scala’s functional programming capabilities and clear
syntax make it a good choice for creating Spark applications.
m

2. Scala’s Spark API: It enables developers to express complicated data processing

tasks using a high-level, expressive syntax. Scala’s Spark API is user-friendly.
3. Functional Transformations: To perform transformations on Resilient Distributed
)A

Datasets (RDDs) and DataFrames, Spark makes use of Scala’s functional programming
capabilities. This enables effective and distributed data processing.
4. Ease of Development: Scala’s compatibility with Java libraries and its capacity
to seamlessly integrate with Spark’s native libraries contribute to an effective and
streamlined development process.
(c

A versatile and potent programming language, Scala combines the functional

and object-oriented paradigms. It is a great option for data processing thanks to its

Amity Directorate of Distance & Online Education

 160 Internet of Things

functionality and interoperability with Java libraries, especially when combined with
Notes

e
frameworks like Apache Spark.

Setting up Scala Environment

in
●● Decide on an Integrated Development Environment (IDE) like IntelliJ IDEA, Eclipse
with a Scala plugin, or Visual Studio Code with Metals.
●● Download and install the Scala Build Tool (SBT) to create projects and manage

nl
dependencies.
●● Become familiar with Scala’s Read-Eval-Print Loop (REPL) for interactive coding
and experimentation.

O
Data Types and Basic Syntax
●● • Get to know the semicolons, comments and code blocks used in Scala’s syntax.
●● • Examine the basic types of data, such as Int, Long, Float, Double, Char, String,

ity
Boolean, etc.
●● • Recognise type inference, in which Scala infers a variable’s data type without
requiring a declaration.
●● • Use the ‘var’ and ‘val’ keywords to define variables and constants.

Control Flow (if-else, Loops)

●● rs
• For conditional branching, use ‘if-else’ statements.
ve
●● • Learn how to use ‘while’ loops, ‘for’ comprehensions and functional-style ‘foreach’,
among other loop constructions.

Functions and Methods

ni

Use the ‘def’ keyword to define functions.

●● • Be familiar with function call syntax, return types and parameters.
●● • Examine higher-order functions that return or accept parameters from other
U

functions.
Use the ‘=>’ syntax for creating anonymous functions (lambdas).
●● • To specifically return a value from a function, use the ‘return’ keyword.
ity

Object-Oriented Programming in Scala

1. Classes and Objects
™™ Use the ‘class’ keyword to define classes.
™™ Use the ‘new’ keyword to create instances of classes.
m

™™ Member variables and class-specific methods.

™™ Definitions of constructors (including primary and auxiliary constructors).
)A

™™ The function of companion objects in supplying static-like behaviour.

2. Inheritance
By using the ‘extends’ keyword, create a subclass.
™™ Replacing parent class methods.
(c

™™ Access modifiers for members (public, protected and private).

™™ Using ‘super’ to get around the diamond difficulty.

Amity Directorate of Distance & Online Education

 Internet of Things 161

Traits and Mixins

Notes

e
1. Traits
™™ Use the ‘trait’ keyword to define traits.

in
™™ Comparable to interfaces in other languages, but method implementations are
also possible.
™™ Use the ‘with’ keyword to combine traits into classes.

nl
2. Mixins
™™ Create complex classes by combining several traits.

O
™™ Use traits-based abstract procedures.
™™ Some features can be extended by other traits, facilitating composition.

Pattern Matching

ity
1. Pattern Matching Basics
™™ The syntax of the ‘match’ expression.
™™ Pattern matching across a range of data types (values, variables, case classes).
™™ Guards for patterns that add conditions.
2. Matching Case Classes

rs
™™ Breaking down case classes with the use of pattern matching.
ve
™™ Data extraction from constructor parameters for case classes.
™™ Using ‘@’ to bind variables.
3. Sealed Traits and Exhaustiveness Checking
™™ Building closed hierarchies using sealed traits.
ni

™™ Making use of sealed attributes to provide thorough pattern matching.

Case Classes and Objects

U

1. Case Classes
™™ Using the ‘case class’ keyword to declare case classes.
™™ Companion objects that are automatically created and have handy methods
ity

(apply, unapply, copy, equals, hashcode and toString).

™™ Suitable for representing data because it is immutable by default.
2. Case Objects
™™ Using the ‘case object’ keyword to declare case objects.
m

™™ Comparable to case classes but without constructor inputs.

Type Hierarchy and Polymorphism

)A

1. Type Hierarchy
™™ The Any, AnyVal and AnyRef types in Scala.
™™ The Nothing and Null types.
™™ Contravariance and covariance
(c

2. Polymorphism
™™ Making use of classes and characteristics with subtype polymorphism.

Amity Directorate of Distance & Online Education

 162 Internet of Things

™™ Dynamic dispatch and ways for overriding.

Notes

e
™™ Using ‘asInstanceOf’ and ‘isInstanceOf’ for type casting and testing.
3. Abstract Classes and Traits

in
™™ Using the ‘abstract’ keyword to define abstract classes.
™™ Abstract fields and procedures.
™™ Developing features to enable multiple inheritance.

nl
4. Type Parameterisation and Generics
™™ Using type parameters for defining generic classes and methods.

O
™™ Using type boundaries for generic types (upper and lower bounds).
The topics covered in this review of Scala’s Object-Oriented Programming include
classes, objects, inheritance, traits, mixins, pattern matching, case classes and objects,
type hierarchy and polymorphism.

ity
Integrating with Spark
Scala’s role in Apache Spark
Apache Spark’s main programming language, Scala, is essential to the structure and

rs
operation of the platform. It acts as the native language for developing Spark libraries
and building Spark applications. It also grants access to key Spark functions. Scala is a
strong option for data processing, analytics and machine learning applications inside the
Spark environment because to its expressive syntax, functional programming features
ve
and close integration with Spark. Fewer lines of code can be written using its syntax,
making the code simpler to read and maintain.
Spark applications are made more reliable and effective by Scala’s static typing
and sophisticated language capabilities. UDFs (User-Defined Functions) let Spark
ni

transformations perform unique operations on data. In well-known IDEs like IntelliJ IDEA
and Eclipse, Scala has great tooling support and its active community offers a wealth of
documentation, tutorials, forums and online resources for learning and troubleshooting.
U

Scala-based Spark Application Development

Utilising distributed data processing and analytics is made possible with the
integration of Scala and Apache Spark. We will go over how to develop Spark apps in
ity

Scala in this part.

1. Establishing the Spark Environment
i. Get and set up Apache Spark.
ii. Configure classpath and Spark home environment variables.
m

 Create a Spark Session (the starting point for Spark functionality) and
initialise it.
)A

2. Importing Necessary Libraries

™™ Import the required Spark libraries in your Scala application:
import [Link]._
import [Link]._
(c

3. Creating a SparkSession
™™ Create a SparkSession object using [Link]():

Amity Directorate of Distance & Online Education

 Internet of Things 163

val spark = [Link]()

Notes

e
.appName(“MySparkApp”)
.config(“[Link]”, “local”) // Use local mode for development

in
.getOrCreate()
4. Loading Data

nl
™™ Read data from various sources (CSV, JSON, Parquet, etc.) into a DataFrame:
val data = [Link](“path/to/your/[Link]”)
5. Transforming Data

O
™™ Apply transformations to the DataFrame using Spark’s built-in functions:
val transformedData = [Link](“column1”, “column2”).filter($”column1” > 10)
6. Performing Actions

ity
™™ Perform actions on the DataFrame to trigger computation and view results:
val count = [Link]()
[Link]()

rs
7. Writing Output
™™ Write the processed data to various output formats (Parquet, CSV, etc.):
[Link](“output/path”)
ve
8. Operation of the Application
™™ Utilise a build tool like SBT to compile and package your Scala application.
™™ Use the’spark-submit’ command to submit the application to Spark:
ni

spark-submit --class [Link] \

--master local[2] \
path/to/your/[Link]
U

9. Monitoring and Debugging

™™ Follow the application’s development in the Spark UI (the default address is
[Link]
ity

™™ Examine the logs and troubleshoot any problems that occur during operation.
Using Scala to build Spark applications offers a strong framework for distributed
data processing. To create reliable and effective data processing pipelines, you can
explore increasingly sophisticated Spark capabilities and optimisations as you acquire
m

experience.

Scala Spark Transformations and Actions

)A

The distributed and parallel processing of massive amounts of data is a strength

of Spark. The Resilient Distributed Dataset (RDD) API of Spark includes two essential
operations: transformations and actions. They give you the ability to create a flow of data
processing operations and run calculations on distributed datasets. These operations
enable data processing and analysis and serve as the foundation for Spark applications.
(c

Transformations
Transformations are actions that turn an existing RDD into a new one.

Amity Directorate of Distance & Online Education

 164 Internet of Things

Transformations are carried out lazily, which means they are not computed right away.
Notes

e
Instead, they combine to form a directed acyclic graph (DAG), which symbolises the order
in which the changes will be carried out. Only when an action is called are transformations
carried out and Spark optimises the execution strategy based on the DAG.

in
Here are a few typical Scala modifications for Spark:
1. map: Returns a new RDD after applying a function to each member of the RDD.

nl
val rdd: RDD[Int] = ...
val transformedRDD = [Link](x => x * 2)

O
2. filter: Returns a new RDD that only includes the elements that meet a specific
predicate.
val rdd: RDD[Int] = ...
val filteredRDD = [Link](x => x % 2 == 0)

ity
3. flatMap: Flattens the outcomes into a new RDD after applying a function to each
element that returns an iterator.
val rdd: RDD[String] = ...

rs
val wordsRDD = [Link](line => [Link](“ “))
4. groupByKey: Groups the elements by key and returns a pair RDD of (key, values).
val rdd: RDD[(String, Int)] = ...
ve
val groupedRDD = [Link]()
5. reduceByKey: Performs aggregation on values for each key and returns a pair RDD.
val rdd: RDD[(String, Int)] = ...
ni

val aggregatedRDD = [Link]((a, b) => a + b)

Actions
U

Actions are operations that start the computation of transformations, return a value
to the driver programme and either write data to an external storage system or write data
to an external storage system. The DAG’s actions, which compel the execution of every
transformation, are where the actual processing happens.
ity

Following are a few typical Spark operations in Scala:

1. collect: Gives the driver programme an array containing every element of the RDD.
val rdd: RDD[Int] = ...
m

val collectedArray = [Link]()

2. count: Returns the number of elements in the RDD.
val rdd: RDD[Int] = ...
)A

val count = [Link]()

3. reduce: Aggregates the elements of the RDD using a specified function.
val rdd: RDD[Int] = ...
(c

val sum = [Link]((a, b) => a + b)

4. saveAsTextFile: Writes the elements of the RDD to a text file in HDFS or a local file
system.
Amity Directorate of Distance & Online Education
 Internet of Things 165

val rdd: RDD[String] = ...

Notes

e
[Link](“[Link]”)
5. foreach: Applies a function to each element of the RDD, often used for side effects like

in
printing.
val rdd: RDD[String] = ...

nl
[Link](element => println(element))

Using Transformations and Actions

Understanding the delayed evaluation feature of transformations is crucial when

O
utilising transformations and actions in Spark. Through transformations, a DAG that
reflects the computing plan is built up. By carrying out the transformations in the right
order, actions start the actual calculation.

ity
When caching or persisting intermediate results using techniques like “persist()”
or “cache(),” it is critical to take the data lineage into account as well as the probable
requirement for recomputation.
The fundamental tools for processing and analysing data in Apache Spark in

rs
a distributed and parallel fashion are transformations and actions. Actions start the
execution of the logical execution plan that was created by transformations and the
results are returned to the driver programme. Building productive and scalable Spark
applications in Scala requires an understanding of how to use transformations and
ve
actions.

4.5 Kafka
ni

Apache Kafka, an open-source distributed streaming platform, empowers the

development of real-time, event-centric applications. It facilitates the generation of
data record streams, encompassing events, which chronicle past actions and their
U

occurrences. These events can encompass diverse activities, such as order placement,
seat selection, or form submission, triggering subsequent actions in a process. These
data streams enable the creation of real-time applications that adeptly respond to events,
permitting developers to craft applications that swiftly and accurately process these
ity

streams, prioritising their correct sequential order. LinkedIn initiated Kafka in 2011 as a
high-throughput message broker and it has since evolved into the most widely employed
streaming infrastructure. Its capabilities encompass the intake and processing of trillions
of entries daily, scaling seamlessly without noticeable performance setbacks. Renowned
corporations like Target, Microsoft, AirBnB and Netflix utilise Kafka to deliver data-driven,
m

real-time experiences to their clientele.

4.5.1 Kafka
)A

Apache Kafka serves as an open-source and distributed streaming platform,

empowering the creation of applications that operate in real-time and respond to events.
These events stem from a continuous flow of data records originating from billions of
data sources. Each event encapsulates a digital record of a past action and the precise
(c

moment it unfolded. Typically, an event triggers subsequent actions within a process. This
architecture lays the foundation for real-time applications capable of swift, high-speed
processing of these streams, driven by accurate sequencing.

Amity Directorate of Distance & Online Education

 166 Internet of Things

LinkedIn introduced Kafka in 2011 as a high-throughput message broker for its

Notes

e
internal needs. Since then, it has burgeoned into one of the most extensively employed
streaming infrastructures. Renowned entities, including Target, Microsoft, AirBnB and
Netflix, leverage Kafka to provide real-time and data-driven experiences to their clientele.

in
The platform’s global adoption is expanding rapidly, with over one-third of Fortune 500
companies embracing it. Kafka serves various purposes, ranging from real-time data
streaming, extensive data collection, to immediate analysis. Its applications encompass

nl
sustaining in-memory microservices, powering complex event streaming systems (CEP)
and automating IoT/IFTTT-style systems.

O
ity
rs
ve
[Link]
nt%2Fuploads%2F2019%2F03%[Link]&tbnid=BY6Oiso25HSlRM&vet=1&imgrefurl=https
%3A%2F%[Link]%2Fwhat-is-kafka%2F&docid=SjErKGLonS2e3M&w=1500&h=844&sourc
e=sh%2Fx%2Fim%2F0
ni

How does Kafka Work so Easily?

The seamlessness of Kafka’s operation can be attributed to its emphasis on
simplicity. Its effectiveness is evident not only in its setup and utilisation but also in its
U

commitment to stability, reliable durability and versatile built-in features like publishing,
subscribing and maintaining queues. These attributes are particularly crucial when
dealing with numerous client groups, ensuring robust replication and offering a consistent
approach such as Kafka topic splitting.
ity

An exceptional characteristic of Kafka is its seamless integration with data stream

systems, enabling the aggregation, transformation and loading of other storage systems
for enhanced usability. All of these attributes are underpinned by Kafka’s remarkable
performance, a trait that wouldn’t be achievable if the platform were sluggish.
m

Delving into the operational layer at the OS level further elucidates Kafka’s user-
friendly nature. Here’s how Kafka functions at this level:
●● It leverages the zero-copy concept and harnesses OS kernels to expedite data
)A

movement.
●● Kafka batches data records into chunks, which consumers can access from the file
system, known as the Kafka topic log.
●● Data batching facilitates efficient compression and reduces I/O latency.
(c

●● The platform’s horizontal scalability is facilitated by sharding, dividing a topic log into
numerous partitions. This design empowers Kafka to effortlessly handle extensive
workloads.
Amity Directorate of Distance & Online Education
 Internet of Things 167

What Can Be Achieved with Kafka?

Notes

e
Kafka becomes indispensable for businesses regularly dealing with substantial data
volumes, which is why it finds widespread adoption across numerous industries.

in
™™ LinkedIn employs Kafka for data tracking and operational metric monitoring.
™™ Twitter utilises Kafka to establish stream processing infrastructures.
This roster of companies extends to include prominent names like Uber, Spotify,

nl
Goldman Sachs and Cisco.

Benefits

O
Outlined below are the advantages associated with Kafka:
●● High Throughput: Kafka’s ability to efficiently manage massive data volumes
while handling high data inflow rates is a notable advantage. Few platforms can

ity
match Kafka’s capability to deliver message throughput at scales of thousands of
messages per second.
●● Low Latency: The substantial message generation necessitates low latency
processing, which Kafka effectively delivers.

rs
●● Fault Tolerance: Kafka offers fault tolerance, a crucial attribute, as its functioning
can be constrained by a single cluster node without system-wide breakdown.
●● Durability: Operational resilience is a hallmark of Kafka’s design, making it an
attractive choice for major multinational corporations. Kafka ensures that messages
ve
remain durable over the long run, guarding against data loss.

Essential Skills
Becoming a Kafka professional doesn’t demand any specific prerequisites, though
ni

certain domains and roles stand out:

●● Developers keen on entering the realm of Big Data Streams and aspiring to elevate
their careers.
U

●● Testing experts specialising in Queuing and Messaging systems find ample

opportunities in Kafka.
●● Architects play a pivotal role, considering that every architecture requires a
ity

framework that can be refined regularly. Learning Kafka is beneficial for Big Data
architects.
●● Project managers, in the presence of the aforementioned experts, facilitate resource
management and unlock higher-tier Kafka-related opportunities.
m

Advantages of Utilising Kafka

It is widely used for data tracking and manipulation according to business
requirements. It allows for real-time data streaming and real-time analytics. It is built
)A

for fault tolerance and is quick, scalable and durable. There are numerous use cases
available on the web that demonstrate why JMS, RabbitMQ and AMQP are not even
considered for use when massive volumes and responsiveness are required.
It offers a high throughput, dependable setup and replication capabilities, making it
(c

an excellent choice for working on IoT sensors.

Another incentive to utilise it is its compatibility, which has made it acceptable
internationally. It is simple to set up to operate with the applications listed below. This
Amity Directorate of Distance & Online Education
 168 Internet of Things

combination is critical for many businesses to develop and survive (since it saves time
Notes

e
and money).
™™ Flume

in
™™ Spark Streaming
™™ HBase
™™ Spark for real-time ingestion, processing and analysis of data.

nl
™™ It is used to feeding Hadoop BigData.

Kafka Broker

O
In the composition of a Kafka cluster, there are typically one or more servers,
referred to as Kafka brokers, that host the Kafka infrastructure. Acting as initiators,
producers are responsible for delivering data to Kafka topics within the designated
broker, essentially pushing messages via triggers. On the other hand, a topic consumer

ity
extracts messages from a Kafka topic.

rs
ve
[Link]
ni

Kafka Topic: A Kafka Topic can be likened to a named category or feed where
messages are stored and disseminated during operations. These messages primarily
U

consist of byte arrays capable of accommodating diverse objects in various formats. This
versatility of using byte arrays to store any object is one of Kafka’s standout features.
It’s worth emphasising that Kafka messages are systematically organised into topics. In
the process of sending a message, it’s directed to a specific topic and when reading a
ity

message, the source topic is equally specified.

Consumers and Consumer Groups: One remarkable aspect of Kafka lies in its
consumer and consumer group dynamics. Consumers possess the flexibility to access
messages from a designated offset and they can initiate reading from any point within the
offset range. This characteristic empowers consumers to seamlessly join the cluster at
m

any given time, resulting in streamlined operations and enhanced ease of use.
Partitions: Kafka employs the concept of partitions to enable efficient parallelisation
of data within a topic. This involves distributing data from a specific topic across multiple
)A

brokers.

Real time Applications

●● Twitter: On Twitter, unregistered users are limited to reading tweets, whereas
registered users have the privilege of both reading and posting tweets. The stream
(c

processing architecture of Twitter incorporates Storm-Kafka.

●● LinkedIn: Apache Kafka plays a pivotal role at LinkedIn, handling activity stream

Amity Directorate of Distance & Online Education

 Internet of Things 169

data and operational metrics. This messaging system is integral to various LinkedIn
Notes

e
products, including the likes of LinkedIn Newsfeed and LinkedIn Today for online
message consumption. Furthermore, Kafka is a foundational component of offline
analytical systems such as Hadoop.

in
●● Netflix: A global leader in on-demand Internet streaming media, Netflix employs
Kafka for real-time monitoring and event processing, bolstering its operational
efficiency.

nl
●● Box: Kafka finds a strategic application in Box’s operational landscape, serving as
a cornerstone in both the production analytics pipeline and real-time monitoring
architecture.

O
How Kafka works
Kafka encompasses three fundamental capabilities:
1. Facilitating applications to publish or subscribe to streams of data or events.

ity
2. Ensuring the precise and fault-tolerant storage of records in their original chronological
sequence.
3. Enabling the real-time processing of records as they unfold.

rs
Developers can harness these Kafka capabilities through four distinct APIs:
1. Producer API: This API empowers a program to publish data streams to Kafka topics.
A topic represents a named log that preserves records in their chronological order.
ve
Once a record is written to a topic, it remains unchangeable and intact, persisting for
a designated duration or until storage space is exhausted.
2. Consumer API: With this API, applications can subscribe to one or multiple topics
and consume and process the stored data stream. It facilitates real-time handling of
ni

records within topics or the ingestion and processing of historical data records.
3. Streams API: Extending the Producer and Consumer APIs, the Streams API provides
advanced processing capabilities. It enables applications to carry out continuous
U

end-to-end stream processing. This involves consuming records from various topics,
performing analysis, aggregation, or transformation operations and subsequently
publishing the processed streams to the same or different topics. While basic stream
processing can be achieved with the Producer and Consumer APIs, the Streams API
ity

facilitates the creation of more intricate data- and event-streaming applications.

4. Connector API: This API empowers developers to craft connectors, which are reusable
producers or consumers designed to simplify and automate the integration of data
sources into a Kafka cluster.
m

Kafka Performance
Kafka operates as a distributed platform, forming a fault-tolerant and highly available
cluster across numerous computers and even spanning multiple data centres. Kafka
)A

topics are partitioned and replicated to enable scalability for accommodating high
volumes of concurrent users without sacrificing performance. According to [Link],
Kafka exhibits consistent performance regardless of whether the server has 50 KB or 50
TB of persistent storage.
Kafka finds application in building two primary types of applications:
(c

1. Real-time Streaming Data Pipelines: These applications are purpose-built to facilitate

the seamless transfer of massive volumes of data or event records across enterprise

Amity Directorate of Distance & Online Education

 170 Internet of Things

systems. They operate at scale and in real-time, ensuring reliable and secure
Notes

e
movement of data without the typical pitfalls of data corruption or duplication that often
arise when dealing with such extensive data streams at high velocities.

in
2. Real-time Streaming Applications: These applications are fueled by data or event
streams and generate their own streams in response. In your daily online interactions,
you encounter numerous instances of these applications. They range from e-commerce
platforms that dynamically update product availability in local stores to websites that

nl
provide tailored recommendations or advertisements based on the analysis of user
clickstreams.

O
Summary
●● Certain researchers categorise the challenges of dealing with Big Data using
the concept of the “3Vs,” while others may extend this classification by adding
dimensions like Value and Validity.

ity
●● DFS stands for distributed file system, a concept that involves the distribution of files
across multiple nodes. This approach creates the illusion of a unified and extensive
system, whose storage capacity corresponds to the collective storage of all nodes
within a cluster.

rs
●● The purpose of building Hadoop is to create a low-cost, dependable and scalable
framework for storing and analysing the growing amount of large data. Hadoop uses
a master-slave design to effectively store and handle massive amounts of data. The
slave nodes are assigned responsibilities by the master nodes.
ve
●● DataNodes are slave daemons that store the actual data blocks of files. They are
responsible for serving read and write requests from clients and they periodically
report their health and block status to the NameNode.
●● YARN, which stands for Yet Another Resource Negotiator, serves as Hadoop’s
ni

resource management layer. It was introduced in Hadoop 2 with the purpose

of decoupling job scheduling and resource management functions into distinct
daemons.
U

●● One of Apache Spark’s notable achievements is its ability to handle real-time

streams with minimal latency. This technology has found application in diverse
fields, such as constructing a more efficient pipeline for functional magnetic
resonance imaging (fMRI) using PySpark—a data analysis and pipeline language
ity

that exposes Spark’s programming model to Python.

●● Apache Spark is an open-source distributed processing system. It is used for ‘Big
data’ tasks. For fast queries across any scale of data, Spark uses optimised query
execution and in-memory caching.
m

●● Spark is employed in the healthcare industry since it allows for a full study of patient
records as well as past medical data. This assists in identifying which patients are
likely to experience health issues in the near future, hence avoiding hospital re-
)A

admission and lowering costs for hospitals and patients alike, as it is now possible
to deploy home care for the identified patient.
●● While not a direct part of the Spark ecosystem, it’s important to note that Spark
can be deployed on various cluster managers like Apache Hadoop YARN, Apache
Mesos and Kubernetes. These managers handle resource allocation and scheduling
(c

of Spark applications on clusters.

●● The fundamental units of Apache Spark’s data processing are RDDs. They stand

Amity Directorate of Distance & Online Education

 Internet of Things 171

for a distributed, immutable collection of data elements that may be processed

Notes

e
concurrently by a cluster of computers. RDDs are essential to Spark’s processing
architecture because they possess a number of crucial qualities.

in
●● For quicker access, RDDs can be cached in memory. When several operations
must be carried out on a single RDD or when iterative methods are used, caching is
especially helpful.
●● The Spark execution model lies at the heart of the Apache Spark framework,

nl
defining how computations are organised and executed in a distributed and efficient
manner across clusters.
●● Actions are operations that start the computation of transformations, return a value

O
to the driver programme and either write data to an external storage system or write
data to an external storage system.
●● Scala is a strong option for data processing, analytics and machine learning

ity
applications inside the Spark environment because to its expressive syntax,
functional programming features and close integration with Spark.
●● An exceptional characteristic of Kafka is its seamless integration with data stream
systems, enabling the aggregation, transformation and loading of other storage
systems for enhanced usability.

Glossary
●● Integration: the action or process of integrating. rs
ve
●● Substantial: of considerable importance, size, or worth.
●● Metadata: a set of data that describes and gives information about other data.
●● Nodes: a point at which lines or pathways intersect or branch; a central or
connecting point.
ni

●● Deployment: the movement of troops or equipment to a place or position for military

action.
●● Cluster: a group of similar things or people positioned or occurring closely together.
U

Check Your Understanding

1. Big Data in IoT is a __________ and rapidly evolving field in which many different
methodologies and strategies can play a role.
ity

a) complex
b) simple
c) vast
m

d) small
2. While resource allocation for processing might entail costs, it is a strategic investment
for enhancing overall __________ revenue.
)A

a) individual
b) organisational
c) government
d) personal
(c

3. DFS stands for __________, a concept that involves the distribution of files across
multiple nodes.

Amity Directorate of Distance & Online Education

 172 Internet of Things

a) Distributed File Storage

Notes

e
b) Distributed File System
c) Data File System

in
d) Data File Storage
4. Slave nodes hold actual __________ data, while master nodes keep metadata.

nl
a) business
b) personal
c) administrative

O
d) financial
5. A partition can contain multiple keys and their associated values, but records for a
particular key are grouped within a single __________.

ity
a) reducer
b) partitioner
c) key
d) partition
6.

rs
Data written by the partitioner is shuffled and sorted to the __________ node. Each
data component is sorted and grouped into a substantial data list.
a) reducer
ve
b) master
c) slave
d) partition
ni

7. YARN comprises three main components: ResourceManager, NodeManager and

ApplicationMasters, each associated with __________ applications.
a) distributed
U

b) individual
c) separate
d) collective
ity

8. The NameNode and ResourceManager daemons run on the __________ node, while
DataNode and NodeManager daemons operate on slave nodes.
a) slave
m

b) master
c) distributed
d) central
)A

9. Specifically engineered for deployment on cost-effective hardware, HDFS boasts

exceptional __________.
a) performance
b) scalability
(c

c) fault tolerance
d) security

Amity Directorate of Distance & Online Education

 Internet of Things 173

10. Spark emerges as a robust platform for processing vast datasets, embracing a hybrid
Notes

e
structure that accommodates both __________ and stream processing.
a) batch

in
b) real-time
c) sequential
d) manual

nl
11. RDD (Resilient Distributed Dataset) is the fundamental data structure of Apache
Spark, consisting of an __________ collection of objects that compute on each node
of the cluster.

O
a) mutable
b) unstructured
c) immutable

ity
d) dynamic
12. RDDs are partitioned into more manageable logical pieces known as __________. A
portion of the data is stored in each partition.

rs
a) clusters
b) groups
c) sections
ve
d) partitions
13. When several operations must be carried out on a single RDD or when iterative
methods are used, ___________is especially helpful.
a) caching
ni

b) groups
c) sections
U

d) partitions
14. The acronym Scala stands for “Scalable Language,” a cutting-edge, statically typed
programming language that combines the concepts of both __________ and object-
oriented programming (OOP).
ity

a) procedural
b) functional
c) imperative
m

d) event-driven
15. Scala was created to provide a clear and expressive syntax while getting around
some of the drawbacks of __________ programming languages.
)A

a) conventional
b) modern
c) specialised
d) niche
(c

16. Using Scala to build Spark applications offers a strong framework for __________
data processing.

Amity Directorate of Distance & Online Education

 174 Internet of Things

a) centralised
Notes

e
b) sequential
c) distributed

in
d) local
17. Building productive and scalable Spark applications in Scala requires an understanding
of how to use __________ and actions.

nl
a) commands
b) loops

O
c) transformations
d) functions
18. LinkedIn introduced Kafka in 2011 as a high-throughput __________ for its internal

ity
needs.
a) database
b) messaging system
c) search engine
d) programming language

rs
19. Kafka batches data records into chunks, which consumers can access from the file
system, known as the Kafka __________ log.
ve
a) data
b) topic
c) event
ni

d) record
20. In the composition of a Kafka cluster, there are typically one or more servers, referred
to as Kafka __________, that host the Kafka infrastructure.
U

a) masters
b) brokers
c) nodes
ity

d) hosts

Exercise
1. Provide an explanation of how Hadoop operates?
m

2. What are the types of Big Data?

3. What benefits does Kafka offer and how do these advantages enhance its features
and capabilities?
)A

4. Explain the important elements of Apache Spark.

5. Write a short note on Apache Hadoop.
6. Explain briefly the Spark Execution Model.
7. What is Scala’s role in data processing, especially when used with Apache Spark?
(c

Learning Activities
1. Look into how Kafka is used by businesses like Cisco, Uber, Spotify, Goldman Sachs,
Amity Directorate of Distance & Online Education
 Internet of Things 175

LinkedIn and Twitter. Describe how these businesses use Kafka to meet their unique
Notes

e
needs and how it improves their capacity for data processing.
2. Evaluate how well you comprehend how the Hadoop Distributed File System (HDFS)

in
operates. Write a succinct summary outlining the most important lessons you learned
from researching HDFS and its use in distributed data processing.

Check Your Understanding- Answers

nl
1. c) 2. b) 3. b) 4. a)
5. d) 6. a) 7. b) 8. b)

O
9. c) 10. a) 11. c) 12. d)
13. a) 14. b) 15. a) 16. c)
17. c) 18. b) 19. b) 20. b)

ity
Further Readings and Bibliography
1. “Big Data at Work: Dispelling the Myths, Uncovering the Opportunities” by
Thomas H. Davenport
2. “Hadoop in Action” by Chuck Lam
3.

rs
“Apache Spark Deep Learning Cookbook: Over 80 recipes for implementing big
data processing and analytics using Apache Spark” by Ahmed Sherif, Amrith
Ravindra
ve
4. “Kafka Streams in Action: Real-time apps and microservices with the Kafka
Streams API” by Bill Bejeck
ni
U
ity
m
)A
(c

Amity Directorate of Distance & Online Education