CST 322 DATA ANALYTICS

Module - 4 (Big Data Analytics)

Ojus Thomas Lee

CE Kidangoor

Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 1 / 66
CO’s

Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 2 / 66
CO - Module Mapping

Mod 1 Mod 2 Mod 3 Mod 4 Mod 5

CO1 X
CO2 X
CO3 X
CO4 X
CO5 X
CO6 X

Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 3 / 66
PO’s

Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 4 / 66
CO-PO-Mapping

Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 5 / 66
Syllabus - Module – 4 (Big Data Analytics)

Big Data Overview – State of the practice in analytics, Example

Applications - Credit Risk Modeling, Business Process [Link]
Data Analytics using Map Reduce and Apache Hadoop, Developing
and Executing a HadoopMapReduce Program.

Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 6 / 66
 1
What is Big Data?

The term “big data” refers to data that is so large, fast or complex
that it’s difficult or impossible to process using traditional methods.
Using big data, companies can know radically more about their
businesses, and directly translate that knowledge into improved
decision making and performance.

1
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 7 / 66
 2
Types of Big Data

Structured
Semi-Structured
Unstructured

2
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 8 / 66
 3
Types of Big Data

Structured

3
[Link]
data-vs-semi-structured-data/
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 9 / 66
 4
Types of Big Data

Semi-Structured

Eg : XML, JSON, CSV etc

4
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 10 / 66
 5
Types of Big Data

Un-Structured

5
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 11 / 66
 6
Characteristics of Big Data

6
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 12 / 66
 7
Characteristics of Big Data

Volume – Volume is the V most frequently associated with big data –

data quantity can be so big it can reach incomprehensible proportions.
For example, Facebook stores more than 250 billion images uploaded
by people, in addition to all the individual posts (over 2.5 trillion
posts).
Overall, close to 2.5 exabytes (1 Exabyte = 109 Gigabytes) of data is
being produced every day.
And, the total data in the world is expected reach 44 zettabytes (1
Zettabyte = 1012 Gigabytes) by 2020.

7
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 13 / 66
 8
Characteristics of Big Data

Velocity – Velocity is the measure of how fast the data is being

generated and collected. For example, in Facebook, more than 350
million photos are being uploaded every day.
This data needs to be collected, stored, filed, and available to be
retrieved whenever required. Data velocity highlights the need to
process the data quickly, and most importantly, use it at a faster rate
than ever before.
Many types of data have a limited shelf-life and their value can
diminish very quickly.
For example, to improve sales in a retail business, out of stock
products should be identified within minutes rather than days or
weeks.

8
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 14 / 66
 9
Characteristics of Big Data

Variety – Data can come in all forms – photos, videos, sensor data,
tweets, encrypted packets and so on.
Data is not always accumulated in the form of rows and columns in a
database – it can either be structured or unstructured.
With an increase in data sources, there are more varieties of data in
different formats-from traditional documents and databases, to
semi-structured and unstructured data including click streams, GPS
location data, and social media apps.
Different data formats mean it’s tougher to derive value from the
data because it must all be extracted for processing in different ways.

9
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 15 / 66
 10
Characteristics of Big Data

Veracity – Data veracity is the degree to which data is accurate,

precise, and trusted.
It refers to the biases, noise, and abnormality in the data. To avoid
‘dirty data’ accumulating in our systems, we need to have a strategy
to keep the data usable.
Having diverse and messy data requires a lot of cleanups.
Obtaining and cleaning datasets still takes more time for a data
scientist than putting their investigational skills (statistics, machine
learning, and algorithms) to use.

10
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 16 / 66
Big Data Tools11

Apache Hadoop
Hadoop is an open-source framework from Apache and runs on
commodity hardware.
It is used to store process and analyze Big Data.
Apache Spark
Spark supports both real-time as well as batch processing.
It also supports in-memory calculations, which makes it 100 times
faster than Hadoop.
Apache Storm
Apache Storm is an open-source big data tool, distributed real-time
and fault-tolerant processing system. It efficiently processes unbounded
streams of data.
The processing speed of Storm is very high. It is easily scalable and
also fault-tolerant.

11
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 17 / 66
Big Data Tools12

Apache Cassandra
Apache Cassandra is a distributed database that provides high
availability and scalability without compromising performance efficiency.
Cassandra works quite efficiently under heavy loads.
It does not follow master-slave architecture so all nodes have the same
role.
Apache Cassandra supports the ACID (Atomicity, Consistency,
Isolation, and Durability) properties.
MongoDB
MongoDB is an open-source data analytics tool, NoSQL database that
provides cross-platform capabilities.
Apache Flink
Apache Flink is an Open-source data analytics tool distributed
processing framework for bounded and unbounded data streams.

12
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 18 / 66
Big Data Tools13

Kafka
Apache Kafka is an open-source platform that was created by LinkedIn
in the year 2011.
Apache Kafka is a distributed event processing or streaming platform
which provides high throughput to the systems.
It can handle trillions of events a day.
It is highly scalable and also provides great fault tolerance.
R Programming
R is an open-source programming language and is one of the most
comprehensive statistical analysis languages.
helps in generating the results of data analysis in graphical as well as
text format.

13
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 19 / 66
Big Data Tools14

HDFS
HDFS is a Filesystem of Hadoop designed for storing very large files
running on a cluster of commodity hardware.
It is designed on the principle of storage of less number of large files
rather than the huge number of small files.
HDFS Nodes
Name nodes
Data Node

14
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 20 / 66
Hadoop15
Hadoop Distributed File System (HDFS)

15
[Link]
hdfs/[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 21 / 66
 16
Hadoop - HDFS

Name nodes
NameNode works as a Master in a Hadoop cluster that guides the
Datanode(Slaves).
Namenode is mainly used for storing the Metadata i.e. the data about
the data.
Meta Data can be the transaction logs that keep track of the user’s
activity in a Hadoop cluster.
Meta Data can also be the name of the file, size, and the information
about the location(Block number, Block ids) of Datanode that
Namenode stores to find the closest DataNode for Faster
Communication.
Namenode instructs the DataNodes with the operation like Delete,
Create, Replicate, etc.

16
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 22 / 66
 17
Hadoop - HDFS

Data Nodes
DataNodes works as a Slave DataNodes are mainly utilized for storing
the data in a Hadoop cluster,
the number of DataNodes can be from 1 to 500 or even more than
that.
The more number of DataNode, the Hadoop cluster will be able to
store more data.
So it is advised that the DataNode should have high storing capacity to
store a large number of file blocks.

17
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 23 / 66
 18
Hadoop - HDFS

File Block In HDFS:

Data in HDFS is always stored in terms of blocks.
So the single block of data is divided into multiple blocks of size
128MB which is default and you can also change it manually.

18
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 24 / 66
 19
Hadoop - HDFS

19
[Link] sschung/cis612/LectureNotesH adoopFinal1 .pdf
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 25 / 66
 20
Hadoop - HDFS

Replication In HDFS
Replication ensures the availability of the data.
Replication is making a copy of something and the number of times
you make a copy of that particular thing can be expressed as it’s
Replication Factor.
As we have seen in File blocks that the HDFS stores the data in the
form of various blocks at the same time Hadoop is also configured to
make a copy of those file blocks.

20
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 26 / 66
Hadoop - HDFS
21

21
[Link] sschung/cis612/LectureNotesH adoopFinal1 .pdf
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 27 / 66
Hadoop - HDFS-Replication
22

22
Ojus[Link]
Thomas Lee CE Kidangoor sschung/cis612/LectureNotes
CST 322 DATA ANALYTICS Module - 4 H adoopFinal
(Big [Link]
Data Analytics) 28 / 66
Hadoop - HDFS

23

Rack Awareness
The rack is nothing but just the physical collection of nodes in our
Hadoop cluster (maybe 30 to 40).
A large Hadoop cluster consists of so many Racks,
With the help of this Racks information Namenode chooses the closest
Datanode to achieve the maximum performance while performing the
read/write information which reduces the Network Traffic.

23
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 29 / 66
Hadoop - HDFS

24

24
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 30 / 66
Hadoop - HDFS-Write
25

25
Ojus[Link]
Thomas Lee CE Kidangoor sschung/cis612/LectureNotes
CST 322 DATA ANALYTICS Module - 4 H adoopFinal
(Big [Link]
Data Analytics) 31 / 66
Hadoop - HDFS-Read
26

26
Ojus[Link]
Thomas Lee CE Kidangoor sschung/cis612/LectureNotes
CST 322 DATA ANALYTICS Module - 4 H adoopFinal
(Big [Link]
Data Analytics) 32 / 66
Hadoop - HDFS-Components
27

27
Ojus[Link]
Thomas Lee CE Kidangoor sschung/cis612/LectureNotes
CST 322 DATA ANALYTICS Module - 4 H adoopFinal
(Big [Link]
Data Analytics) 33 / 66
Hadoop - HDFS-Components- JobTracker
28

28
Ojus[Link]
Thomas Lee CE Kidangoor sschung/cis612/LectureNotes
CST 322 DATA ANALYTICS Module - 4 H adoopFinal
(Big [Link]
Data Analytics) 34 / 66
Hadoop- HDFS COmponents-JobTracker
29

Job Tracker –
JobTracker process runs on a separate node and not usually on a
DataNode.
JobTracker is an essential Daemon for MapReduce execution in MRv1.
It is replaced by ResourceManager/ApplicationMaster in MRv2.
JobTracker receives the requests for MapReduce execution from the
client.
JobTracker talks to the NameNode to determine the location of the
data.
JobTracker finds the best TaskTracker nodes to execute tasks based on
the data locality (proximity of the data) and the available slots to
execute a task on a given node.
JobTracker monitors the individual TaskTrackers and the submits back
the overall status of the job back to the client.
JobTracker process is critical to the Hadoop cluster in terms of
MapReduce execution.
When the JobTracker is down, HDFS will still be functional but the
MapReduce execution can not be started and the existing MapReduce
jobsCEwill
Ojus Thomas Lee be halted.
Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 35 / 66
Hadoop - HDFS-Components- Name Node
30

30
Ojus[Link]
Thomas Lee CE Kidangoor sschung/cis612/LectureNotes
CST 322 DATA ANALYTICS Module - 4 H adoopFinal
(Big [Link]
Data Analytics) 36 / 66
Hadoop - HDFS-Components- Name Node
31

31
Ojus[Link]
Thomas Lee CE Kidangoor sschung/cis612/LectureNotes
CST 322 DATA ANALYTICS Module - 4 H adoopFinal
(Big [Link]
Data Analytics) 37 / 66
Hadoop - HDFS-Components- Task Tracker
32

32
Ojus[Link]
Thomas Lee CE Kidangoor sschung/cis612/LectureNotes
CST 322 DATA ANALYTICS Module - 4 H adoopFinal
(Big [Link]
Data Analytics) 38 / 66
Hadoop- HDFS COmponents-JobTracker

33

TaskTracker –
TaskTracker runs on DataNode. Mostly on all DataNodes.
TaskTracker is replaced by Node Manager in MRv2.
Mapper and Reducer tasks are executed on DataNodes administered by
TaskTrackers.
TaskTrackers will be assigned Mapper and Reducer tasks to execute by
JobTracker.
TaskTracker will be in constant communication with the JobTracker
signalling the progress of the task in execution.
TaskTracker failure is not considered fatal. When a TaskTracker
becomes unresponsive, JobTracker will assign the task executed by the
TaskTracker to another node.

33
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 39 / 66
Hadoop - MapReduce

34

Map Reduce
MapReduce nothing but just like an Algorithm or a data structure.
The major feature of MapReduce is to perform the distributed
processing in parallel in a Hadoop cluster which Makes Hadoop working
so fast.
When you are dealing with Big Data, serial processing is no more of
any use.
MapReduce has mainly 2 tasks which are divided phase-wise:
Map Task
Reduce Task

34
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 40 / 66
Hadoop - MapReduce

35

Map Reduce Schematic Diagram

35
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 41 / 66
Hadoop -Map Split
36

36
Ojush[Link]
Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 42 / 66
Hadoop - Map Split
37

37
Ojush[Link]
Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 43 / 66
Hadoop - MapReduce

38

Map Reduce - Map Task:

RecordReader :- The purpose of recoredreader is to break the records.
It is responsible for providing key-value pairs in a Map() function.
Map : A map is nothing but a user-defined function whose work is to
process the Tuples obtained from record reader.

38
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 44 / 66
Hadoop - MapReduce

39

Map Reduce - Map Task:

Combiner: Combiner is used for grouping the data in the Map
workflow. It is similar to a Local reducer. The intermediate key-value
that are generated in the Map is combined with the help of this
combiner.
Partitioner : Partitioner is responsible for fetching key-value pairs
generated in the Mapper Phases. The partitioner generates the shards
corresponding to each reducer.

39
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 45 / 66
Hadoop - MapReduce

40

Map Reduce - Reduce Task:

Shuffle and Sort: The process in which the Mapper generates the
intermediate key-value and transfers them to the Reducer task is
known as Shuffling. Using the Shuffling process the system can sort
the data using its key value.
Reduce: The main function or task of the Reduce is to gather the
Tuple generated from Map and then perform some sorting and
aggregation sort of process on those key-value depending on its key
element.
OutputFormat: Once all the operations are performed, the key-value
pairs are written into the file with the help of record writer, each record
in a new line, and the key and value in a space-separated manner.

40
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 46 / 66
Hadoop - MapReduce

41

Map Reduce Workflow

41
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 47 / 66
Hadoop - MapReduce
42

Map Reduce Workflow

42
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 48 / 66
Hadoop - MapReduce
43

Map Reduce Example

43
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 49 / 66
Hadoop - MapReduce
44

Map Reduce Example

44
[Link] dell/teaching/cc/book/ditp/ditpc [Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 50 / 66
Hadoop - MapReduce

45

Developing and Executing a Hadoop MapReduce Program

Develop a Hadoop MapReduce program by writing Java code using an
Interactive Development Environment (IDE) tool such as Eclipse
A typical MapReduce program consists of three Java files: one each for
the driver code,
map code, and
reduce code.
The Java code is compiled and stored as a Java Archive (JAR) file.
The JAR file created is then executed against the specified HDFS input
files.

45
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 51 / 66
Hadoop - MapReduce- Main Class -WordCount
46

Map Reduce Example

46
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 52 / 66
Hadoop - MapReduce-WordCount - MapReduceFuntion
47

Map Reduce Example

47
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 53 / 66
Hadoop - MapReduce

48

With a neat diagram show the MapReduce workflow for word count
of the text :
“book pen pencil pen pencil book pencil book pen eraser sharpener
book sharpener”

48
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 54 / 66
Hadoop - MapReduce - Creating Input
49

Map Reduce Example

49
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 55 / 66
Hadoop - MapReduce - Running MapReduce
50

Map Reduce Example

50
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 56 / 66
Hadoop - MapReduce - Running MapReduce
51

Map Reduce Example

51
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 57 / 66
Hadoop - MapReduce

52

Developing and Executing a Hadoop MapReduce Program

For users who prefer to use a programming language other than Java,
there are some other options.
One option is to use the Hadoop Streaming API, which allows the user
to write and run Hadoop jobs with no direct knowledge of Java.
User can use programming languages such as Python, C, or Ruby.

52
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 58 / 66
Hadoop - MapReduce - HadoopStreaming
53

Map Reduce Example

53
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 59 / 66
Hadoop - MapReduce - HadoopStreaming

54

When a script is specified for mappers(map executable), each

mapper task will launch the script as a separate process when the
mapper is initialized.
As the mapper task runs, it converts its inputs into lines and feed the
lines to the standard input (STDIN) of the process.
The mapper collects the line-oriented outputs from the standard
output (STDOUT) of the process and converts each line into a
key/value pair, which is collected as the output of the mapper.

54
[Link] treaming .htm
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 60 / 66
Hadoop - MapReduce - HadoopStreaming

55

When a script is specified for reducers(reduce executable), each

reducer task will launch the script as a separate process, then the
reducer is initialized.
As the reducer task runs, it converts its input key/values pairs into
lines and feeds the lines to the standard input (STDIN) of the process.
In the meantime, the reducer collects the line-oriented outputs from
the standard output (STDOUT) of the process, converts each line
into a key/value pair, which is collected as the output of the reducer.

55
[Link] treaming .htm
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 61 / 66
Hadoop - MapReduce

56

Developing and Executing a Hadoop MapReduce Program

A second alternative is to use Hadoop pipes,
The mechanism that uses compiled C++ code for the map and
reduced functionality.

56
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 62 / 66
Hadoop - YARN

57

YARN Stands for Yet Another Resource Negotiator.

YARN is responsible for Resource management part.
Yarn execution model is more generic as compare to Map reduce
YARN can execute those applications as well which dont́ follow Map
Reduce model
In the place of job tracker and task tracker Application, the master
comes into the picture.
YARN is more isolated and scalable
YARN has Name Node, Data node, secondary Name node, Resource
Manager and Node Manager.

57
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 63 / 66
Hadoop - YARN

58

YARN allows the data stored in HDFS (Hadoop Distributed File

System) to be processed and run by various data processing engines
such as
Batch processing,
Stream processing,
Interactive processing,
Graph processing and many more.

58
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 64 / 66
Hadoop - MapReduce
59

Hadoop Yarn

HadoopYarn
59
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 65 / 66
Hadoop - MapReduce
60

Hadoop Yarn

HadoopYarn
60
[Link]
Ojus Thomas Lee CE Kidangoor CST 322 DATA ANALYTICS Module - 4 (Big Data Analytics) 66 / 66