SSPB3243: Big Data
in International
Business
Hadoop and Big Data Ecosystem:
Dr. Norazlina Mohd Yasin
Senior Lecturer
School of Professional and
Continuing Education (SPACE)
Universiti Teknologi Malaysia
Big Data Infrastructure
Search and Visualization (Lucene)
Apache Lucene is a search library that can be
used to add search capabilities to applications.
Data Orchestration (Hbase, Flume,
ZooKeeper)
HBase is a NoSQL database for big data;
Flume is a service for efficiently collecting,
aggregating, and moving large amounts of log
data; ZooKeeper is a centralized service for
maintaining configuration information and
providing distributed synchronization.
Data Access (Pig, Hive, Sqoop, Avro):
Tools that help in accessing data from the
Hadoop ecosystem. Pig and Hive allow for
scripting and querying (HiveQL)
respectively; Sqoop is for transferring data
between Hadoop and relational databases;
Avro is a data serialization system.
Hadoop (HDFS, MapReduce): Hadoop
Distributed File System (HDFS) for
storing data across multiple machines,
and MapReduce for processing large
data sets.
Apache Hadoop was born to enhance the
usage and solve major issues of big data.
Very difficult to manage the data of around one
billion pages of content
Google invented a new methodology of
processing data popularly known as
MapReduce
Hadoop biggest strength is scalability. It
upgrades from working on a single node to
thousands of nodes without any issue in a
seamless manner.
Hadoop runs the applications on the basis of
MapReduce where the data is processed in
parallel and accomplish the entire statistical
analysis on large amount of data.
Oozie: This is a Zookeeper: Open source centralized
scheduling system for service which is used to provide
workflow management, coordination between distributed
executing workflow applications of Hadoop. It offers the
routes for successful registry and synchronization service
completion of the task on a high level.
in a Hadoop.
Hive: It is a data Pig: Pig is a high-level framework
warehouse tool which ensures us to work in
basically used for coordination either with Apache
analyzing, querying Spark or MapReduce to analyze
and summarizing of the data. The language used to
analyzed data code for the frameworks are
concepts on top of the known as Pig Latin.
Hadoop framework.
•Hadoop HDFS (Storage layer) – HBase: it is a non-relational, Mapreduce •Hadoop YARN –Hadoop YARN is a
Hadoop Distributed File SystemorHDFS distributed database management (Processing/Computation layer): framework used for job scheduling
is based on the Google File System (GFS) system that works efficiently on MapReduce is a parallel and cluster resource management.
which provides a distributed file system sparse data sets and it is highly programming model mainly used for
that is especially designed to run on scalable. writing large amount of data
commodity hardware. It reduces the distribution applications devised
faults or errors and helps incorporate from Google for efficient processing
low-cost hardware. It gives high level of large amounts of datasets, on
processing throughput access to large group of clusters.
application data and is suitable for
applications with large datasets.
The Hadoop High-level Architecture
HDFS (Hadoop Distributed File System)
Serves as the storage system of Hadoop, designed to store large
files across multiple machines.
Fault-tolerant, high throughput, suitable for applications with
large datasets, stores data in a distributed manner, and works
well with MapReduce.
MapReduce
A programming model for processing large datasets.
Efficiently processes large volumes of data in parallel
across a Hadoop cluster, breaks down tasks into
smaller chunks (Map tasks), and then consolidates
the results (Reduce tasks).
Class Activity:
Explain how the process of big data using HDFS, MapReduce
and Yarn, ZooKeeper, HBase, Hive, Pig
Thank you
for today.