MODULE 1

Classification of data, Characteristics, Evolution and

definition of Big data, What is Big data, Why Big data,
Traditional Business Intelligence Vs Big Data, Typical
data warehouse and Hadoop environment.
Big Data Analytics: What is Big data Analytics,
Classification of Analytics, Importance of Big Data
Analytics, Technologies used in Big data Environments,
Few Top Analytical Tools , NoSQL, Hadoop.
 Classification of data
 Digital data can be broadly classified into Structured, Semi Structured
and Unstructured data.
 Structured Data
 When data conforms to a pre defined schema/structure, we say it is
structures data
 Data model- a model of the types of business data that we intend to
store, process and access.
 RDBMS
 Most of Structured Data is held in RDBMS.
 RDBMS conforms to relational data model wherein the data is stored in
rows / columns.
 Cardinality of relation - number of records/ rows/ tuples
 Degree of a relation – number of columns
 Example of good Structured table.

 Sample record in Employee table

Sources of Structured Data
 Ease of working with Structured Data
 1. Insert/update/delete
 [Link]
 3. Indexing
 4. Scalability
 5. Transaction Processing –ACID Properties
 Semi Structured Data
 Semi Structured Data referred to as self describing structures.

 It has the following features:

 It does not conform to the data models that one typically associates with relational databases or any other form of
data tables.
 It uses tags to segregate semantic elements
 Tags are also used to enforce hierarchies of records and fields within data.
 There is no separation between data and schema.
 In semi structured data, entities belonging to the same class and also grouped together need not necessarily have
the same set of attributes.
 Sources of Semi Structured Data
 The main Sources of Semi structured data are : XML and JSON

 1. XML: eXtensible Markup Language(XML)- popular webservices developed utilizing

SOAP ( Simple Object Access Protocol) principles.
 2. JSON (Java Script Object Notation): Used to transmit data between server and web
applications. It is popular webservices developed utilizing the REST( Representational
State Transfer) – an architecture style for creating scalable web services.
 MongoDB and Couchbase Store data in JSON format.
 Sample XML file
<?xml version="1.0" encoding="UTF-8"?>
<bookstore>
<book>
<title lang="en">The Great Gatsby</title>
<author>F. Scott Fitzgerald</author>
<year>1925</year>
<price>10.99</price>
</book>
<book>
<title lang="es">Cien años de soledad</title>
<author>Gabriel García Márquez</author>
<year>1967</year>
<price>12.99</price>
</book>
</bookstore>
Sample JSON file
{
"bookstore": {
"books": [
{
"title": "The Great Gatsby",
"author": "F. Scott Fitzgerald",
"year": 1925,
"price": 10.99,
"language": "en"
},
{
"title": "Cien años de soledad",
"author": "Gabriel García Márquez",
"year": 1967,
"price": 12.99,
"language": "es"
}
]
}
}
 Unstructured data
 Unstructured data does not conform to any pre defined data models.
Sources of Unstructured Data
 How to deal with Unstructured Data
 The following Techniques are used to find patterns in Unstructured Data
 Data Mining
Association rule mining
Regression analysis
Collaborative filtering
 Text analytics or Text mining
 Natural Language Processing(NLP)
 Noisy Text Analysis
 Manual Tagging with Meta Data
 Part of Speech Tagging
 UIMA( Unstructured Information Management Architecture) – Open source platform
from IBM
 Characteristics of Data
Data has 3 key Characteristics

 Composition – It deals with structure of data, ie, the source of data, the
granularity, the types and the nature of data as whether is static or real time
streaming
 Condition – It deals with the state of date, ie, “Can one us this data as is for
analysis ?” or “Does it require cleansing for further enhancement and
enrichment ?”.
 Context – Context deals with “Where has this data been generated ?”, “Why
was this data generated ?”, ” How sensitive is this data ?”, “What are the events
associated with this data ?” and so on.
 Evolution of Big data
 Although the concept of big data itself is relatively new, the origins of large data sets go back
to the 1960s and '70s when the world of data was just getting started with the first data centers
and the development of the relational database.
 The 1970s and before was the era of mainframes.
 The data was primitive and structured.
 The relational databases evolved in 1980s and 1990s.
 The era was of data intensive applications
 The WWW and IoT have led to Structured, Unstructured and Multimedia data.
 Definition of Big data
What is Big Data?
 According to Gartner, the definition of Big Data – “Big data” is high-volume, velocity,
and variety information assets that demand cost-effective, innovative forms of
information processing for enhanced insight and decision making.”

 Big Data refers to complex and large data sets that have to be processed and analyzed
to uncover valuable information that can benefit businesses and organizations.
Challenges with Big Data
 What is Big Data
 Big data is data that is big in volume , velocity and variety.
 volume
 We have seen it data grow from bits to byte to peta bytes and exa bytes.
 Where does this data get generated ?
 Sources of big data
 velocity
 We have moved from the days of batch processing to real time processing.

Batch -> Periodic Processing -> Near real time -> Real time processing
variety
 Variety deals with wide range of date types and sources of data.
 1. Structured data
 2. Semi Structured data
 3. Unstructured data
 Why Big Data
 The more data we have for analysis, the greater will be the analytical accuracy.
 Then greater would be the confidence in decision based on these analytical
findings.
 It leads to greater positive impact on enhancing operational efficiencies,
reducing cost and time, innovating new product, new services, optimizing
existing services.
Traditional Business Intelligence Vs Big Data
Differences between Traditional BI and Big Data
 In traditional BI environment, all the enterprise’s data stored in a
central server, whereas in Big data environment, data resides in
distributed file system.
 In traditional BI, data is generally analyzed in offline mode,
whereas in Big data, it analyzed in real time as well as in offline
mode.
 Traditional BI is about structured data and here data is taken to
processing functions ( ie, move data to the code) whereas Big data is
about variety: structured, semi- structured and unstructured data and
here processing functions are taken to the data (ie, move code to
data)
Typical data warehouse environment
 Operational or day-to-day business data is gathered from ERP systems,
CRM, legacy systems and several third-party applications.
 The data from these sources may differ in format. [ It can be different
databases like Oracle, MS SQL Server, DB2, MySql, TeraData or
Spreadhseet(.xls,.xlsx), .csv,.txt etc ]
 Data may come from data sources located in same or different geographies.
 This data is then integrated, cleaned up, transformed and standardized
through the process of Extraction, Transformation and Loading(ETL).
 The transformed data is then loaded into enterprise Data warehouse or Data
marts.
 A host of market leading business intelligence and analytical tools are then
used to enable decision making from the use of ad-hoc queries, SQL,
Enterprise dashboard, data mining etc.
Typical data warehouse environment
Typical Hadoop environment
 Typical Hadoop environment
 A Hadoop Environment is designed for storing and processing massive
amounts of structured and unstructured data. Unlike a data warehouse,
Hadoop is highly scalable, fault-tolerant, and can handle data in diverse
formats such as logs, social media, and sensor data.
 Key Components of a Hadoop Environment:
1. HDFS (Hadoop Distributed File System):
1. A distributed file system that stores large datasets across many nodes
in a cluster.
2. Data is divided into blocks (default 128 MB or 256 MB), and each
block is replicated to ensure fault tolerance.
3. Can store both structured and unstructured data (e.g., images, logs,
JSON, videos).
 Typical Hadoop environment
 MapReduce (or alternative frameworks like Spark):
• The computation framework for processing large datasets in a distributed
manner.
• MapReduce splits data processing into two phases: the Map phase (data
transformation) and the Reduce phase (aggregation of results).
• Apache Spark is increasingly used as a faster alternative to MapReduce
for in-memory processing.
 YARN (Yet Another Resource Negotiator):
• Manages resources across the cluster and schedules tasks.
• It allows multiple applications to share resources in the Hadoop
environment.
 Typical Hadoop environment
 Hadoop Ecosystem:
• Hive: A data warehouse-like tool built on top of Hadoop for running SQL-like
queries.
• HBase: A NoSQL database that allows for real-time querying of large datasets.
• Pig: A high-level platform for writing MapReduce programs using the Pig Latin
scripting language.
• Flume: A tool for collecting, aggregating, and moving log and event data into
HDFS.
• Sqoop: For importing and exporting data between relational databases and
HDFS.
• Oozie: Workflow scheduler to manage and coordinate Hadoop jobs.
• ZooKeeper: Coordination service for distributed applications.
 Typical Hadoop environment
 Hadoop Workflow:
1. Data Collection: Data is ingested from various sources (e.g., logs,
databases, external data streams) using tools like Flume or Sqoop.
2. Storage: The data is stored in HDFS in a distributed and fault-tolerant
manner.
3. Data Processing: Processing can be done via MapReduce, Spark, or
other frameworks to perform transformations, aggregations, or machine
learning tasks.
4. Data Analysis: Data analysis can be done using tools like Hive (SQL-like
querying) or HBase (real-time queries).
5. Data Output: Processed data is exported or used for reporting.
 Typical Hadoop environment
 Hadoop Key Features:
• Handles massive volumes of data (terabytes or petabytes).
• Supports both structured and unstructured data types.
• Can scale horizontally by adding more nodes to the cluster.
• Highly fault-tolerant with data replication in HDFS.
• Supports batch processing with MapReduce and real-time
processing with Spark and HBase.
What is Big Data Analytics?
 Big Data Analytics refers to the process of examining large and complex sets of data,
often referred to as "big data," to uncover patterns, correlations, trends, and insights that
are not immediately obvious.
 It involves using advanced computational tools and algorithms to analyze and process
vast amounts of data that traditional data-processing software cannot handle efficiently.
 The main goals of Big Data Analytics include:
1. Extracting meaningful insights: Analyzing data to gain deeper understanding and
inform decision-making in areas such as business strategy, operations, customer behavior,
and more.
2. Improving business outcomes: By identifying trends and predictive patterns, businesses
can optimize processes, enhance customer experiences, and gain a competitive edge.
3. Enabling predictive analytics: Using historical data and algorithms to predict future
trends or events, such as forecasting sales, customer churn, or product demand.
 Move code to data for
greater speed and
Better, faster efficiency
Richer, deeper
decisions in real time insights into
customers, partners
and the business

Working with data sets

whose volume and variety
is beyond the storage & Competitive
processing capability of a
Big Data Analytics
Advantage
typical Database Software

Technology enabled
IT’s collaboration with analytics
business users & data
scientists Time-sensitive decisions
made in near real time by
processing a steady
stream of real-time data
 What Big Data Analytics isn’t?
Only about Volume
“One-size fit all” traditional RDBMS
built on shared disk & memory

Big Data Analytics isn’t ... Just about Technology

Only used by huge online

companies like Google or Amazon
Meant to replace RDBMS

Meant to replace Data Warehouse

 Classification of Analytics
There are basically two schools of thought.
1. Classify analytics into– basic, operationalized,
advanced and monetized.
2. Classify analytics into– analytics 1.0, analytics
2.0, analytics 3.0
First School of Thought
Second School of Thought
 Second School of Thought
 Analytics 1.0, 2.0 and 3.0
 Importance of Big Data Analytics
Various Approaches to analysis of data
 1. Reactive – Business Intelligence: It is about analysis of past or historical data and
then displaying the findings of the analysis or reports in the forms of enterprise
dashboard, alerts, notifications etc.
 It has support for both pre specified report as well as ad hoc querying.
 2. Reactive- Big Data analytics: Here the analysis is done on huge data sets, but the
approach is still reactive as it is still based on static data.
 3. Proactive- Analytics: This is to support futuristic decision making by the use of Data
mining , predictive modeling, text mining and statistical analysis. This analysis still uses
database management practices and have limitations like storage capacity and processing
capability.
 4. Proactive- Big Data analytics: This is sieving through terabytes, petabytes and
exabytes of information to filter out relevant data to analyze.
 This Includes high performance analytics to gain rapid insights from big data and the
ability to solve complex problems with more data.
NoSQL ( NOT ONLY SQL)
Features of NoSQL
They are open source
They are non relational
They are distributed
They are schema less
They are cluster friendly
They are born out of 21st century web applications.
NoSQL ( NOT ONLY SQL)
Where it is used

 Widely used in big data and other real-time applications

 Used to stock log data.
 Used to store social media data and all such data which cannot be stored and
analyzed in RDBMS
NoSQL ( NOT ONLY SQL)
What is NoSQL?
Non-relational data storage systems

No fixed table schema

No Joins
NoSQL

No multi-document transactions

Relaxes one or more ACID properties

 Types of NoSQL
1. Key-Value: It maintain a big hash table of keys and values.
Examples: Dynamo, Redis, Riak etc.
Sample Key-Value pair in Key-value database:

2. Document: It maintains data in collections constituted of documents.

Examples: MongoDB, Apache CouchDB, Couchbase, MarkLogic etc.
Sample document in document database:
Types of NoSQL
3. Column: Each storage block has data from only one column.
Examples: Cassandra, Hbase etc.

4. Graph: Also called Network database.

A graph stores data in nodes.
Examples: Neo4j, HyperGraphDB, etc.
Sample graph in graph database.
 Types of NoSQL

Key value Column-oriented Document Graph data

data store data store data store store

• Riak • Cassandra • MongoDB • InfiniteGraph

• Redis • HBase • CouchDB • Neo4
• Membase • HyperTable • RavenDB • Allegro
Graph
Why NoSQL
 It has the scale out architecture instead of the monolithic architecture in
relational databases.
 It can house large volume of structured, semi structured and unstructured
data.
 Dynamic Schema: NoSQL database allows insertion of data without a
pre defined schema.
 Auto-sharding: It automatically spreads data across an arbitrary number
of servers.
 Replication: It offers good support for replication which guarantee high
availability, fault tolerance and disaster recovery.
Advantages of NoSQL
Cheap, Easy to implement

Easy to distribute

Can easily scale up & down

Advantages of NoSQL
Relaxes the data consistency
requirement

Doesn’t require a pre-defined

schema

Data can be replicated to

multiple nodes and can be
partitioned
NoSQL Vendors
SQL Vs. NoSQL
 NewSQL

SQL interface for application interaction

ACID support for transactions

Characteristics of NewSQL An architecture that provides higher per node

performance vis-a-vs traditional RDBMS solution

Scale out, shared nothing architecture

Non-locking concurrency control mechanism so

that real time reads will not conflict with writes
SQL Vs. NoSQL Vs. NewSQL
 Hadoop
 Hadoop is an open-source project of Apache foundation.
 It is framework written in Java
 Developed by Doug Cutting in 2005, who was working with yahoo.
 It was created to support distribution for “Nutch”, the text search engine.
 Hadoop uses Google’s MapReduce and Google file system technologies as its
foundation. Hadoop
Apache Open-Source Software Framework

Inspired by
- Google MapReduce
- Google File System

Hadoop Distributed File System

MapReduce

 Hadoop is now a core part of the computing infrastructure for companies such
as Yahoo, Facebook, LinkedIn, Twitter etc.
 Features of Hadoop
 It is optimized to handle massive quantities of structured, semi structured
and un structured data using relatively inexpensive computers.
 Hadoop has a shared nothing architecture.
 It replicates data across multiple computers.
 Hadoop is for high throughput rather than low latency
 It complements OLTP and OLAP
 It is not good for processing small files.
 It is not good when work cannot be parallelized or dependencies within
data
Key Advantages of Hadoop
 Stores data in its native format
 Scalable
 Cost-effective
 Resilient to failure
 Flexibility
 Fast
Hadoop versions

Hadoop 1.0 Hadoop 2.0

MapReduce
MapReduce Others
(Cluster Resource Manager
(Data Processing) (Data Processing)
& Data Processing)

HDFS YARN
(redundant, reliable storage) (Cluster Resource Manager)
HDFS
(redundant, reliable storage)
 Hadoop Ecosystem
Ambari
(Provisioning, Managing & Monitoring Hadoop Cluster)

Sqoop Mahout Pig R Hive

(Machine learning) (Data Flow) (Statistics) (Data Warehouse) Oozie
(Relational Database
(Workflow)
Data Collector)
Map Reduce Hbase
(Distributed Processing) (Distributed Table Store)
Flume/Chukwa Zookeeper
(Log Data Collector) HDFS (Coordination)
(Hadoop Distributed File System)
Hadoop Ecosystem
Components that help with Data Ingestion are:
1. Sqoop
2. Flume
Components that help with Data Processing are:
1. MapReduce
2. Spark
Components that help with Data Analysis are:
1. Pig
2. Hive
3. Impala
Hadoop Ecosystem Components for Data
Ingestion
Sqoop:
 Sqoop stands for SQL to Hadoop. It can provision the data from
external system on to HDFS and populate tables in Hive and HBase.
Flume:
 Flume is an important log aggregator (aggregates logs from different
machines and places them in HDFS) component in the Hadoop
Ecosystem.
 Hadoop Ecosystem Components for Data
Processing
MapReduce:
 It is a programing paradigm that allows distributed and parallel processing
of huge datasets. It is based on GoogleMapReduce.
Spark:
 It is both a programming model as well as a computing model. It is an open
source big data processing framework.
 It is written in Scala. It provides in-memory computing for Hadoop.
 Spark can be used with Hadoop coexisting smoothly with MapReduce
(sitting on top of Hadoop YARN) or used independently of Hadoop
(standalone).
Hadoop ecosystem components for Data Analysis
Pig
 It is a high level scripting language used with Hadoop. It serves as an alternative to
MapReduce. It has two parts:
 Pig Latin: It is a SQL like scripting language.
 Pig runtime: is the runtime environment.
Hive:
 Hive is a data warehouse software project built on top of Hadoop. Three main tasks performed
by Hive are summarization, querying and analysis
Impala:
 It is a high performance SQL engine that runs on Hadoop cluster. It is ideal for interactive
analysis. It has very low latency measured in milliseconds. It supports a dialect of SQL called
Impala SQL.