ADT 302

CONCEPTS IN BIG DATA ANALYTICS

Preamble

● This course helps the learner to understand the basic concepts of big data
analytics.
● This course covers on big data technologies used for storage, analysis and
manipulation of data.
● The student will learn about fundamentals of Hadoop, MapReduce, Pig, Hive,
R and have hand on training on the same It also help to develop projects and
apply existing data analytics tools to gain comprehensive knowledge on Data
analytics.
● It enables the learners to perform data analysis on a real-world scenario using
appropriate tools.
Course Outcomes
Syllabus

Module – 1 (Introduction to Big Data)

Introduction to Big data, Conventional Data vs Big data, Big data architecture, Big
data platforms,Nature of data, Analytic processes and tools, 5 V’s of Big data, Big
data analytical method,Intelligent data analysis, Big data analytics life cycle.

Module - 2 (Introduction to Stream Computing)

Introduction to stream concepts – Streaming data architecture, Stream data

model, Sampling techniques for efficient stream processing, Filtering streams –
Bloom filter, Count distinct problem– Flajolet martin algorithm, Estimating
moments, Counting oneness in a window – DGIM Algorithm
Module - 3 (Hadoop Distributed File System)

History of Hadoop, Hadoop Ecosystem, Core Components, HDFS- Architecture,

Using HDFS Files, HDFS Design, Blocks, Namenodes and Data nodes, Basic File
system Operations, Hadoop Specific File Types, Anatomy of a file read, Anatomy
of a file write. Data Processing with MapReduce: Execution Pipeline, Runtime
Coordination and Task Management in MapReduce,Designing MapReduce
implementations: Using MapReduce as a framework for parallel processing,
Example-Road Enrichment.
Module - 4 (Pig, Hive, HBase)

Pig : Introduction to PIG, Execution Modes of Pig, Comparison of Pig with

Databases, Grunt, Pig Latin, User Defined Functions, Data Processing operators.
Hive : Hive Shell, Hive Services, Hive Metastore, Comparison with Traditional
Databases, HiveQL, Tables, Querying Data and User Defined Functions. Hbase :
HBasics, Concepts, Clients, Example, Hbase Versus RDBMS.
Module - 5 (Introduction to R programming)

Introduction to R – Overview of modern data analytic tools, Introduction to R, R

Graphical User Interfaces - Features of R Language, Vectors, Filtering, Creating
Matrices , Applying Functions to Matrix Rows and Columns, Lists , Creating List ,
General List Operations, Data Frames , Creating Data Frames , Matrix like
Operations in Frames , Applying Functions to Data Frames ,Reading and Writing
Files.
Big Data

● Big data refers to data that is so large, fast and complex that it’s
difficult or impossible to process using traditional methods.
● Data that contains greater variety, arriving in increasing
volumes and with more velocity.
● This is also known as the three Vs.
Example of Big Data
● Stock Exchange:

The New York Stock Exchange is an example of Big Data that

generates about one terabyte of new trade data per day.

● Social Media:
The statistic shows that 500+terabytes of new data get ingested into the
databases of social media site Facebook, every day.
● Jet engine:

A single Jet engine can generate 10+terabytes of data in 30 minutes of

flight time. With many thousand flights per day, generation of data
reaches up to many Petabytes.
Comparison of Big Data with Conventional Data

Big data Conventional data

● Data set size in control.
● Huge data sets.
● Normally structured data such as numbers
● Unstructured data such as text, video,
and categories, but it can take other forms
and audio.
as well.
● Hard-to-perform queries and analysis.
● Relatively easy-to-perform queries and
● Need tools such as Hadoop, Hive,Hbase,
analysis.
Pig and so on.
● Tools such as SQL, SAS (Statistical
● Generated by big financial institutions,
Analysis System), R, and Excel alone may
Facebook, Google, Amazon, eBay,
be sufficient.
Walmart, and so on.
● Generated by small enterprises and small
banks.
Characteristics of Big Data

● Big data is a collection of data from many different sources and is often
described by five characteristics: volume, value, variety, velocity, and veracity.
Characteristics of Big Data

Volume
● It describes the enormous amount of information
● Which is the defining criterion for whether we can consider a dataset as Big
Data or not.
Velocity
● Velocity describes how rapidly the data is generated and how quickly it
moves.
● This data flow comes from sources such as mobile phones, social media,
networks, servers, etc.
● For instance, a consumer with wearable gadget that has a sensor connected
to a network will keep gathering and sending data to the source.
Variety
● Variety describes the diversity of the data types and its heterogeneous sources.
● Big Data information draws from a vast quantity of sources,comes in three
different types
○ Structured Data: information with a defined length and format.
■ An Excel spreadsheet with customer names, e-mails, and cities is an
example of structured data.
○ Unstructured Data: information that can’t neatly fit in the rigid, traditional
row and column structure.
■ Unstructured data includes images, texts, and videos
○ Semi-structured Data: information that features like metadata.
■ This category includes emails, web pages, and TCP/IP packets.
Veracity
● Veracity describes the data’s accuracy and quality.
● Gathered data could have missing pieces, may be inaccurate or may not be
able to provide real, valuable insight.
● Veracity, overall, refers to the level of trust there is in the collected data.
Value
● The most important characteristic from the perspective of the business.
● The value of big data usually comes from insight discovery and pattern
recognition that lead to more effective operations, stronger customer
relationships and other clear and quantifiable business benefits.
● The more insights derived from the Big Data, the higher its value.
Big Data Platform

● Big Data platform is an IT solution which combines several Big Data tools and
utilities into one package for managing and analyzing Big Data.
● It is an enterprise class IT platform that enables organization in developing,
deploying,operating and managing a big data infrastructure /environment.
● There are several Open source and commercial Big Data Platform in the
market with varied features which can be used in Big Data environment.
● Big data platform generally consists of big data storage, servers, database,
big data management, business intelligence and other big data management
utilities.
● The primary benefit behind a big data platform is to reduce the complexity of
multiple vendors/ solutions into a one cohesive solution.
Features of Big Data Platform

● Big Data platform should be able to accommodate new platforms and tool
based on the business requirement.
● It should support linear scale-out
● It should have capability for rapid deployment.
● It should support variety of data format.
● Platform should provide data analysis and reporting tools.
● It should provide real-time data analysis software.
● It should have tools for searching the data through large data sets.
List of Big Data Platforms

a) Hadoop
b) Cloudera
c) Amazon Web Services
d) Hortonworks
e) MapR
f) IBM Open Platform
g) Microsoft HDInsight
h) Intel Distribution for Apache Hadoop
i) Datastax Enterprise Analytics
j) Teradata Enterprise Access for Hadoop
k) Pivotal HD
Big Data- challenges
● Storage

● Processing

● Security

● Data Validation and Finding and Fixing Data Quality Issues.

● Scaling Big Data Systems

● Evaluating and Selecting Big Data Technologies

● Real-Time Insights
Hadoop

● "Hadoop" refers to the entire ecosystem of open-source software projects that

provide a framework for distributed storage and processing of large data sets.
● It is an umbrella term that encompasses various components and tools
designed to work together to solve big data challenges.
● Hadoop includes both storage (Hadoop Distributed File System or HDFS)
and processing (MapReduce) frameworks, as well as additional components
for data management, processing, and analysis.
Apache Hadoop

● Apache Hadoop is Big Data platform and software package which is Apache
sponsored project.
● Under Apache Hadoop project various other software is being developed
which runs on the top of Hadoop system to provide enterprise grade data
management and analytics solutions to enterprise.
● Apache Hadoop is open-source, distributed file system which provides data
processing and analysis engine for analyzing large set of data.
● Hadoop can run on Windows, Linux and OS X operating systems, but it is
mostly used on Ubuntu and other Linux variants.
HDFS
● The Hadoop Distributed File System (HDFS) is a distributed file system designed to
store and manage very large data sets reliably and fault-tolerantly across multiple
commodity hardware nodes.
● It is a key component of the Apache Hadoop ecosystem and serves as the primary
storage system for data in Hadoop clusters.
● Key features and characteristics of HDFS include:
○ Distributed Storage
○ Fault Tolerance
○ Master-Slave Architecture
○ Streaming Data Access
○ Designed for Commodity Hardware
MapReduce

● The MapReduce engine was originally written by Google and this is the
system which enables the developers to write program which can run in
parallel on 100 or even 1000s of computer nodes to process vast data sets.
● After processing all the job on the different nodes it comes the results and
return it to the program which executed the MapReduce job.
● This software is platform independent and runs on the top of Hadoop
ecosystem.
● It can process tremendous data at very high speed in Big Data environment.
Hadoop – Architecture
Apache Spark

● Apache Spark is software that runs on the top of Hadoop and provides API for
real-time,in-memory processing and analysis of large set of stored in the
HDFS.
● It stores the data into memory for faster processing.
● Apache Spark runs program 100 times faster in-memory and 10 times faster
on disk as compared to the MapRedue.
● Apache Spark is here to faster the processing and analysis of big data sets in
Big Data environment.
● Apache Spark is being adopted very fast by the business to analyze their data
set to get real value of their data.
Big Data Analytics Life Cycle
Discovery

● The data science team learn and investigate the problem.

● Develop context and understanding.
● Identify data sources needed and available for the project.
● The team formulates initial hypothesis that can be later tested with data
Data Preparation

● Methods to investigate the possibilities of pre-processing and preparing data

before analysis and modelling.
● Data preparation tasks can be repeated and not in a predetermined
sequence.
● Some of the tools used commonly for this process include - Hadoop, Alpine
Miner, Open Refine, etc.
Model Planning

● The team studies data to discover the connections between variables, selects the

most significant variables as well as the most effective models.

● The data science teams create data sets that can be used for training and testing.

● Some of the tools used commonly for this stage are MATLAB and STASTICA.
Model Building

● The team builds and implements models based on the work completed in the
model planning phase.
● The team is also evaluating whether its current tools are sufficient to run the

models or if they require an even more robust environment to run models.

● Tools that are free or open-source or free tools Octave, WEKA.

● Commercial tools - MATLAB, STASTICA.

Communication Results

● Following the execution of the model, team members will need to evaluate
the outcomes of the model to establish criteria for the success or failure of the
model.
● The team is considering how best to present findings and outcomes to the
various members of the team and other stakeholders.
● The team should determine the most important findings, quantify their value
to the business and create a narrative to present findings and summarize
them to all stakeholders.
Operationalize

● The team distributes the benefits of the project to a wider audience.

● It sets up a pilot project that will deploy the work in a controlled manner prior to
expanding the project to the entire enterprise.
● This technique allows the team to gain insight into the performance and
constraints related to the model within a production setting at a small scale and
then make necessary adjustments before full deployment.
● The team produces the last reports, presentations, and codes.
● Open source or free tools such as WEKA, SQL, MADlib, and Octave
Big Data Architecture

● Big data solutions typically involve one or more of the following types of
workload:
○ Batch processing of big data sources at rest.
○ Real-time processing of big data in motion.
○ Interactive exploration of big data.
○ Predictive analytics and machine learning.
● A big data architecture is designed to handle the ingestion, processing,
and analysis of data that is too large or complex for traditional database
systems.
Big Data Architecture-Components
Most big data architectures include some or all of the following components:
Data sources
● All big data solutions start with one or more data sources.
● Examples include:
○ Application data stores, such as relational databases.
○ Static files produced by applications, such as web server log files.
○ Real-time data sources, such as IoT devices.
Data storage
● Data for batch processing operations is typically stored in a distributed file store
that can hold high volumes of large files in various formats (data lake).
● Options for implementing this storage include Azure Data Lake Store or blob
containers in Azure Storage.
Batch Processing

● It is a method computers use to periodically complete high-volume, repetitive

data jobs, such as backups, filtering, and sorting.
● It can be compute intensive and inefficient to run on individual data
transactions.
● Instead, data systems process such tasks in batches, often in off-peak times .
● For example
○ consider an ecommerce system that receives orders throughout the day. Instead
of processing every order as it occurs, the system might collect all orders at the
end of each day and share them in one batch with the order fulfillment team.
Real-time Message Ingestion

● If the solution includes real-time sources, the architecture must include a way
to capture and store real-time messages for stream processing.
● This might be a simple data store, where incoming messages are dropped
into a folder for processing. However, many solutions need a message
ingestion store to act as a buffer for messages, and to support scale-out
processing, reliable delivery, and other message queuing semantics.
● This portion of a streaming architecture is often referred to as stream
buffering. Options include Azure Event Hubs, Azure IoT Hub, and Kafka.
Stream Processing

● After capturing real-time messages, the solution must process them by

filtering, aggregating, and preparing the data for analysis.
● The processed stream data is then written to an output sink.
● Azure Stream Analytics provides a managed stream processing service
based on perpetually running SQL queries that operate on unbounded
streams.
● You can also use open source Apache streaming technologies like Spark
Streaming in an HDInsight cluster.
Machine Learning

● Reading the prepared data for analysis (from batch or stream processing),
machine learning algorithms can be used to build models that can predict
outcomes or classify data.
● These models can be trained on large datasets, and the resulting models can
be used to analyze new data and make predictions.
● This can be done using Azure Machine Learning
Analytical Data Store
● Many big data solutions prepare data for analysis and then serve the processed data
in a structured format that can be queried using analytical tools.
● The analytical data store is used to save the processed data.
Analytics & Reporting
● Analytics is the technique of examining data and reports to obtain actionable insights
that can be used to realise and improve business performance.
● To empower users to analyze the data, the architecture may include a data modeling
layer, such as a multidimensional OLAP cube or tabular data model in Azure Analysis
Services.
● It might also support self-service BI, using the modeling and visualization technologies
in Microsoft Power BI or Microsoft Excel.
● Analysis and reporting can also take the form of interactive data exploration by data
scientists or data analysts
Orchestration

● Most big data solutions consist of repeated data processing operations, that
transform source data, move data between multiple sources and sinks, load the
processed data into an analytical data store, or push the results straight to a
report or dashboard.
● Orchestration technology is used to automate these workflows, tools such as
Azure Data Factory or Apache Oozie and Sqoop are used for this purpose.
Advantages Of Big Data Processing

Ability to process Big Data brings in multiple benefits, such as-

● Businesses can utilize outside intelligence while taking decisions

Access to social data from search engines and sites like facebook, twitter are
enabling organizations to fine tune their business strategies.
● Improved customer service
Traditional customer feedback systems are getting replaced by new systems
designed with Big Data technologies.
In these new systems, Big Data and natural language processing technologies are
being used to read and evaluate consumer responses.
● Early identification of risk to the product/services, if any
● Better operational efficiency
Big Data technologies can be used for creating a staging area or landing
zone for new data before identifying what data should be moved to the data
warehouse.
In addition, such integration of Big Data technologies and data warehouse
helps an organization to offload infrequently accessed data.
Data warehouse

● A data warehouse is a central repository of information that can be analyzed

to make more informed decisions.
● Data flows into a data warehouse from transactional systems, relational
databases, and other sources, typically on a regular interval.
● Business analysts, data engineers, data scientists, and decision makers
access the data through business intelligence (BI) tools, SQL clients, and
other analytics applications.
● Data and analytics have become indispensable to businesses to stay
competitive.
● Business users rely on reports, dashboards, and analytics tools to extract
insights from their data, monitor business performance, and support decision
making.
● Data warehouses power these reports, dashboards, and analytics tools by
storing data efficiently to minimize the input and output (I/O) of data and
deliver query results quickly to hundreds and thousands of users concurrently.