Welcome to AIKTC
Module 1| Introduction to Big Data
Prof. Nusrat Jahan,
Department of Computer Engineering.
AIKTC – Anjuman-I-Islam’s Kalsekar Technical Campus.
Module 1 : Introduction to Big Data and Hadoop(Syllabus)
➢ Introduction to Big Data
➢ Big Data characteristics and
➢ Types of Big Data
➢ Traditional vs. Big Data business approach
➢ Case Study of Big Data Solutions
➢ Concept of Hadoop, Core Hadoop Components; Hadoop Ecosystem
A.I. Kalsekar Technical Campus, New Panvel.
Topic : Introduction to Big Data
What is Data?
The quantities, characters, or symbols on which operations are performed by a
computer, which may be stored and transmitted in the form of electrical
signals and recorded on magnetic, optical, or mechanical recording media.
What is Big Data?
Big Data is a collection of data that is huge in volume, yet growing
exponentially with time. It is a data with so large size and complexity that
none of traditional data management tools can store it or process it efficiently.
Big data is also a data but with huge size.
A.I. Kalsekar Technical Campus, New Panvel.
Topic : Example of Big Data
1. New York Stock Exchange
The New York Stock Exchange is an example of Big Data that generates about one
terabyte of new trade data per day.
2. Social Media
The statistic shows that 500+terabytes of new data get ingested into the databases of social
media site Facebook, every day. This data is mainly generated in terms of photo and video
uploads, message exchanges, putting comments etc.
3. Jet engine
A single Jet engine can generate 10+terabytes of data in 30 minutes of flight time. With
many thousand flights per day, generation of data reaches up to many Petabytes.
4. Government Data: Government agencies generate large volumes of data related to demographics,
public health, transportation, education, and more. Analyzing this data can aid in policy-
making, resource allocation, and improving public services.
A.I. Kalsekar Technical Campus, New Panvel.
Example of Big Data…..
5. Weather and Climate Data: Meteorological agencies collect extensive data on temperature,
humidity, atmospheric pressure, wind patterns, and more. Analyzing this data helps in
weather forecasting, climate modeling, and understanding long-term climate trends.
6. Internet of Things (IoT): The proliferation of IoT devices, such as smart home appliances,
wearables, and industrial sensors, generates enormous data streams. Analyzing IoT data
can provide insights into energy consumption patterns, predictive maintenance, and
optimizing resource allocation.
7. Financial Data: Banks, credit card companies, and stock exchanges generate massive amounts
of data related to transactions, customer behavior, market trends, and risk analysis.
Analyzing this data can enable fraud detection, algorithmic trading, and customer
segmentation.
8. E-commerce Data: Online shopping platforms generate enormous volumes of data,
including customer transactions, browsing history, product reviews, and clickstream data.
Analyzing this data can help in personalized marketing, recommendation systems, and
inventory management.
Types Of Big Data
➢ Structured
➢ Unstructured
➢ Semi-structured
A.I. Kalsekar Technical Campus, New Panvel.
Examples Of Structured Data
An ‘Employee’ table in a database is an example of Structured Data
Emp_ID Emp_Name Gender Department Salary_In_lacs
236 Rajesh Kulkarni Male Finance 650000
339 Pratibha Joshi Female Admin 650000
746 Shushil Roy Male Admin 500000
750 Shubhojit Das Male Finance 500000
769 Priya Sane Female Finance 550000
A.I. Kalsekar Technical Campus, New Panvel.
Examples Of Un-Structured Data
The output returned by ‘Google Search’
A.I. Kalsekar Technical Campus, New Panvel.
Semi-structured Data
➢ Semi-structured data refers to data that does not fit neatly into traditional structured formats like
relational databases but still has some form of organization and can be processed with additional
efforts. Example of semi-structured data:
JSON (JavaScript Object Notation):
{
"name": "John Smith",
"age": 30,
"email": "john@[Link]",
"address": {
"street": "123 Main Street", ➢ Semi-structured data like JSON is commonly used in web
"city": "Anytown", development, data interchange formats, and NoSQL
"state": "California", databases where the structure can vary between different
"postalCode": "12345" documents or records.
}
}
Example Of Semi-structured Data
XML (eXtensible Markup Language): XML is another widely used semi-structured data format. It uses
tags to define elements and their hierarchical relationships.
Personal data stored in an XML file-
<rec><name>Prashant Rao</name><sex>Male</sex><age>35</age></rec>
<rec><name>Seema R.</name><sex>Female</sex><age>41</age></rec>
<rec><name>Satish Mane</name><sex>Male</sex><age>29</age></rec>
<rec><name>Subrato Roy</name><sex>Male</sex><age>26</age></rec>
<rec><name>Jeremiah J.</name><sex>Male</sex><age>35</age></rec>
A.I. Kalsekar Technical Campus, New Panvel.
Data Growth over the years
A.I. Kalsekar Technical Campus, New Panvel.
Topic : Characteristics Of Big Data
➢ Volume
➢ Variety
➢ Velocity
➢ Veracity
➢ Value
A.I. Kalsekar Technical Campus, New Panvel.
Topic :
A.I. Kalsekar Technical Campus, New Panvel.
Topic
A.I. Kalsekar Technical Campus, New Panvel.
Topic : Big Data Case Studies
➢ Walmart
➢ Uber
➢ Netflix
➢ eBay
➢ Procter & Gamble
A.I. Kalsekar Technical Campus, New Panvel.
What is HADOOP?
Topic : What is Hadoop?
➢ Apache Hadoop is an open source software framework used to develop data processing applications
which are executed in a distributed computing environment.
➢ It was developed by Doug Cutting and Mike Cafarella in the year 2005 and is now maintained by
the Apache Software Foundation.
➢ Applications built using HADOOP are run on large data sets distributed across clusters of
commodity computers.
➢ Commodity computers are cheap and widely available. These are mainly useful for achieving
greater computational power at low cost.
➢ Similar to data residing in a local file system of a personal computer system, in Hadoop, data
resides in a distributed file system which is called as a Hadoop Distributed File system.
➢ Hadoop provides a reliable, scalable, and fault-tolerant platform for big data processing.
A.I. Kalsekar Technical Campus, New Panvel.
Hadoop Architecture
Java library and
Job Scheduling
utilities(JavaScript)
and resource
management
A.I. Kalsekar Technical Campus, New Panvel.
Hadoop Architecture contd…
Hadoop Distributed File System (HDFS): HDFS is a distributed file system that stores data
across multiple machines in a Hadoop cluster. It provides high throughput access to data and is
designed to handle large files and streaming data.
MapReduce: MapReduce is a programming model and processing framework for distributed
computing. It allows you to process and analyze large datasets in parallel across a Hadoop cluster.
YARN (Yet Another Resource Negotiator)
YARN is a Framework on which MapReduce works. YARN performs 2 operations that are Job
scheduling and Resource Management. The Purpose of Job schedular is to divide a big task into
small jobs so that each job can be assigned to various slaves in a Hadoop cluster and Processing can
be Maximized.
Hadoop common or Common Utilities
Hadoop common or Common utilities are nothing but our java library and java files or we can say
the java scripts that we need for all the other components present in a Hadoop cluster. these utilities
are used by HDFS, YARN, and MapReduce for running the cluster.
Topic : Hadoop Ecosystem and Components
A.I. Kalsekar Technical Campus, New Panvel.
Topic : Hadoop Architecture
A.I. Kalsekar Technical Campus, New Panvel.
Topic : Features Of ‘Hadoop’
➢ Suitable for Big Data Analysis
➢ Hadoop is Open Source
➢ Hadoop cluster is Highly Scalable
➢ Hadoop provides Fault Tolerance
➢ Hadoop provides High Availability
➢ Hadoop is very Cost-Effective
➢ Hadoop is Faster in Data Processing
➢ Hadoop is Easy to use
A.I. Kalsekar Technical Campus, New Panvel.
Topic : Some Important previous year questions
1. Explain Hadoop ecosystem with core components. Explain its physical
Architecture.
2. Explain Big data and its characteristics.
3. What is Hadoop? How big data and Hadoop are linked?
4. What are the types of big data? Explain with example.
A.I. Kalsekar Technical Campus, New Panvel.
Keep Learning, Keep Growing…
Your feedback is valuable to us, so please keep giving it.
You can reach us at
[Link]
and can contact at
@[Link]
A.I. Kalsekar Technical Campus, New Panvel.