Chapter 1
Main Variety of Data Sources
Social Media Sources
Facebook, Instagram, Twitter, WhatsApp
Likes, comments, shares, posts, images, videos
Transactional Data
Banking transactions
Online shopping payments
Ticket bookings
Used by Businesses-
Social Media Data
Likes, shares, comments, mentions
Business use:
sentiment analysis and targeted marketing
sentiment analysis is the process of understanding how people feel about a brand by analyzing
their opinions expressed in data like:
social media posts
comments
reviews
tweets
Walmart is a multinational retail company and one of the largest retailers in the world.
Walmart handles huge volumes of customer, sales, and inventory data, so it uses Big Data
analytics to:
manage stock
predict demand
improve customer service
data storage units
Kilobyte (KB)
→ Megabyte (MB)
→ Gigabyte (GB)
→ Terabyte (TB)
→ Petabyte (PB)
→ Exabyte (EB)
→ Zettabyte (ZB)
Volume
Huge amount of data generated every day.
Facebook stores billions of photos and videos uploaded by users.
Velocity
Speed at which data is generated and processed.
Online payments and stock market data are processed in real time.
Clickstream is the record of all clicks and actions a user performs while browsing a website or mobile
app.
What it captures
Pages visited
Buttons clicked
Time spent on pages
Search actions
Add-to-cart and checkout steps
Variety
Different types and formats of data.
Veracity
Accuracy and reliability of data.
Fake reviews on shopping websites reduce data quality.
Value
Useful information extracted from data.
Netflix uses viewing data to recommend movies, increasing customer satisfaction.
Types of big data
Structured Data
Data that is organized in a fixed format (rows and columns) and stored in databases.
Student records (Roll No, Name, Marks)
Tool used: RDBMS (MySQL, Oracle)
Semi-Structured Data
Data that does not have a fixed table structure, but contains tags or markers to organize information.
XML files Tags (<name>, <age>, <price>)
Different XML Files with Different Tags
<student>
<name>Ravi</name>
<rollno>12</rollno>
<marks>85</marks>
</student>
<student>
<name>Asha</name>
<class>TE IT</class>
<email>asha@[Link]</email>
</student>
JSON data (Key–value pairs)
Emails (Header = structured Body = unstructured)
Tool used: NoSQL databases
Unstructured Data
Data that has no predefined structure.
Social media posts
Images, videos
Traditional Data
Batch processing is a method where data is collected over a period of time, stored together, and then
processed all at once instead of immediately.
Small to moderate amount of data
Mostly structured (tables, rows, columns)
Stored in RDBMS (MySQL, Oracle)
Batch processing
Big Data
Very large volume of data (TBs, PBs, ZBs)
Structured + semi-structured + unstructured data
Stored in HDFS, NoSQL databases
A Hadoop cluster is a group of connected computers (nodes) that work together to store and
process Big Data using the Hadoop framework.
Instead of one powerful machine, Hadoop uses many low-cost machines working as a team.
Advantages of big data
Scalability
When data increases, Hadoop does not slow down
we can add more computers to the cluster
In Traditional RDBMS
Data must be cleaned and structured first
Fixed schema (rows & columns) is required
Unstructured data cannot be stored easily
In big data
no schema is required. (Predefined structure)
Technologies Available for Big Data
Big Data technologies are tools and frameworks used to store, process, manage, and analyze huge
volumes of data efficiently.
Storage Technologies
HDFS (Hadoop Distributed File System)
Stores large data across multiple machines
NoSQL Databases
Handle unstructured & semi-structured data
Processing Technologies
MapReduce
Challenges
Data Storage
Huge volume of data requires large and scalable storage systems.
Data Processing
Processing massive data quickly, especially in real time, is difficult.
Data Variety
Handling structured, semi-structured, and unstructured data together is complex.
Data Quality (Veracity)
Data may be incomplete, noisy, or incorrect.
Data Security & Privacy
Protecting sensitive data from unauthorized access.
Cost Management
Infrastructure, tools, and skilled professionals are expensive.
Case studies of Big Data Solutions
Clickstream Example
Website informs public about various products and services. Click stream
data is generated when the customers/visitors interact through website.
Some organizations allow people to transact through their website. These
data include the pages they load, the time spent by them on each page,
the links they clicked, the frequency of their visit, from which page do
they exit, etc. This gives good amount of information about their
customers and helps in understanding them better. Eventually website
content, its organization, navigation and transaction completion can be
improved. This ensures that the customers can easily find what they
want. The website owner can understand the customers’ buying behavior.