Module 1 – Understanding Big Data
Introduction
Big Data refers to extremely large and complex datasets that traditional data processing
software cannot efficiently handle. These datasets are generated from diverse sources such as
sensors, social media, business transactions, and IoT devices.
Big Data is not just about volume; it represents an ecosystem of technologies and
methodologies for collecting, storing, managing, and analyzing data to extract actionable
insights. It fuels AI, machine learning, and data-driven decision-making, often earning the
nickname "the new oil" due to its strategic business value.
Traditional Data vs. Big Data
Feature Traditional Data Big Data
Type Structured Structured, Semi/Un-structured
Storage Relational Databases (SQL) Distributed Systems, NoSQL, Data Lakes
Volume MB to GB TB to PB or more
Processing Tools SQL, Statistical Methods Hadoop, Spark, ML, stream processing
Scalability Vertical Horizontal (distributed)
Big Data Ecosystem Evolution
1990s: Initial use of structured data in relational databases.
2000s: Explosion of unstructured and semi-structured data due to the internet and digital
devices.
Modern Era:
Hadoop: Introduced distributed storage & processing (HDFS & MapReduce).
Apache Spark: Brought in-memory distributed computation.
Cloud Platforms: Enabled elastic scaling and on-demand infrastructure.
5. Big Data Architecture
1. Data Collection:
Tools: Apache Kafka, Apache NiFi
Sources: Logs, IoT, APIs, transactions, social media
2. Data Storage:
Data Lakes: Raw data in native format; flexible and scalable.
Data Warehouses: Structured, processed data; optimized for analytics.
Data Lakehouses: Hybrid of lake and warehouse functionalities.
3. Data Processing:
Batch: Hadoop MapReduce
Real-Time: Apache Spark, Flink
ETL/ELT Pipelines: Data transformation and movement
4. Analytics & Visualization:
Tools: Python, R, Tableau, Power BI
Techniques: Machine Learning, Statistical Analysis, Predictive Modeling
6. Key Technologies in Big Data
Hadoop: Distributed file system (HDFS) and processing framework (MapReduce)
Apache Spark: Fast, general-purpose cluster computing with in-memory capabilities
NoSQL Databases: MongoDB, Cassandra, Redis—optimized for scalability and flexibility
7. Benefits of Big Data
Informed Decision-Making: Predictive insights for strategic actions
Customer Personalization: Targeted marketing, recommendation systems
Operational Efficiency: Process optimization, predictive maintenance
Product Development: Feedback-driven innovation
Dynamic Pricing: Real-time demand-based price adjustments
Risk & Fraud Detection: Behavioral anomaly detection in finance, cybersecurity
Healthcare Advancements: Real-time monitoring, personalized treatment planning
8. Challenges in Big Data
Data Quality: Noise, duplicates, inconsistencies
Scalability: Infrastructure must grow with data
Data Security & Privacy: Compliance with regulations like GDPR, HIPAA
Integration Complexity: Consolidating disparate sources
Skill Shortage: Demand for data scientists, engineers, analysts
9. Big Data in AI & ML
Modern AI systems, especially deep learning models, require massive amounts of data to learn
representations and make predictions. Big Data provides:
• Volume for deep model training
• Variety for generalization
• Veracity for reliable outputs
Technologies like large language models (LLMs) and recommendation engines are built on
massive datasets sourced and processed using Big Data tools and frameworks.
Classification of Big Data
Data type Definition Characteristics Example
Structured Data Data that conforms to a - Follows data models - Relational databases
fixed schema (organized - Stored in tables (SQL)
into rows and columns). (RDBMS) - Spreadsheets
- Supports CRUD - Chess match move
operations tables
- ACID compliant
- Easy indexing and
retrieval
Semi-structured Data Data that does not follow - Tag-based hierarchy - XML documents
strict schema but contains - No fixed schema like - JSON files
tags or markers to RDBMS - Emails with headers
separate data elements. - Contains metadata - Sensor data with
- Easier to organize than structure
unstructured data
Multi-structured Data Combination of multiple - Found in big data - Streaming data (social
formats: structured, semi- systems media + sensor logs)
structured, and - Aggregated from - Chess records
unstructured. various sources (structured tables) +
- High variety in formats match reviews/blogs
(unstructured)
- Web logs + transaction
records
Unstructured Data Data without predefined - Lacks relational - Social media posts
format or schema; does structure (tweets, comments)
not reside in traditional - Difficult to analyze - Videos (YouTube)
databases. directly - Website content
- Needs - Text messages, chat
metadata/schemas logs
created separately - Images, traffic video,
- Often voluminous atmospheric data
Distributed file system
In the realm of Big Data Analytics, understanding distributed file systems (DFS) is
fundamental. A DFS is essentially a network of interconnected computers, or nodes, where
each node stores a piece of a larger dataset. This architecture allows users to access and process
vast amounts of data from any location within the network, providing significant advantages
in terms of security and reliability.
How Distributed File Systems Work
DFS operates by creating a virtual file system spread across multiple individual
computers. Each computer maintains its own local file system, holding a specific segment of
the overall data. When a user requests a file, the request is directed to a file server. This server
then orchestrates the retrieval of the necessary data from the various computers holding its
parts. This process, often involving data replication, is crucial for ensuring data
integrity and security by creating multiple copies of the data across different nodes.
The Benefits of Using a Distributed File System in Big Data
For Big Data analytics, the benefits of employing a DFS are substantial:
• Enhanced Reliability: By distributing data across numerous computers, a DFS
significantly reduces the risk of data loss, making it highly resilient to individual node
failures.
• Improved Performance: Data replication across multiple machines enables parallel
access and retrieval, drastically improving data processing speeds, which is critical for
large datasets.
• Increased Storage Capacity: Organizations can seamlessly expand their storage
capacity by simply adding more nodes, eliminating the need for expensive hardware
upgrades on single machines.
• Improved Scalability: A DFS inherently supports scaling storage needs upwards
without requiring a complete overhaul of the existing infrastructure.
• Easier Storage Management: Data can be stored in multiple locations and accessed
in a distributed manner, simplifying the management of complex data landscapes.
• Increased Security: Centralized data access and storage across multiple computers in
a DFS can reduce the risk of data loss or theft by spreading the risk.
• Increased Availability: Data can be made highly available to organizations as it can
be stored in various locations, ensuring continuous access even if some nodes are
offline.
• Improved Cost-Efficiency: By optimizing data storage and access, a DFS can lead to
significant cost savings compared to maintaining multiple, disparate storage
systems. Encrypting data during replication further enhances both security and
reliability.
Types of Distributed File Systems
Distributed file systems can be broadly categorized into three types, each with its own
characteristics:
1. Local Area Network (LAN) Based DFS
A LAN-based DFS is confined to a specific geographical area. Within this network,
computers can share resources like files, printers, and applications. In a Big Data context, a
LAN DFS connects remote computers to the system, allowing users to access files regardless
of their physical location within that specific network. This is particularly advantageous in
corporate environments with multiple physical locations needing shared access to replicated
files, ensuring everyone is working with the most current information. Similarly, in a home
setting, a LAN DFS allows multiple computers to access shared files without the need for
manual transfers.
2. Wide Area Network (WAN) Based DFS
A WAN-based DFS is designed for storing and accessing data over a large geographical
area. The WAN acts as a bridge, enabling access to the same file from any node across the
network by routing data using specific protocols. It forms the backbone of the distribution file
system, ensuring secure and reliable data transmission. Common WAN networks used in DFS
include Ethernet, Fibre Channel, and Token Ring, each offering varying levels of performance
and security that need to be considered for Big Data applications.
3. Cloud-Based Distributed File Systems
Cloud-based DFS leverage the internet to store and access data. Prominent examples
include Amazon S3, Microsoft Azure, and Google Cloud Storage. These systems store files
across a distributed network of computers, rather than a single machine. This not only improves
performance by distributing the load but also enhances security by spreading files across
multiple locations. To use a cloud-based DFS, files are uploaded to a remote server (private,
secure storage, or public cloud), after which they become accessible from any device within
the distributed network.
Examples of Distributed File Systems in Big Data
Several prominent distributed file systems are crucial for Big Data analytics:
• Apache Hadoop Distributed File System (HDFS): Developed specifically for
processing massive datasets, HDFS is an open-source platform that enables the
storage and analysis of vast amounts of distributed data. It finds extensive applications
in e-commerce, healthcare, and financial sectors.
• GlusterFS: Known for its scalability and high performance, GlusterFS is an open-
source distributed file system. It's a clustered file system that utilizes commodity
hardware to create a single, large, high-performance storage pool.
• Google File System (GFS): Used internally by Google to store its data, GFS is
engineered for massive scalability and handling high volumes of data traffic. It is
highly reliable and fault-tolerant, making it suitable for Google's vast data needs.
• Ceph: An open-source distributed storage system, Ceph is built on an object
storage model, where data is treated as a collection of objects rather than traditional
files and directories. Ceph is a highly scalable and fault-tolerant system capable of
storing petabytes of data, making it ideal for Big Data environments.
Big data and its importance
According to Gartner, the definition of Big Data –
“Big data” is high-volume, velocity, and variety information assets that demand cost-effective,
innovative forms of information processing for enhanced insight and decision making.” In
other terms, Big Data refers to complex and large data sets that have to be processed and
analysed to uncover valuable information that can benefit businesses and organizations.
However, there are certain basic tenets of Big Data that will make it even simpler to answer
▪ It refers to a massive amount of data that keeps on growing exponentially with time.
▪ It is so voluminous that it cannot be processed or analysed using conventional data
▪ processing techniques.
▪ It includes data mining, data storage, data analysis, data sharing, and data visualization.
▪ The term is an all-comprehensive one including data, data frameworks, along with the
tools and techniques used to process and analyze the data.
With the digital transformation of industries, the generation of real-time and historical data has
exploded. Big data analytics plays a crucial role in extracting actionable insights to drive
decision-making, optimize operations, and foster innovation. It enables organizations to:
▪ Improve customer engagement through personalization.
▪ Predict trends and future outcomes.
▪ Streamline supply chains.
▪ Identify inefficiencies and hidden costs.
▪ Innovate products and services based on real-time feedback.
Five V’s – Drivers for big data
• Volume – Refers to the enormous amount of data generated.
• Velocity – Describes the speed at which data is generated and processed.
• Variety – Covers the diverse formats of data: text, images, videos, logs, etc.
• Veracity – Concerns data quality, consistency, and trustworthiness.
• Value – The actionable insights that data can generate to drive decision-making.
Big data analytics
Big Data Analytics refers to the systematic application of advanced analytic techniques to large
and complex data sets—characterized by high volume, velocity, and variety. These datasets
often originate from web platforms, mobile apps, IoT devices, social media, and enterprise
systems, and include structured, semi-structured, and unstructured formats. Traditional
analytics tools are insufficient to manage this scale and complexity, necessitating specialized
tools and infrastructures.
The typical analytics pipeline includes the following stages:
1. Collection: Data is acquired from various sources (sensors, applications, transactions,
social media) and stored in a centralized repository—either a data lake (raw,
unstructured data) or a data warehouse (processed, structured data).
2. Processing: Data is sorted, filtered, and transformed to enhance query performance.
Frameworks like Apache Spark and Hadoop MapReduce facilitate this step.
3. Scrubbing: Ensures data quality by resolving conflicts, removing duplicates, and
correcting format inconsistencies.
4. Analysis: Utilizes data mining, machine learning, AI, and statistical methods to detect
patterns, build predictive models, and generate insights.
Big Data Analytics empowers organizations to transform data into strategic assets. By
leveraging sophisticated tools and frameworks, businesses can uncover deep insights, automate
decision-making, and build resilient, data-centric operations across all verticals. Understanding
its full ecosystem—from data ingestion to actionable intelligence—is critical to maximizing
value in the modern digital economy.
Big data applications
Big Data has revolutionized how industries operate by offering tools to analyze massive
volumes of structured and unstructured data in real time. In the banking and securities sector,
for example, financial institutions face challenges like fraud detection, customer churn, and
risk management. By leveraging predictive analytics and real-time monitoring systems, they
can identify fraudulent transactions as they occur and make data-driven investment decisions.
Regulatory bodies like the U.S. Securities and Exchange Commission (SEC) also use big data
to monitor financial markets and detect irregular trading patterns.
In the communications, media, and entertainment industry, companies must cater to diverse
audiences across devices and platforms. Big data helps in content recommendation, audience
segmentation, and real-time personalization. For instance, media platforms analyze user
engagement to recommend music, shows, or news based on individual preferences. Events like
Wimbledon even use sentiment analysis to gauge public reactions on social media.
The healthcare industry deals with fragmented data from various sources like clinical records,
wearable devices, and insurance forms. Big data enables hospitals and research institutions to
integrate these datasets, leading to better clinical decisions, patient tracking, and epidemic
forecasting. Public health policies, such as those implemented during the Obamacare initiative,
rely heavily on big data analytics for population health management.
In the field of education, big data is helping administrators and instructors personalize learning
and track student progress. Universities, such as the University of Tasmania, use learning
management systems to monitor students’ online activities and provide timely interventions.
At the same time, data is being used to evaluate teaching methods and improve curriculum
effectiveness.
Manufacturing and natural resources sectors face the challenge of underutilized operational
and sensor data. Big data analytics allows companies to optimize supply chains, manage
inventories, and predict equipment failures. In oil, gas, and mining, seismic data and historical
production records are analysed to enhance exploration strategies and safety protocols.
Government agencies leverage big data to improve public services, ensure national security,
and combat fraud. By analyzing social programs, tax records, and citizen feedback,
governments make data-informed decisions. For example, the Social Security Administration
and Homeland Security use big data to detect fraud and analyze risk patterns, while the Food
and Drug Administration uses it to evaluate health data trends.
In the insurance industry, companies traditionally relied on generic risk assessments, but big
data allows for customized offerings based on behaviour. Telematics data from vehicles,
weather forecasts, and real-time claim information are now used to set premiums, detect
fraudulent claims, and predict future risks. Similarly, companies like Climate Corp use weather
and soil data to offer specialized crop insurance to farmers.
The retail and wholesale sectors benefit from big data by optimizing inventory, understanding
consumer preferences, and improving customer loyalty programs. Data from point-of-sale
terminals, online browsing behavior, and loyalty cards is used to fine-tune marketing strategies
and personalize discounts. Retailers also analyze footfall and seasonal trends to allocate staff
efficiently and reduce checkout fraud.
In transportation, big data helps companies analyze traffic conditions, manage logistics, and
improve safety. For instance, airlines and courier companies analyze route data and weather
patterns to optimize scheduling and fuel consumption. Autonomous vehicles use big data from
sensors and cameras to make split-second driving decisions.
The energy and utilities industry uses big data in smart grid technologies, allowing real-time
monitoring of energy usage, predictive maintenance of infrastructure, and efficient load
distribution. Utility providers also use historical consumption data to detect outages and
prevent system failures.
Emerging sectors like autonomous vehicles and the Internet of Things (IoT) heavily depend
on big data. These technologies rely on massive real-time data streams from embedded sensors
to make intelligent decisions — for example, a smart home thermostat adjusting based on
occupancy patterns or an autonomous car detecting nearby objects in milliseconds.
In marketing, big data is used to design hyper-personalized campaigns. Customer behavior is
tracked across channels to deliver the right message at the right time. Marketers use A/B testing
and sentiment analysis to adjust campaigns dynamically and improve ROI.
Finally, business intelligence (BI) is a foundational application of big data that cuts across all
industries. Through dashboards, visualizations, and statistical models, organizations use BI to
monitor KPIs, detect anomalies, and forecast trends. From daily operations to long-term
strategy, big data enhances decision-making by transforming raw data into actionable insights.
Algorithms using MapReduce
MapReduce is a powerful programming model developed by Google for processing vast
volumes of data in parallel across a distributed cluster of machines. It provides a highly
scalable, fault-tolerant method to perform large-scale data computations, and forms the core of
the Hadoop ecosystem. The model is centered around two fundamental operations - Map,
which processes input data and transforms it into intermediate key-value pairs, and Reduce,
which merges and aggregates these intermediate values to produce the final output.
Phase Input Output
Mapper (k,v) (k,v)
Shuffle & Sort (k,v) (k,list(v))
Reducer (k,list(v)) (k,v)
Input processing without and with Combiner
Several classical and modern data processing algorithms can be restructured within the
MapReduce framework. These implementations demonstrate its ability to handle problems
ranging from basic search and indexing to complex machine learning and graph processing
tasks.
• Sorting:
MapReduce is inherently suitable for sorting large datasets. The map phase can emit
key-value pairs where the key is the value to be sorted, and the reduce phase can
simply output these sorted keys.
• Word Count:
This is a classic example where the map phase emits (word, 1) pairs for each word
encountered, and the reduce phase sums the counts for each unique word.
• Inverted Index:65
Used in search engines, the map phase emits (word, document_ID) pairs, and the
reduce phase groups these by word, creating a list of document IDs where each word
appears.
• Graph Algorithms:
PageRank: Iteratively calculates the importance of nodes (web pages – number
and quality of links) in a page (graph). Each iteration involves map and reduce
steps to update page ranks based on incoming links. The underlying assumption
is that more important websites are likely to receive more links from other
websites.
Breadth-First Search (BFS) / Single Source Shortest Path (SSSP): Can be
implemented by iteratively expanding visited nodes and updating distances in the
map and reduce phases.
• Data Mining Algorithms:
Apriori Algorithm (for Frequent Itemset Mining): MapReduce can be used to
efficiently generate frequent item sets by distributing the candidate generation and
support counting across the cluster.
K-Means Clustering: The map phase assigns data points to clusters, and the
reduce phase recalculates cluster centroids. This process is repeated iteratively
until convergence.
• Text Processing and Analysis:
Sentiment Analysis: Mapping text to sentiment scores and reducing to aggregate
sentiment.
Topic Modeling: Analyzing large text corpora to identify underlying themes.
Keyword Extraction: Identifying important keywords from documents.
• Log Analysis:
Processing large volumes of log data for anomaly detection, trend analysis, and
performance monitoring.
The core idea in implementing these algorithms with MapReduce involves expressing the
problem in terms of key-value pairs and defining the mapfunction to process input and emit
intermediate key-value pairs, and the reduce function to aggregate values associated with the
same key.
Matrix-Vector multiplication by MapReduce
MapReduce is a technique in which a huge program is subdivided into small tasks and run
parallelly to make computation faster, save time, and mostly used in distributed systems. It has
2 important parts:
1. Mapper: It takes raw data input and organizes into key, value pairs. For example, In a
dictionary, you search for the word "Data" and its associated meaning is "facts and
statistics collected together for reference or analysis". Here the Key is Data and the
Value associated with is facts and statistics collected together for reference or analysis.
2. Reducer: It is responsible for processing data in parallel and produce final output.
Let us consider the matrix multiplication example to visualize MapReduce. Consider the
following matrices:
1 2 5 6
𝐴= [ ] ,𝐵 = ⌊ ⌋
3 4 7 8
Here matrix A is a 2x2 matrix which means the number of rows(i)=2 and the number of
columns(j)=2. Matrix B is also a 2x2 matrix where number of rows(j)=2 and number of
columns(k)=2. Each cell of the matrix is labelled as Aij and Bjk. Eg: element 3 in matrix A is
called A21 i.e. 2nd-row 1st column. Now One step matrix multiplication has 1 mapper and 1
reducer. The Formula is:
Mapper for Matrix A (k, v)=((i, k), (A, j, Aij)) for all k
Mapper for Matrix B (k, v)=((i, k), (B, j, Bjk)) for all I
Non-square matrix multiplication
1 0 2
𝐴= [ ] ,𝐵 = [ ]
3 2 1