MapReduce Architecture
Fundamentals of Big
Data Computing
A comprehensive exploration of distributed computing paradigms for processing
massive datasets across commodity hardware clusters in computer science
engineering.
Introduction to Big Data
Challenges and Solutions
The Big Data The MapReduce
Challenge Solution
Modern organisations generate MapReduce provides a distributed
petabytes of data daily4from social computing framework that:
media interactions to sensor
Parallelises computation across
readings. Traditional computing
clusters
systems struggle with:
Handles hardware failures
Volume: Processing terabytes of automatically
information efficiently
Scales horizontally with
Velocity: Real-time data stream commodity hardware
handling
Simplifies complex distributed
Variety: Structured and programming
unstructured data formats
Veracity: Ensuring data quality
and reliability
What is MapReduce: Core
Concepts and Philosophy
Divide and Conquer Functional
Programming
Break massive datasets into
smaller, manageable chunks that Inspired by map() and reduce()
can be processed independently functions, applying
across multiple machines transformations to data
simultaneously. collections without side effects
for predictable outcomes.
Data Locality
Move computation to data rather than data to computation, minimising
network overhead and maximising processing efficiency.
Developed by Google in 2004, MapReduce revolutionised how we approach
large-scale data processing. It abstracts complexity, allowing developers to focus
on business logic rather than distributed systems management.
MapReduce Programming Model: Map and Reduce
Functions
Input Data
Raw data split into independent chunks, each processed by separate mapper instances in parallel.
Map Phase
Transforms input records into key-value pairs. Each mapper processes its chunk independently, emitting intermediate results.
Shuffle & Sort
Groups all values associated with the same key together, preparing data for the reduce phase across the network.
Reduce Phase
Aggregates values for each key, producing final output. Multiple reducers process different keys concurrently.
Output
Final results written to distributed storage, ready for analysis or further processing pipelines.
Example: Word Count 3 Map emits (word, 1) for each occurrence; Reduce sums counts per word to produce final frequencies.
Hadoop Ecosystem: MapReduce Implementation
Framework
HDFS MapReduce Engine
Hadoop Distributed File System stores Core processing framework that executes
data across cluster nodes with replication map and reduce jobs across the
for fault tolerance. distributed cluster.
Ecosystem Tools YARN
Hive, Pig, HBase, and Spark provide Yet Another Resource Negotiator
higher-level abstractions and specialised manages cluster resources and schedules
processing capabilities. application workloads efficiently.
Apache Hadoop emerged as the open-source implementation of MapReduce, becoming the de facto standard for big data processing. Its
ecosystem provides comprehensive tools for ingestion, storage, processing, and analysis at massive scale.
MapReduce Architecture Components: JobTracker
and TaskTracker
JobTracker (Master)
Single master node coordinating entire MapReduce job execution:
Receives job submissions from clients
1
Schedules tasks across TaskTrackers
Monitors task progress and health
Handles task failures and re-execution
TaskTracker (Slave)
Multiple worker nodes executing assigned tasks:
Spawns map and reduce task processes
2
Reports progress via heartbeat signals
Manages local task execution environment
Handles data locality for efficient processing
Note: In Hadoop 2.0+, YARN's ResourceManager and NodeManager replace JobTracker and TaskTracker, providing improved
scalability and resource utilisation.
Data Flow in MapReduce: From Input to Output
Processing
01 02
Input Splitting Map Task Execution
HDFS divides large files into fixed-size blocks (typically 128MB), Each split processed by a mapper, transforming input records into
distributed across cluster nodes with replication factor of 3. intermediate key-value pairs stored in local memory buffers.
03 04
Partitioning Shuffle and Sort
Intermediate data partitioned by key using hash function, Framework transfers partitioned data to reducers over the network,
determining which reducer receives each key-value pair for sorting by key to group all values for efficient aggregation.
processing.
05 06
Reduce Task Execution Output Consolidation
Reducers process sorted groups, applying aggregation logic to Multiple reducer outputs merged into final result set, available for
produce final output written back to HDFS for persistence. querying, visualisation, or downstream processing pipelines.
Fault Tolerance and
Reliability Mechanisms in
MapReduce
Data Replication Heartbeat Monitoring
HDFS maintains three copies of TaskTrackers send periodic
each data block across different heartbeats to JobTracker. Missing
nodes, ensuring availability even heartbeats trigger automatic task
when hardware fails reassignment to healthy nodes
catastrophically. within seconds.
Task Re-execution
Failed tasks automatically restarted on different nodes. Speculative
execution runs duplicate tasks to mitigate slow-running stragglers.
Checkpoint & Master Node
Recovery Resilience
Intermediate results periodically While JobTracker represents a single
saved to disk. Upon failure, tasks point of failure in MapReduce v1,
resume from last checkpoint rather modern implementations use high-
than restarting completely, availability configurations with
minimising wasted computation. standby masters for seamless
failover.
Performance Optimisation Techniques and Best
Practices
Combiner Functions Data Compression Custom Partitioning
Mini-reducers executed on mapper output, Compress intermediate and output data Implement intelligent partitioning strategies
performing local aggregation before shuffle using codecs like Snappy or LZO, reducing ensuring balanced data distribution across
phase to dramatically reduce network traffic. I/O overhead and network bandwidth reducers, preventing skewed processing
consumption. bottlenecks.
Memory Tuning Input Format Selection Monitoring & Profiling
Configure heap sizes, buffer Choose appropriate InputFormat Leverage JobTracker web UI, logs,
allocations, and sort parameters classes (TextInputFormat, and profiling tools to identify
based on data characteristics and SequenceFileInputFormat) bottlenecks, tune configurations,
available cluster resources. matching data structure for optimal and optimise job performance
parsing efficiency. iteratively.
Real-world Applications and Case Studies in
Industry
Google: Search Indexing Facebook: Social Analytics
Processes billions of web pages daily, building inverted indexes Analyses user interactions, friend connections, and content
for lightning-fast search results across the entire internet. engagement across 3 billion users for personalised news feeds
and advertisements.
Amazon: Recommendation Engine Healthcare: Genomic Analysis
Processes purchase history and browsing patterns across Sequences and analyses DNA data from genome projects,
millions of products, generating personalised identifying disease markers and accelerating pharmaceutical
recommendations driving 35% of revenue. drug discovery pipelines.
Financial Services Telecommunications
Banks leverage MapReduce for fraud detection, risk modelling, and Mobile operators analyse call detail records and network logs to
algorithmic trading, processing millions of transactions per second optimise infrastructure, predict equipment failures, and enhance
with sub-millisecond latency requirements. customer experience through data-driven insights.