- In Word Count, the Combiner can aggregate the counts of the

same word emitted by different Mappers.

- For instance, if Mapper 1 emits (hello, 1) and Mapper 1 also
emits (hello, 1), the Combiner can combine these into (hello, 2).
6. Shuffle and Sort Phase:
9. Counter Phase: - Intermediate key-value pairs generated by the Mappers are shuffled
across the network and sorted based on the keys.
- Counters are used to keep track of various statistics
For example, all occurrences of the word "hello" emitted by
-

and metrics during the MapReduce job execution.

different Mappers are sent to the same Reducer.
For example, a counter might track the total number
7. Reduce Partitioner:
of words processed or the number of unique words
encountered. The Reduce Partitioner determines which Reducer will process each
-

intermediate key-value pair based on the key.

10. Output Data:
In Word Count, the Partitioner ensures that all occurrences of the
-

- Finally, the output of the Reducer phase is written to

same word are sent to the same Reducer.

the distributed file system as the final result of the 8. Reducer Phase:
MapReduce job.
Each Reducer processes the intermediate key-value pairs assigned
InWord Count, the output might be key-value pairs to it. It typically applies a Reduce function.
where the key is a word, and the value is its total
For Word Count, the Reducer the counts of the same word
-
sums up
count across all input documents.
across different documents, producing the final count for each
word.

# Fault Tolerance im Map

N Reduce
Answer: Fault tolerance mechanisms play a crucial role in the MapReduce framework to ensure the reliability
and robustness of distributed data processing. MapReduce is designed to handle large-scale data processing tasks
across clusters of commodity hardware, where failures are common. The fault tolerance mechanisms in

MapReduce are primarily aimed at detecting and mitigating failures that may occur during the execution of Map
and Reduce tasks. Here's an analysis ofthe key fault tolerance mechanisms in MapReduce:

1. Task Redundancy:

MapReduce creates redundant copies of tasks and assigns them to multiple nodes in the cluster. If
a task fails to execute successfully on one node, it can be rerun on another node that hosts a
redundant copy.

Task redundancy ensures that failures of individual nodes or tasks do not impact the overall
progress of the MapReduce job.
2. Task Monitoring and Re-execution:

MapReduce monitors the progress of individual tasks and detects failures by tracking heartbeats
and status updates from task trackers.

If a task fails or exceeds its allotted execution time, MapReduce reschedules the task to be
executed on another available node.

Task re-execution ensures that failed tasks are retried on different nodes, reducing the likelihood
of job failure due to node failures or transient errors.

3. Data Replication:
MapReduce replicates input data blocks across multiple nodes in the cluster to ensure data
availability and fault tolerance.

By maintaining multiple copies of data blocks, MapReduce can recover from node failures or data
corruption by accessing replicated copies from other nodes.
Data replication also facilitates data locality, where tasks can be executed closer to the data they
process, improving overall performance.

4. Checkpointing:
Checkpointing is a mechanism used to periodically save the intermediate state of tasks to durable
storage, such as HDFS or distributed file systems.

In the event of a task failure, MapReduce can restart the task from the last checkpointed state
rather than rerunning the entire task from scratch.

Checkpointing reduces the computational overhead and improves the efficiency of recovering
from failures, especially for long-running tasks.

5. Speculative Execution:

MapReduce employs speculative execution to identify and mitigate straggler tasks, ie, tasks that
are significantly slower than others due to factors such as resource contention or hardware
failures.

When a task is detected to be running slower than expected, MapReduce launches a speculative
copy of the task on another node.
 stems like HDFS.

Apache Storm Architecture:

Twitter API) and emit tuples into the topology.

Master
6. Bolt: Bolts process tuples emitted by spouts or other bolts in the topology. They perform various
Nimbus Zookeeper operations such as filtering, aggregation, and transformation.
Cluster
Designinga Storm Topology for Processing Sensor Data:

Worker Nodes Let's design a Storm topology to process incoming streams of sensor data:

1. Spout: Create a spout to read sensor data fromamessage queue like Apache Kafka.
Supervisor Supervisor Supervisor Supervisor Supervisor
2. Bolts:

Workers Worker
Data Parsing Bolt: Parse the incoming sensor data and extract relevant information (e.g. sensor

Worker Node 1 Worker Worker Node 4

ID, timestamp, sensor readings).
Worker Node 3 Worker Node 5

Filtering Bolt: Filter out irrelevant data or apply quality checks to ensure data integrity.

Fille system/Database Aggregation Bolt: Aggregate sensor data over a time window (eg. 1 minute, 5 minutes) to

ralculate statistics like average, maximum, minimum readings.

Apache Storm architecture consists of the following key compon Alerting Bolt: Detect anomalies or trigger alerts based on predefined thresholds or patterns in the
Nimbus: Nimbus is the maste
Storm, responsible for sensor data.
assigning tasks to workers, and monitoring their health.
Topalogy: Define the data flow between spouts and bolts to process sensor data sequentially or in
2. Supervisor: Supervisors run on worker nodes and are responsible worker parallel,based on the processing requirements and performance considerations.
processes

Deployment: Deploy the Storm topology on a Storm cluster, ensuring that it is configured to scale
3. Worker: Each worker process executes a subset of a topology's tasks, which units o

computation in Storm. dynamically based on the incoming data volume and processing requirements.

Topology: A topology is a directed acyelic graph (DAG) of spouts and bolts that define the data flow and
processing logic.

5.
Spout: Spouts are the source of data streams in Storm. They ingest data from external sources (eg, Kafka,

storm казка
The official inventor of Storm is The official inventor of Kafka is
Twitter. LinkedIn.

It is a real time message processing It is a distributed message

platform. processing platform.

It is not used to store the data while, it

It is used to maintain the local file
acts as a mediator and transfer the
system and store the data
data.

By RTU PRONOTES Dercloper Team..

It does not contain any external It contains dependence on
dependency. Zookeeper to run the Kafka server.

Himank Knabya 22 ESKCX0 45

It provides the feature of Security. It does not offer any security of data.
Ansn kuatod 22 ESKCS029
The latency power offered by Apache The latency power offered by
Ansn Kasat 22 ESKCS028
Storm is generally less than 1-2
seconds.
Apache Kafka is generally
milliseconds.
is in
Vaibnat Upadnayay 22 ESKCX119
Harsh Mumar T22 ESKCx088
it provides the capability to be used It is mostly used with the Java
with any programming language. programming language.

Features like scalability and

It is easy to use and operate.
durability are offered.

It is written by using language like It is written by using language like

Clojure and Java. Scala and JVM.