Name – Raj Bhuva BDA Assignment Enroll no – 201310132060
Q1 - Explain different types of NoSQL data architecture patterns.
NoSQL, which stands for "Not Only SQL," encompasses a variety of database technologies that
differ from traditional relational databases (SQL databases) in terms of data models, consistency,
scalability, and more. NoSQL databases are designed to handle large volumes of unstructured or
semi-structured data, offering greater flexibility and performance for certain use cases. Here are
the main types of NoSQL data architecture patterns:
1. Document Stores:
Imagine a system that stores documents in a way similar to how we store files on a
computer. These documents can be in various formats like JSON or XML and can contain a
mix of different types of information. This type of NoSQL database is useful when we have
data that doesn't fit neatly into tables (like in traditional databases) and is more flexible and
adaptable to changes.
2. Key-Value Stores:
Think of a key-value store as a big dictionary where you can quickly look up values using a
unique key. The 'key' is like a word, and the 'value' is its definition. It's straightforward and
efficient, great for situations where you need speedy access to specific pieces of
information.
3. Column-Family Stores (Wide-Column Stores):
Picture organizing data in a way where each 'row' can have a varying number of 'columns,'
and each 'column' can hold different types of information. It's like having a table where
each row can have a different set of columns based on what you need to store. This
structure is excellent for handling a large amount of data efficiently.
4. Graph Databases:
Imagine mapping out relationships between various entities, like people in a social
network. Each person is a 'node,' and the connections between them (friendship, follow,
etc.) are the 'edges.' Graph databases are excellent for understanding and querying
complex relationships, making them perfect for social networks, recommendation systems,
and more.
5. Time-Series Databases:
Picture tracking data over time, like temperature readings every minute or stock prices
every second. Time-series databases excel at handling this kind of data, making it easy to
analyze trends, patterns, and anomalies over time. They are vital in IoT applications,
financial analysis, and monitoring systems.
Name – Raj Bhuva BDA Assignment Enroll no – 201310132060
Each of these NoSQL data architecture patterns has its strengths and weaknesses, and the choice
of the appropriate type depends on the specific requirements of the application. Often, a hybrid
approach or utilizing multiple types of NoSQL databases in a polyglot persistence strategy can
provide the best solution for diverse data management needs.
Q2 – Explain working of various phases of MapReduce with appropriate example
and diagram.
MapReduce is a programming model and computational framework designed for processing and
generating large-scale data sets with parallel, distributed algorithms on a cluster of commodity
hardware. It involves two main phases: the Map phase and the Reduce phase. Let's explain each
phase with an appropriate example and a diagram.
1. Map Phase:
The Map phase is the first step in a MapReduce job. In this phase, input data is divided into smaller
sub-problems, and each sub-problem is independently processed. The Map phase typically
comprises two main steps: splitting and mapping.
A. Splitting: Input data is divided into manageable chunks, each of which is processed
independently. These chunks are called input splits. The size of an input split can vary
depending on the Hadoop configuration.
B. Mapping: For each input split, the MapReduce program applies a user-defined map
function to process the data. The map function processes each record independently
and emits key-value pairs based on the processing logic. The output of the map function
is called intermediate key-value pairs.
Example: Let's consider a simple example of word count. In this case, the map function
processes each word in a document and emits key-value pairs, where the word is the
key and the count is the value.
2. Shuffle and Sort:
After the Map phase, the intermediate key-value pairs are passed through a process known as
Shuffle and Sort. In this phase, the MapReduce framework groups and sorts the intermediate key-
value pairs based on the keys. This step is essential to ensure that all values corresponding to a
particular key are brought together.
Name – Raj Bhuva BDA Assignment Enroll no – 201310132060
3. Reduce Phase:
The Reduce phase is the second step in a MapReduce job. In this phase, the grouped and sorted
intermediate key-value pairs are passed to the reduce function, which aggregates and processes
the data based on the keys.
A. Reducing: The reduce function is a user-defined function that processes each group of
intermediate values associated with the same key. The function can perform various
operations, such as summing up the values, finding the maximum, or any custom
computation based on the application's requirements.
Example: In the word count example, the reduce function sums up the counts for each
word to provide the total count of occurrences for each word.
MapReduce Flow Diagram:
In summary, MapReduce operates in a distributed and parallelized manner, allowing for efficient
processing of large-scale data sets. The Map phase divides and processes the data, the Shuffle and
Sort phase organizes the results, and the Reduce phase aggregates and produces the final output.
Together, these phases enable scalable and effective data processing.
Q3 – Draw and explain HDFS architecture.
Hadoop Distributed File System (HDFS) is the primary storage system used by Hadoop applications.
It is designed to store and manage very large files across a cluster of commodity hardware. Here's
an overview of its architecture:
Name – Raj Bhuva BDA Assignment Enroll no – 201310132060
1. Block Storage:
• HDFS divides a file into fixed-size blocks (e.g., 128MB or 256MB). Each block is
stored across the cluster of machines.
• Blocks are the basic units of storage in HDFS and are managed by the NameNode
and DataNodes.
2. NameNode:
• The NameNode is the master node in the HDFS cluster.
• It stores metadata about the file system, including the directory structure, file
permissions, and the mapping of blocks to DataNodes.
• However, it does not store the actual data or the block contents.
• The metadata is usually stored in memory for fast access.
3. DataNodes:
• DataNodes are worker nodes in the HDFS cluster.
• They store the actual data blocks and are responsible for serving read and write
requests from the clients.
• DataNodes periodically send heartbeat signals and block reports to the NameNode
to provide information about the blocks they are storing.
4. Block Replication:
• HDFS replicates each block multiple times (typically three times by default) across
the cluster.
• Replication provides fault tolerance and ensures data reliability. If a block or a node
fails, the data can still be accessed from other replicas.
Name – Raj Bhuva BDA Assignment Enroll no – 201310132060
5. Rack Awareness:
• HDFS is rack-aware, meaning it understands the network topology and organizes
block replicas to minimize network traffic and increase fault tolerance.
• It tries to place replicas on different racks to ensure data resilience even if a whole
rack or a network link goes down.
6. Client:
• Clients interact with the HDFS to read, write, and manage files.
• When a client wants to read or write a file, it contacts the NameNode to get
information about the locations of the blocks.
• Subsequently, the client directly interacts with the appropriate DataNodes to read
or write the data.
7. Secondary NameNode:
• Despite its name, the Secondary NameNode does not act as a backup to the
NameNode.
• It periodically merges the edit log with the current fsimage (a snapshot of the file
system metadata) to reduce the NameNode's startup time in case of failure.
• It does not provide high availability on its own.
In summary, HDFS provides a robust and fault-tolerant storage system for big data applications. It
achieves this through block storage, replication, rack awareness, and efficient metadata
management using the NameNode and DataNodes.
Q4 – Explain role of ZooKeeper.
ZooKeeper is a distributed coordination service that plays a crucial role in managing and
synchronizing distributed systems. It acts as a centralized repository for configuration information,
distributed synchronization, and group services within a distributed environment. Let's delve into
the key roles and functionalities of ZooKeeper:
1. Configuration Management: ZooKeeper is widely used for centralized configuration
management. In a distributed system, various components often need to share
configuration settings. ZooKeeper provides a reliable way to store and manage these
configurations, allowing distributed nodes to dynamically retrieve and update configuration
data.
2. Synchronization and Locking: Distributed systems require synchronization and
coordination among nodes to maintain consistency and avoid conflicts. ZooKeeper provides
primitives like locks, semaphores, and barriers, enabling distributed processes to
synchronize their actions and share access to resources in a controlled manner.
Name – Raj Bhuva BDA Assignment Enroll no – 201310132060
3. Group Services: ZooKeeper facilitates the creation of groups and memberships. It allows
processes to join a group, detect members' presence, and react accordingly. This feature is
essential for scenarios where a distributed system needs to organize nodes into logical
groups for efficient communication and coordination.
4. Leader Election: ZooKeeper provides a robust mechanism for leader election in a
distributed environment. In scenarios like high availability or fault tolerance, where a group
of nodes needs to elect a leader to manage the operations, ZooKeeper helps achieve a
consensus on leader selection.
5. Event Notification: ZooKeeper allows clients to register for event notifications triggered
by changes in the distributed system's state. Clients can receive notifications when a node's
data changes, a node is added or removed, or other relevant events occur. This event-
driven model helps applications react to changes promptly and adapt to the evolving
distributed system.
6. High Availability: ZooKeeper is designed to ensure high availability and fault tolerance. It
achieves this through replication, where data is stored on multiple nodes (ensemble). If a
node fails, the system continues to function, ensuring reliability and uninterrupted service.
7. Sequential Node Naming: ZooKeeper provides a feature to assign a sequential name to
each node created in a certain znode. This feature is crucial in scenarios like distributed
queuing, where multiple nodes are contending for resources, and order of access matters.
Q5 – Difference between Master Slave and Peer to Peer Architecture.
Master-Slave and Peer-to-Peer are two fundamental architectural models used in various
computing systems, each with its own set of characteristics and use cases. Let's explore the
differences between these architectures in detail.
Master-Slave Architecture:
Name – Raj Bhuva BDA Assignment Enroll no – 201310132060
In a Master-Slave architecture, the system is designed around a central, controlling element known
as the "Master," which directs and delegates tasks to other connected devices or processes called
"Slaves." The Master node governs and coordinates the overall operation of the system.
1. Control Hierarchy:
• Master Node: Central control, responsible for decision-making and task delegation.
• Slave Nodes: Carry out assigned tasks, report results, and do not make high-level
decisions.
2. Task Distribution:
• The Master assigns tasks and distributes the workload to Slave nodes based on their
capabilities and resources.
• Slaves execute tasks independently and report back to the Master.
3. Communication:
• Communication primarily occurs between the Master and Slaves.
• The Master initiates communication and directs the flow of data and instructions.
4. Fault Tolerance:
• The system can be more fault-tolerant, as failures in a Slave node do not necessarily
disrupt the entire system.
• Redundancy and failover mechanisms can be implemented to ensure continuous
operation.
5. Scalability:
• Scalability can be limited, as the Master can become a bottleneck as the number of
Slave nodes increases.
• Adding more Slave nodes might require modifications in the Master node to handle
the load effectively.
Peer-to-Peer (P2P) Architecture:
Name – Raj Bhuva BDA Assignment Enroll no – 201310132060
In a Peer-to-Peer architecture, all nodes, known as "Peers," have equal status and are capable of
both requesting and providing services. Each node can act as both a client and a server, sharing
resources and responsibilities.
1. Equal Status:
• All nodes have equal status and can initiate communication and share resources
directly with one another.
• Each node can independently request and provide services to other nodes.
2. Resource Sharing:
• Peers can share resources (like files, processing power, or bandwidth) directly with
other peers in a decentralized manner.
• Resources are distributed across the network, enhancing efficiency and
performance.
3. Communication:
• Communication can occur directly between any pair of peers without relying on a
central authority.
• Peers collaborate and exchange information without a single point of control.
4. Fault Tolerance:
• P2P networks tend to be more resilient to node failures, as the network can still
function even if some peers are unavailable.
• No single point of failure ensures better fault tolerance.
5. Scalability:
• P2P architectures are highly scalable, as adding more peers to the network
increases resources and can improve performance without a central bottleneck.