Distributed System
Distributed System
Ans: A Distributed Database Management System (DDBMS) is the software that manages
these distributed databases and ensures that they function as a single coherent system.
Here are the key components and features of a Distributed Database System within a
DDBMS:
Key Components
1. Distributed Databases:
o Fragmentation: The database is divided into smaller pieces called fragments,
which can be stored at different sites. Fragmentation can be horizontal (rows)
or vertical (columns).
o Replication: Copies of the same data are stored at multiple sites to enhance
reliability and availability.
o Allocation: Deciding where each fragment or replica will be stored.
2. Sites:
o These are individual locations or nodes in the network where the fragments of
the database are stored. Each site can be a computer system with its own local
DBMS.
3. Network:
o The communication infrastructure that connects the sites. This could be a
Local Area Network (LAN), Wide Area Network (WAN), or the internet.
1. Data Distribution:
o Data is distributed across multiple locations to improve accessibility,
performance, and reliability.
2. Transparency:
o Location Transparency: Users do not need to know the location of the data.
o Replication Transparency: Users are unaware of the replication of data.
3. Autonomy:
o Each site operates autonomously and can manage its own database
independently.
4. Scalability:
o The system can be scaled easily by adding more sites or databases.
5. Fault Tolerance:
o The system can continue to operate in the event of a failure at one or more
sites due to replication and distributed control.
6. Concurrency Control:
o Managing simultaneous operations on the distributed database to ensure
consistency.
7. Query Processing:
o Efficiently processing queries across the distributed databases, often requiring
the coordination of data retrieval from multiple sites.
8. Distributed Transactions:
oTransactions that involve multiple sites. Ensuring ACID (Atomicity,
Consistency, Isolation, Durability) properties across distributed transactions is
a complex task.
9. Data Consistency:
o Ensuring that all copies of the data remain consistent, despite updates
occurring at different sites.
Advantages
• Improved Performance: Data can be processed in parallel and accessed from the
nearest location, reducing latency.
• Reliability and Availability: Replication and distribution enhance the system’s fault
tolerance and availability.
• Scalability: The system can grow by adding more nodes or databases without
significant restructuring.
Challenges
• Local Data Access: Distributed databases allow data to be stored closer to where it is
needed, reducing latency and improving response times for local users.
• Parallel Processing: Distributed systems can perform parallel query processing,
leveraging multiple servers to handle different parts of a query simultaneously, thus
speeding up data retrieval and processing.
2. Scalability
4. Data Localization
• Local Autonomy: Different sites can have control over their own data, which can be
particularly useful for large organizations with distinct departments or geographical
divisions.
• Regulatory Compliance: Data can be stored in specific geographic locations to
comply with local data protection laws and regulations.
• Distributed Access: Users from different locations can access and share data without
significant delays, fostering better collaboration across geographically dispersed
teams.
• Data Integration: Organizations can integrate data from various sources and
locations more seamlessly.
6. Cost Efficiency
8. Load Balancing
While there are numerous benefits, there are also challenges associated with distributed
databases:
• Complexity: Managing and maintaining a distributed database system is more
complex than a centralized one.
• Security: Ensuring consistent security policies and measures across all sites can be
challenging.
• Scope: Spans large geographical areas, such as cities, countries, or even continents.
• Characteristics:
o Slower data transfer rates compared to LAN due to longer distances.
o Higher latency.
o Often relies on third-party carriers or telecommunications providers.
• Examples: The internet, leased lines, MPLS (Multiprotocol Label Switching).
• Scope: Covers a larger area than a LAN but smaller than a WAN, typically a city or a
large campus.
• Characteristics:
o Intermediate data transfer rates and latency.
o Usually owned by organizations or municipalities.
• Examples: Fiber optic networks used by city governments or large campuses.
• Scope: Can span both small and large geographical areas depending on the
implementation.
• Characteristics:
o Each node acts as both a client and a server.
o Decentralized, with no single point of failure.
o Nodes share resources directly with each other.
• Examples: BitTorrent, decentralized file sharing networks.
5. Client-Server Network
• Scope: Can be implemented over LAN, MAN, or WAN.
• Characteristics:
o Central servers provide resources and services to client machines.
o Centralized control with dedicated servers.
o Can be easily managed and secured.
• Examples: Traditional web applications, email servers.
6. Wireless Networks
1. Horizontal Fragmentation
Horizontal fragmentation involves dividing a table into subsets of rows, where each fragment
contains a subset of the rows of the table. Each subset is typically based on a condition or a
predicate that applies to the rows.
2. Vertical Fragmentation
Vertical fragmentation involves dividing a table into subsets of columns, where each
fragment contains a subset of the columns of the table. Each subset includes the primary key
to ensure that the original table can be reconstructed if needed.
[Link]-Join Programs?
Ans: A semi-join operation is used to join two tables, but unlike a full join, it only returns
rows from the first table that have matching rows in the second table, without including the
columns from the second table.
Purpose of Semi-Join
The main purpose of using semi-join operations in DDBMS is to minimize data transfer and
improve query performance in distributed environments.
1. Local Selection: Apply a local selection on one of the tables to reduce the number of
rows that need to be considered for the join.
2. Projection: Project the join attributes from the selected rows and send them to the
other site.
3. Remote Selection: Use the received join attributes to perform a selection on the
remote table.
4. Transfer Matching Rows: Send the matching rows (or the necessary attributes) from
the remote site back to the local site.
5. Final Join: Perform the join operation using the reduced sets of data.
• Reduced Data Transfer: Only the necessary attributes are sent across the network,
reducing communication costs.
• Improved Performance: Less data transfer means faster query execution times,
especially in networks with limited bandwidth.
1. Query Decomposition
• Parsing and Translation: Validate the query and convert it into a canonical form.
• Normalization: Break the query into smaller, optimizable components.
2. Data Localization
• Fragment Localization: Identify the relevant data fragments needed for the query.
• Access Path Determination: Determine the optimal paths to access these fragments.
• Join Ordering: Determine the optimal order for joining tables to minimize
intermediate results and communication costs.
• Semi-Join Strategies: Use semi-joins to reduce data transfer by only sending
necessary attributes.
• Local Execution Plans: Generate optimal execution plans for local operations at each
site.
• Index Utilization: Use indexes to speed up local data retrieval.
• Combine Local Plans: Integrate local execution plans into a coordinated global
execution plan.
• Coordination and Synchronization: Ensure efficient data flow and synchronization
across sites.
6. Cost Estimation
• Cost Model Development: Develop a cost model including communication, disk I/O,
and processing costs.
• Cost Evaluation: Estimate the total cost of different execution strategies to find the
most efficient one.
7. Optimization Techniques
• Query Execution: Execute the optimized global plan across distributed sites.
• Performance Monitoring: Collect performance data to monitor execution.
1. Consistency
• Ensure data consistency and enforce integrity constraints throughout the transaction.
2. Atomicity
• Guarantee that all parts of a transaction are completed successfully or none at all,
maintaining an all-or-nothing approach.
3. Isolation
4. Durability
• Ensure that once a transaction is committed, its changes are permanently recorded,
even in the case of a system failure.
• Detect and recover from various failures to maintain the robustness of the transaction
system, using recovery protocols to restore consistent database states.
6. Availability
• Maintain high system availability for transaction processing, even in the presence of
failures, through data replication and distributed commit protocols.
• Distributed Locking:
• Versioning: Maintain multiple versions of data items to allow concurrent read and
write operations without blocking.
• Snapshot Isolation: Ensure transactions see a consistent snapshot of the database at
the start of execution.
• Quorum-Based Techniques:
1. Deadlock Detection
• Wait-for Graph: Each site maintains a wait-for graph to track dependencies among
transactions waiting for locks.
• Global Deadlock Detection: Centralized deadlock detection algorithms periodically
gather information from all sites to construct a global wait-for graph and detect cycles
indicating deadlocks.
2. Deadlock Prevention
• Lock Ordering: Enforce a strict order in which transactions acquire locks to prevent
circular waits. This requires transactions to request locks in a predefined global order.
• Avoidance Algorithms: Use algorithms that ensure transactions acquire locks in a
way that avoids potential circular waits, such as Banker's algorithm.
3. Deadlock Resolution
[Link] of Failure?
Ans: Failures in Distributed Database Management Systems (DDBMS) can be categorized
into several types, each affecting the system's operations differently. Here is a concise
summary of the types of failures:
1. Transaction Failures
• Logical Errors: Errors within the transaction, such as violating integrity constraints.
• System-Defined Abort: System aborts a transaction due to reasons like deadlock
detection.
2. System Failures
3. Site Failures
• Single Site Failure: Failure of one site, while the rest of the system may continue to
operate.
• Multiple Site Failures: Failures affecting multiple sites simultaneously, posing a
greater challenge for recovery.
4. Network Failures
5. Media Failures
6. Human-Caused Failures
• Operational Errors: Mistakes by operators, such as incorrect configuration or
accidental data deletion.
• Malicious Actions: Intentional harmful actions like hacking or sabotage.
1. Redundancy and Replication: Store data copies at multiple sites and use redundant
hardware components.
2. Backup and Recovery: Regular data backups and robust recovery protocols.
3. Fault Tolerance: Implement failover mechanisms and transaction rollback
procedures.
4. Distributed Commit Protocols: Use protocols like Two-Phase Commit (2PC) to
ensure transaction atomicity.
5. Network Resilience: Design robust network infrastructure and implement partition
detection and handling strategies.
• Description: 3PC extends 2PC to address some of its limitations, aiming to improve
fault tolerance and reduce the blocking problem.
• Phases:
o Phase 1 (Can Commit):
▪ Coordinator asks participants if they can commit (similar to Phase 1 in
2PC).
▪ Participants respond with "Yes," "No," or "Ready" (prepared and can
commit).
o Phase 2 (Pre-Commit):
▪ Coordinator sends a pre-commit request to all participants that
responded "Yes" in Phase 1.
▪ Participants acknowledge their readiness to commit.
o Phase 3 (Commit/Abort):
▪ Coordinator sends a commit or abort request to all participants based
on their responses in Phase 1 and Phase 2.
▪ Participants then commit or abort the transaction accordingly.
[Link] Architectures ?
Ans: 1. Shared-Nothing Architecture
• Description: Each node (server) in the distributed system operates independently
with its own memory and storage.
• Communication: Nodes communicate via a network, typically exchanging messages
to coordinate distributed transactions or data access.
• Advantages:
o Scalability: Easily scales by adding more nodes without requiring shared
resources.
2. Shared-Disk Architecture
• Description: Multiple nodes (servers) share a common storage (disk) where data is
stored and accessed.
• Communication: Nodes access data through a shared storage system (e.g., SAN or
NAS).
• Advantages:
o Data Centralization: Simplifies data management as all nodes access a
centralized data repository.
3. Shared-Everything Architecture
• Description: All nodes share both storage and memory, operating as a single unified
system.
• Communication: Nodes communicate directly with shared resources (e.g., shared
memory or database).
• Advantages:
o Data Integration: Provides a unified view of data across all nodes.
4. Hybrid Architectures
[Link] Technique ?
Ans: Here's a concise overview of reliability techniques in DDBMS:
1. Replication
2. Fault-Tolerant Protocols
• Purpose: Implement protocols that can detect and recover from failures.
• Examples:
o Two-Phase Commit (2PC): Ensures atomicity of distributed transactions
across multiple nodes.
o Three-Phase Commit (3PC): Enhances 2PC by adding a third phase to
improve fault tolerance.
• Advantages:
o Transaction Integrity: Ensures all or none of the distributed transactions
commit.
3. Redundancy
4. Load Balancing
• Purpose: Periodically save the state of the database to facilitate faster recovery after a
failure.
• Types:
o Deferred Update: Only record the database state changes in the checkpoint.
o Immediate Update: Immediately update the database state and then record
the changes.
2. Logging
• Process: Roll back transactions that were active at the time of failure using the log to
undo their effects.
• Steps:
o Identify transactions that were active but not yet committed or checkpointed.
o Use the log to reverse their changes by applying undo operations.
• Process: Reapply changes to the database that were committed but not yet reflected in
the data files due to failure.
• Steps:
o Identify committed transactions whose changes were not permanently stored
(due to failure).
o Use the log to reapply their changes to the database to ensure consistency.
5. Undo-Redo Logging
• Combined Approach: Use both undo and redo operations recorded in the log to
ensure complete recovery.
• Advantages: Enhances fault tolerance and ensures that all transactions maintain
ACID properties (Atomicity, Consistency, Isolation, Durability).
• Purpose: Ensure that transactions commit atomically across multiple distributed sites.
• Protocols:
o Two-Phase Commit (2PC): Coordinator sends prepare messages to all
participants, then commits or aborts based on their responses.
o Three-Phase Commit (3PC): Enhances 2PC by adding a third phase to
ensure robustness against failures during commit.
7. Recovery Manager
1. Consistency
• Data Consistency: Ensuring the directory is up-to-date and consistent across all
nodes.
• Synchronization: Managing updates and synchronization across distributed sites.
2. Scalability
• Directory Size: Handling the increasing size and complexity of the directory as the
system grows.
• Performance: Ensuring efficient lookups and updates in large-scale systems.
3. Availability
• Fault Tolerance: Keeping the directory available despite failures.
• Replication: Managing consistency and updates in replicated directory instances.
4. Maintenance
5. Security
6. Latency
7. Complexity of Queries
Solutions
1. Partitioning the Directory: Divide into smaller, manageable parts for better
scalability and performance.
2. Caching: Store frequently accessed information locally to reduce lookup times.
3. Consistency Protocols: Use distributed consensus algorithms to maintain
consistency.
4. Hierarchical Directories: Implement a multi-level directory structure for efficient
management.
5. Dynamic Updates: Develop algorithms for efficient directory updates.
• Speedup: Reduce the overall query execution time by distributing tasks across
multiple nodes.
• Scalability: Handle larger datasets and increased query loads by adding more nodes.
• Efficiency: Maximize resource utilization across the distributed system.
1. Data Partitioning
o Horizontal Partitioning: Distribute rows of a table across different nodes.
o Vertical Partitioning: Distribute columns of a table across different nodes.
o Hash Partitioning: Use a hash function to evenly distribute data across nodes.
2. Parallel Execution Strategies
o Intra-Query Parallelism: Execute different parts of a single query in parallel.
▪ Pipeline Parallelism: Different stages of a query are executed
concurrently.
▪ Partitioned Parallelism: The same operation (e.g., a join) is
performed in parallel on different data partitions.
o Inter-Query Parallelism: Execute multiple queries simultaneously on
different nodes.
3. Query Decomposition
o Fragment Queries: Break down a query into sub-queries that operate on data
fragments.
o Task Assignment: Distribute sub-queries to appropriate nodes for execution.
4. Distributed Joins
o Fragment and Replicate: Replicate smaller tables to all nodes and perform
local joins.
o Partitioned Join: Distribute join operations across nodes based on
partitioning criteria.
5. Parallel Sorting and Aggregation
o Parallel Sorting: Use parallel algorithms to sort data distributed across nodes.
o Parallel Aggregation: Perform aggregation operations (e.g., SUM, COUNT)
in parallel on distributed data.