Discuss in detail about distributed system models

Distributed and cloud computing systems are built using large-scale, interconnected autonomous
computer nodes. These nodes are linked through Storage Area Networks (SANs), Local Area
Networks (LANs), or Wide Area Networks (WANs) in a hierarchical manner.

The four major distributed system families are:

• Clusters – Connected by LAN switches, forming tightly coupled systems with hundreds of
machines.

• Grids – Interconnect multiple clusters via WANs, allowing resource sharing across thousands
of computers.

• P2P Networks – Form decentralized, cooperative networks with millions of nodes, used in file
sharing and content distribution.

• Cloud Computing – Operates over massive data centers, delivering on-demand computing
resources at a global scale.

These systems exhibit high scalability, enabling web-scale computing with millions of interconnected
nodes. Their technical and application characteristics vary based on factors such as resource sharing,
control mechanisms, and workload distribution.

Comparison of All Distributed System Models

Feature Clusters P2P Networks Grids Cloud Platforms

Flexible network of Heterogeneous

Network of compute Virtualized cluster
client machines clusters
nodes interconnected of servers over
Architecture logically connected interconnected by
by SAN, LAN, or data centers via
by an overlay high-speed network
WAN SLA
network links

Homogeneous nodes Autonomous client Dynamic resource

Centralized control,
with distributed nodes, free in and provisioning of
Control server-oriented with
control, running out, with self- servers, storage,
authenticated security
UNIX or Linux organization and networks

Web search, utility

High-performance Distributed
File sharing, content computing, and
computing, search supercomputing,
Applications delivery, and social outsourced
engines, and web global problem
networking computing
services solving
services

Google search Google App

Gnutella, eMule, TeraGrid, GriPhyN,
Representative engine, Sun Blade, Engine, IBM
BitTorrent, Napster, UK EGEE, D-Grid,
Systems IBM Road Runner, Bluecloud, AWS,
KaZaA, Skype ChinaGrid
Cray XT4 Microsoft Azure
Explain the basic cluster architecture with a neat diagram

Cluster Architecture — Server Clusters and System Models

Server Clusters and Interconnection Networks

Server clusters consist of multiple interconnected computers using high-bandwidth, low-latency
networks like Storage Area Networks (SANs), Local Area Networks (LANs), and InfiniBand.
These clusters are scalable, allowing thousands of nodes to be connected hierarchically.

• Clusters are connected to the Internet via a VPN gateway, which assigns an IP address to
locate the cluster.

• Each node operates independently, with its own OS, creating multiple system images (MSI).

• The cluster manages shared I/O devices and disk arrays, providing efficient resource
utilization.

Single-System Image (SSI)

An ideal cluster should merge multiple system images into a Single-System Image (SSI), where all
nodes appear as a single powerful machine.
• SSI is achieved through middleware or specialized OS support, enabling CPU, memory, and
I/O sharing across all cluster nodes.

• Clusters without SSI function as a collection of independent computers rather than a unified
system.

Hardware, Software, and Middleware Support

• Cluster nodes consist of PCs, workstations, or servers, interconnected using Gigabit

Ethernet, Myrinet, or InfiniBand.
• Linux OS is commonly used for cluster management.

• Message-passing interfaces (MPI, PVM) enable parallel execution across nodes.

• Middleware supports features like high availability (HA), distributed memory sharing
(DSM), and job scheduling.

• Virtual clusters can be dynamically created using virtualization, optimizing resource

allocation on demand.

Major Cluster Design Issues

Feature Functional Characterization Feasible Implementations

Availability and Hardware and software support for Failover, failback, checkpointing,
Support sustained HA in cluster rollback recovery, nonstop OS

Hardware Fault Automated failure management to Component redundancy, hot swapping,

Feature Functional Characterization Feasible Implementations

Tolerance eliminate all single points of failure RAID, multiple power supplies

Achieving SSI at functional level with Hardware mechanisms or middleware

Single System
hardware, software, middleware, or support to achieve DSM at coherent
Image (SSI)
OS extensions cache level

Efficient To reduce message-passing system Fast message passing, active messages,

Communications overhead and hide latencies enhanced MPI library

Using a global job management

Cluster-wide Job Application of single-job management
system with better scheduling and
Management systems such as LSF, Codine
monitoring

Balancing the workload of all

Dynamic Load Workload monitoring, process migration,
processing nodes along with failure
Balancing job replication and gang scheduling
recovery

Use of scalable interconnect,

Adding more servers to a cluster or
Scalability and performance monitoring, distributed
adding more clusters to a grid as the
Programmability execution environment, and better
workload or data set increases
software tools

• Lack of a cluster-wide OS limits full resource sharing.

• Middleware solutions provide necessary functionalities like scalability, fault tolerance, and
job management.
• Key challenges include efficient message passing, seamless fault tolerance, high
availability, and performance scalability.
Write a short notes on Peer to Peer network families

Definition

P2P systems eliminate central coordination, allowing client machines to act as both servers and
clients.

1. P2P Systems

• Decentralized architecture with self-organizing peers.

• No central authority — all nodes are independent.

• Dynamic membership — peers can join and leave freely.

2. Overlay Networks

Logical connections between peers, independent of the physical network. Two types:

• Unstructured overlays — Randomly connected peers, requiring flooding for data retrieval
(high traffic).

• Structured overlays — Use predefined rules for routing and data lookup, improving
efficiency.

3. P2P Application Families

P2P networks serve four main application categories:

Category Examples Challenges

File Sharing Napster, BitTorrent, Gnutella Copyright issues, security concerns

Collaboration Platforms Skype, MSN, Multiplayer games Privacy risks, spam, lack of trust

Distributed Computing SETI@Home, Genome@Home Security vulnerabilities, selfish nodes

Open P2P Platforms JXTA, .NET, FightingAID@Home Lack of standardization and security

4. P2P Computing Challenges

• Heterogeneity — Varying hardware, OS, and network configurations.

• Scalability — Must handle growing workloads and distributed resources efficiently.

• Data Location & Routing — Optimizing data placement for better performance.
• Fault Tolerance & Load Balancing — Peers can fail unpredictably.
• Security & Privacy — No central control means increased risk of data breaches and malware.
Discuss system attacks and threats to cyberspace resulting in 4 types of losses

4 Types of Losses (Threats to Systems and Networks)

1. Loss of Confidentiality

• Caused by eavesdropping, traffic analysis, and media scavenging.

2. Loss of Integrity

• Caused by penetration attacks, Trojan horses, and unauthorized access.

3. Loss of Availability

• Denial of Service (DoS) and resource exhaustion disrupt system operation.

4. Improper Authentication
• Allows attackers to steal resources, modify data, and conduct replay attacks.

Security Responsibilities

Security in cloud computing is divided among different stakeholders based on the cloud service
model:

Service Model Responsibility

SaaS Cloud provider handles security, availability, and integrity

PaaS Provider manages integrity and availability; users control confidentiality

IaaS Users responsible for most security aspects; providers ensure availability

Copyright Protection

• Collusive piracy in P2P networks allows unauthorized file sharing.

• Content poisoning and timestamped tokens help detect piracy and protect digital rights.

System Defense Technologies (3 Generations)

1. Prevention-based — Access control, cryptography.

2. Detection-based — Firewalls, intrusion detection systems (IDS), Public Key Infrastructure

(PKI).

3. Intelligent response systems — AI-driven threat detection and response.

Data Protection Infrastructure

• Trust negotiation ensures secure data sharing.

• Worm containment & intrusion detection protect against cyberattacks.

• Cloud security responsibilities vary based on the service model (SaaS, PaaS, IaaS).
5. Explain critical cluster design issues and feasible implementation

Critical Cluster Design Issues and Feasible Implementations

Major Cluster Design Issues:

1. Availability and Support

• Functional Characterization: Hardware and software support for sustained High Availability
(HA) in cluster.

• Feasible Implementations: Failover, failback, checkpointing, rollback recovery, nonstop OS,

etc.

2. Hardware Fault Tolerance

• Functional Characterization: Automated failure management to eliminate all single points of

failure.

• Feasible Implementations: Component redundancy, hot swapping, RAID, multiple power

supplies, etc.

3. Single System Image (SSI)

• Functional Characterization: Achieving SSI at functional level with hardware and software
support, middleware, or OS extensions.

• Feasible Implementations: Hardware mechanisms or middleware support to achieve

Distributed Shared Memory (DSM) at coherent cache level.

4. Efficient Communications

• Functional Characterization: To reduce message-passing system overhead and hide

latencies.

• Feasible Implementations: Fast message passing, active messages, enhanced MPI library, etc.

5. Cluster-wide Job Management

• Functional Characterization: Using a global job management system with better scheduling
and monitoring.
• Feasible Implementations: Application of single-job management systems such as LSF,
Codine, etc.

6. Dynamic Load Balancing

 • Functional Characterization: Balancing the workload of all processing nodes along with
failure recovery.

• Feasible Implementations: Workload monitoring, process migration, job replication and gang
scheduling, etc.

7. Scalability and Programmability

• Functional Characterization: Adding more servers to a cluster or adding more clusters to a

grid as the workload or data set increases.

• Feasible Implementations: Use of scalable interconnect, performance monitoring, distributed

execution environment, and better software tools.

Key Observations from the Content

• Lack of a cluster-wide OS limits full resource sharing.

• Middleware solutions provide necessary functionalities like scalability, fault tolerance, and
job management.

• Key challenges include efficient message passing, seamless fault tolerance, high
availability, and performance scalability.

• Virtual clusters and cloud computing are evolving to enhance cluster flexibility and resource
management.
7. Describe VM primitive operations

Virtual Machine (VM) Primitive Operations

Background

• Traditional computing tightly couples OS and hardware, reducing flexibility.

• Virtual Machines (VMs) abstract hardware resources, allowing multiple OS instances on a

single system.

VM Primitive Operations

There are four key VM primitive operations as described:

1. Multiplexing
• Multiple VMs share physical resources.

• VMs can be multiplexed between hardware machines.

• As shown in Figure 1.13(a), the VMs are multiplexed between hardware machines.

2. Suspension and Storage

• A VM can be suspended and stored in stable storage.

• As shown in Figure 1.13(b), the VM is paused and saved for later use.

3. Resumption / Provisioning

• A suspended VM can be resumed or provisioned to a new hardware platform.

• As shown in Figure 1.13(c), the VM is restored and deployed on a new system.

4. Migration

• A VM can be migrated from one hardware platform to another.

• As shown in Figure 1.13(d), the VM is moved across different servers seamlessly.

Virtual Machine Architectures

Three types of VM architectures support these operations:

Architecture Description
Architecture Description

Native VM (Hypervisor- Direct hardware access via bare-metal hypervisors (e.g., VMware
based) ESXi, Xen)

Runs as an application on a host OS (e.g., VirtualBox, VMware

Host VM (Software-based)
Workstation)

Hybrid VM Uses a combination of user-mode and privileged-mode virtualization

Virtual Infrastructure Benefits

• Separates physical hardware from applications, enabling flexible resource management.

• Enhances server utilization from 5–15% to 60–80% (as claimed by VMware).

8. Explain virtual machine with architectures of compared with traditional physical machine

Virtual Machines with Architectures Compared with Traditional Physical Machine

Virtual Machines and Virtualization Middleware:

Traditional Physical Machine (Problems)

• Traditional computing tightly couples OS and hardware, reducing flexibility.

• This tight coupling means the operating system is bound directly to the physical hardware,
making it difficult to share or reallocate resources efficiently.

Virtual Machine (Solution)

• Virtual Machines (VMs) abstract hardware resources, allowing multiple OS instances on

a single system.

• VMs overcome the limitations of traditional physical machines by separating software from
hardware.

Virtual Machine Architectures

There are three types of VM architectures:

1. Native VM (Hypervisor-based)

• Uses bare-metal hypervisors that have direct hardware access.

• The hypervisor sits directly on top of the physical hardware with no host OS in between.

• Examples: VMware ESXi, Xen.

2. Host VM (Software-based)
• Runs as an application on a host OS.

• The VM software is installed on top of an existing operating system.

• Examples: VirtualBox, VMware Workstation.

3. Hybrid VM
• Uses a combination of user-mode and privileged-mode virtualization.

• Combines aspects of both native and host VM approaches for balanced performance.
VM Primitive Operations

The four key operations that VMs support (which traditional physical machines cannot):

Operation Description

Multiplexing Multiple VMs share physical resources simultaneously — Figure 1.13(a)

Suspension & Storage A VM can be suspended and stored in stable storage — Figure 1.13(b)

Resumption / A suspended VM can be resumed or provisioned to a new hardware platform

Provisioning — Figure 1.13(c)

A VM can be migrated from one hardware platform to another — Figure

Migration
1.13(d)

Virtual Infrastructure

• Separates physical hardware from applications, enabling flexible resource management.

• Enhances server utilization from 5–15% to 60–80% (as claimed by VMware).

Data Center Virtualization for Cloud Computing

The role of VMs extends into data centers:

• 43 million servers worldwide (2010), with utilities (power & cooling) exceeding hardware
costs after three years.

• 60% of data center costs go toward maintenance and management, emphasizing energy
efficiency over raw performance.

Low-Cost Design Philosophy Enabled by Virtualization

• Commodity x86 servers & Ethernet replace expensive mainframes & proprietary networking
hardware.

• Software handles fault tolerance, load balancing, and scalability, reducing infrastructure costs.

Convergence of Technologies Enabling Cloud Computing through Virtualization

1. Virtualization & Multi-core Processors — Enable scalable computing.

2. Utility & Grid Computing — Provide a foundation for cloud computing.

3. SOA, Web 2.0, and Mashups — Facilitate cloud-based service integration.

4. Autonomic Computing & Data Center Automation — Improve efficiency and fault
tolerance.
Key Takeaway

• Traditional physical machines suffer from tight OS-hardware coupling and low resource
utilization (5–15%).

• Virtual Machines resolve this by abstracting hardware, enabling multiplexing, suspension,

migration, and provisioning, and boosting server utilization to 60–80%.
9. Explain the following

i) Internet of thing

ii) Cyber physical systems

iii) Memory storage and wide area networking

i) Internet of Things (IoT)

Scalable Computing Over the Internet:

Definition

• The concept of IoT was introduced in 1999 at MIT.

• The traditional Internet connects machines to machines or web pages to web pages.

• The IoT refers to the networked interconnection of everyday objects, tools, devices, or
computers.

Key Characteristics

• One can view the IoT as a wireless network of sensors that interconnect all things in our
daily life.

• It allows objects to be sensed and controlled remotely across existing network infrastructure.

• The IoT is supported by Internet clouds to achieve ubiquitous computing with any object at
any place and time.

IoT Components

• IoT interconnects everyday objects including:

o Computers

o Sensors
o Humans

o RFID, GPS devices

Relationship with Ubiquitous Computing

• Ubiquitous computing refers to computing with pervasive devices at any place and time
using wired or wireless communication.
• The IoT is a networked connection of everyday objects supported by Internet clouds to achieve
ubiquitous computing with any object at any place and time.
ii) Cyber Physical Systems (CPS)

Based on Section 1.1 — The Internet of Things (IoT) and Cyber-Physical Systems (CPS):

Definition

• CPS merges computation, communication, and control (3C) to create intelligent systems
for virtual and physical world interactions.

Key Characteristics

• CPS combines the three Cs:

Component Role

Computation Processing and intelligence

Communication Data exchange between systems

Control Managing physical processes

Role in Future Computing

• Both IoT and CPS will play a significant role in future cloud computing and smart
infrastructure development.

iii) Memory, Storage, and Wide Area Networking

Based on Section 1.2.3 — Memory, Storage, and Wide-Area Networking:

Memory Technology

• DRAM capacity has increased 4x every three years (from 16 KB in 1976 to 64 GB in 2011).

• Memory access speed has not kept pace, causing the memory wall problem, where CPUs
outpace memory access speeds.

Disks and Storage Technology

• Hard drive capacity has grown 10x every eight years, reaching 3 TB (Seagate Barracuda
XT, 2011).
• Solid-State Drives (SSDs) provide:
 o Significant speed improvements

o Better durability (300,000 to 1 million write cycles per block)

• Power and cooling challenges limit large-scale storage expansion.

System-Area Interconnects & Wide-Area Networking

Network Type Purpose

Local Area Networks (LANs) Connect clients and servers

Storage Area Networks (SANs) Support large-scale data storage and retrieval

Network Attached Storage (NAS) Support large-scale data storage and retrieval

Ethernet Speed Evolution

Year Speed

1979 10 Mbps

2011 100 Gbps

Future 1 Tbps links expected

Key Takeaway

• High-speed networking enhances distributed computing efficiency and scalability.

• The memory wall problem remains a key challenge as CPU speeds outpace memory access
speeds.

• SSDs are replacing traditional hard drives due to better speed and durability.
• The integration of memory, storage, networking, virtualization, and cloud data centers is
transforming distributed systems, achieving higher efficiency, flexibility, and cost-
effectiveness, paving the way for future exascale computing.
10. Explain computing paradigm distinction

Computing Paradigm Distinctions

Scalable Computing Over the Internet:

1. Centralized Computing

• A computing paradigm where all computer resources are centralized in a single physical
system.

• In this setup, processors, memory, and storage are fully shared and tightly integrated
within one operating system.

• Many data centers and supercomputers operate as centralized systems, but they are also
utilized in parallel, distributed, and cloud computing applications.

2. Parallel Computing

• In parallel computing, processors are either:

o Tightly coupled with shared memory, or

o Loosely coupled with distributed memory.

• Communication occurs through shared memory or message passing.

• A system that performs parallel computing is called a parallel computer.

• The programs running on it are called parallel programs.

• Writing these programs is referred to as parallel programming.

3. Distributed Computing

• Distributed computing studies distributed systems, which consist of multiple autonomous

computers with private memory communicating through a network via message passing.

• Programs running in such systems are called distributed programs.

• Writing them is known as distributed programming.

4. Cloud Computing

• Cloud computing refers to a system of Internet-based resources that can be either

centralized or distributed.
• It uses parallel, distributed computing, or both.
• Can be established with physical or virtualized resources over large data centers.

• Some regard cloud computing as a form of utility computing or service computing.

• Alternatively, terms such as concurrent computing or concurrent programming are used
within the high-tech community, typically referring to the combination of parallel and
distributed computing.

5. Ubiquitous Computing

• Refers to computing with pervasive devices at any place and time using wired or wireless
communication.

• The Internet of Things (IoT) is a networked connection of everyday objects including

computers, sensors, humans, etc.

• The IoT is supported by Internet clouds to achieve ubiquitous computing with any object at
any place and time.

6. Internet Computing

• Internet computing is even broader and covers all computing paradigms over the
Internet.

• This covers all the aforementioned computing paradigms, placing more emphasis on
distributed and cloud computing and their working systems, including the clusters, grids,
P2P, and cloud systems.

Key Distinctions Summary

Paradigm Key Feature

Centralized Computing All resources in a single physical system

Parallel Computing Multiple processors, shared or distributed memory

Distributed Computing Multiple autonomous computers via network message passing

Cloud Computing Internet-based, virtualized, centralized or distributed

Ubiquitous Computing Pervasive devices at any place and time

Internet Computing Covers all computing paradigms over the Internet

Computing Model Overlaps

• Distributed computing contrasts with centralized computing.

• Parallel computing shares concepts with distributed computing.

• Cloud computing integrates aspects of distributed, centralized, and parallel computing.
 11. Describe the classification of parallel and distributed computing systems

Classification of Parallel and Distributed Computing Systems

Parallel and distributed computing systems are classified based on the way instructions, data, and
computing resources are organized and processed.

A) Parallel Computing Classification (Flynn’s Taxonomy)

1. SISD (Single Instruction Single Data)

• Traditional uniprocessor computer systems.

• A single instruction operates on a single data stream at a time.

• Only one operation is executed at a given moment.

2. SIMD (Single Instruction Multiple Data)

• A single instruction is executed simultaneously on multiple data streams.

• Suitable for matrix operations and vector processing.

• Used in GPUs and multimedia applications.

3. MISD (Multiple Instruction Single Data)

• Multiple instructions operate on the same data stream.

• Rarely used in practice.

• Mainly used in fault-tolerant and aerospace systems.

4. MIMD (Multiple Instruction Multiple Data)

• Different processors execute different instructions on different data simultaneously.

• Most commonly used parallel architecture.

• It is divided into:

o Shared Memory Systems: All processors share a common memory.

o Distributed Memory Systems: Each processor has its own private memory and
communicates through message passing.

B) Distributed Computing Classification

1. Client–Server Systems

• Tasks are divided between clients and servers.

• Servers manage resources and services.

• Clients request and use these services.

2. Peer-to-Peer (P2P) Networks

• Every node acts as both a client and a server.

• Resources are distributed without central control.

• Used in file sharing and content distribution.

3. Cluster Computing

• A group of tightly connected computers work together as a single system.

• Provides high performance and resource sharing.

4. Grid Computing
• Geographically distributed and heterogeneous systems work together.

• Enables sharing of resources for large-scale applications.

5. Cloud Computing

• Provides computing resources as services through the Internet.

• Uses virtualization for on-demand resource allocation.

• Examples include AWS, Microsoft Azure, and Google Cloud.

Thus, parallel computing focuses on simultaneous execution of tasks, while distributed computing
focuses on resource sharing and coordination among multiple autonomous systems.
 Describe Scalable computing over the internet with appropriate examples
Scalable Computing Over the Internet
Definition
Instead of using a centralized computer to solve computational problems, a parallel and
distributed computing system uses multiple computers to solve large-scale problems over
the Internet. Thus, distributed computing becomes data-intensive and network-centric.
Scalable computing is driven by technological advancements that enable High-Performance
Computing (HPC) and High-Throughput Computing (HTC). Several trends have shaped
modern computing:
• Moore's Law — Doubling of processor speed every 18 months.
• Gilder's Law — Doubling of network bandwidth each year.
• The increasing affordability of commodity hardware has also fueled the growth of large-scale
distributed systems.

The Age of Internet Computing

The evolution of computing platforms progressed through several stages:
Era Development

1950 – Mainframes including the IBM 360 and CDC 6400

1970
1960 – Lower-cost minicomputers such as the DEC PDP 11 and VAX Series
1980
1970 – Widespread use of personal computers built with VLSI microprocessors
1990
1980 – Massive numbers of portable computers and pervasive devices in wired and
2000 wireless applications

Since Use of both HPC and HTC systems hidden in clusters, grids, or Internet
1990 clouds has proliferated

HPC and HTC Evolution

HPC Evolution
• Traditional supercomputers (MPPs) are being replaced by clusters of cooperative computers
for better resource sharing.
• HPC has focused on raw speed performance, progressing from Gflops (1990s) to Pflops
(2010s).
HTC and P2P Networks
• HTC systems prioritize high-flux computing, emphasizing task throughput over raw speed.
• P2P networks facilitate distributed file sharing and content delivery using globally distributed
client machines.
• HTC applications dominate areas like Internet searches and web services for millions of
users.

Market Shift from HPC to HTC

HTC systems address challenges beyond speed, including:
• Cost
• Energy efficiency
• Security
• Reliability
The transition from HPC to HTC marks a strategic shift in computing paradigms, focusing on
scalability, efficiency, and real-world usability over pure processing power.

Emerging Paradigms
• Advances in virtualization have led to the rise of Internet clouds, enabling service-oriented
computing.
• Technologies like RFID, GPS, and sensors are fueling the growth of the Internet of Things
(IoT).

Computing Model Overlaps

• Distributed computing contrasts with centralized computing.
• Parallel computing shares concepts with distributed computing.
• Cloud computing integrates aspects of distributed, centralized, and parallel computing.

Degrees of Parallelism
Parallelism in computing has evolved from:
1. Bit-Level Parallelism (BLP) — Transition from serial to word-level processing.
2. Instruction-Level Parallelism (ILP) — Executing multiple instructions simultaneously.
3. Data-Level Parallelism (DLP) — SIMD architectures.
4. Task-Level Parallelism (TLP) — Parallel execution of independent tasks on multicore
processors.
5. Job-Level Parallelism (JLP) — Large-scale distributed job execution in cloud computing.
Coarse-grained parallelism builds on fine-grained parallelism, ensuring scalability in HPC
and HTC systems.

Example
HTC applications dominate areas like Internet searches and web services for millions of
users.
This is a direct example of scalable computing over the Internet — where instead of a single
centralized machine, multiple distributed computers work together to handle millions of
simultaneous user requests, demonstrating the core principle of scalability, task throughput,
and network-centric distributed computing.
 Discuss the key enabling technologies used in network based systems

Key Enabling Technologies Used in Network-Based Systems

Advancements in multicore CPUs and multithreading technologies have played a crucial role in the
development of High-Performance Computing (HPC) and High-Throughput Computing (HTC).
The key enabling technologies for network-based systems are discussed below.

1. Advances in CPU Processors

• Modern multicore processors integrate dual, quad, six, or more processing cores to enhance
parallelism at the instruction level (ILP) and task level (TLP).

• Processor speed growth has followed Moore's Law, increasing from:

o 1 MIPS (VAX 780, 1978)

o 22,000 MIPS (Sun Niagara 2, 2008)

o 159,000 MIPS (Intel Core i7 990x, 2011)

• Clock rates have increased from 10 MHz (Intel 286) to 4 GHz (Pentium 4) but have stabilized
due to heat and power limitations.

2. Multicore CPU and Many-Core GPU Architectures

Multicore CPUs

• Multicore processors house multiple processing units, each with private L1 cache and
shared L2/L3 cache for efficient data access.

• Multicore CPUs continue to evolve from tens to hundreds of cores, but they face challenges
like the memory wall problem, limiting data-level parallelism (DLP).

Many-Core GPUs

• Many-core GPUs (e.g., NVIDIA and AMD architectures) leverage hundreds to thousands of
cores, excelling in data-level parallelism (DLP) and graphics processing.

• GPUs were initially designed for graphics acceleration but are now used for general-purpose
parallel computing (GPGPU).

• Modern GPUs (e.g., NVIDIA CUDA, Tesla, and Fermi) feature hundreds of cores, handling
thousands of concurrent threads.

Example: The NVIDIA Fermi GPU has 512 CUDA cores and delivers 82.4 teraflops, contributing
to the performance of top supercomputers like Tianhe-1A.

Example: Sun Niagara II — Built with eight cores, each supporting eight threads, achieving a
maximum parallelism of 64 threads.
Hybrid Architectures

• Hybrid architectures are emerging, combining fat CPU cores and thin GPU cores on a
single chip for optimal performance.
3. Multithreading Technologies in Modern CPUs

Different microarchitectures exploit parallelism at instruction-level (ILP) and thread-level (TLP):

Multithreading Type Description

Superscalar Processors Execute multiple instructions per cycle

Fine-Grained Multithreading Switches between threads every cycle

Coarse-Grained Multithreading Runs one thread for multiple cycles before switching

Simultaneous Multithreading (SMT) Executes multiple threads in the same cycle

4. GPU vs. CPU Performance and Power Efficiency

• GPUs prioritize throughput, while CPUs optimize latency using cache hierarchies.

• Power efficiency is a key advantage of GPUs — GPUs consume 1/10th of the power per
instruction compared to CPUs.

• Future Exascale Systems will require 60 Gflops/W per core, making power efficiency a
major challenge in parallel and distributed computing.

5. Memory Technology

• DRAM capacity has increased 4x every three years (from 16 KB in 1976 to 64 GB in 2011).

• Memory access speed has not kept pace, causing the memory wall problem, where CPUs
outpace memory access speeds.

6. Disks and Storage Technology

• Hard drive capacity has grown 10x every eight years, reaching 3 TB (Seagate Barracuda
XT, 2011).

• Solid-State Drives (SSDs) provide significant speed improvements and durability (300,000 to
1 million write cycles per block).

• Power and cooling challenges limit large-scale storage expansion.

7. System-Area Interconnects and Wide-Area Networking

• Local Area Networks (LANs) connect clients and servers.

• Storage Area Networks (SANs) and Network Attached Storage (NAS) support large-scale
data storage and retrieval.

• Ethernet speeds have evolved from 10 Mbps (1979) to 100 Gbps (2011), with 1 Tbps links
expected in the future.
 • High-speed networking enhances distributed computing efficiency and scalability.

8. Virtual Machines and Virtualization Middleware

Virtualization in Distributed Systems

• Traditional computing tightly couples OS and hardware, reducing flexibility.

• Virtual Machines (VMs) abstract hardware resources, allowing multiple OS instances on a

single system.
Virtual Machine Architectures

Type Description Example

Native VM (Hypervisor- Direct hardware access via bare-metal

VMware ESXi, Xen
based) hypervisors

Host VM (Software- VirtualBox, VMware

Runs as an application on a host OS
based) Workstation

Uses a combination of user-mode and

Hybrid VM —
privileged-mode virtualization

Virtual Machine Operations

• Multiplexing — Multiple VMs share physical resources.

• Suspension & Migration — VMs can be paused, saved, or migrated across different servers.
• Provisioning — VMs can be dynamically deployed based on workload demand.

Virtual Infrastructure

• Separates physical hardware from applications, enabling flexible resource management.

• Enhances server utilization from 5–15% to 60–80% (as claimed by VMware).

9. Data Center Virtualization for Cloud Computing

Data Center Growth and Cost Breakdown

• 43 million servers worldwide (2010), with utilities (power and cooling) exceeding hardware
costs after three years.
• 60% of data center costs go toward maintenance and management, emphasizing energy
efficiency over raw performance.

Low-Cost Design Philosophy

• Commodity x86 servers and Ethernet replace expensive mainframes and proprietary
networking hardware.
• Software handles fault tolerance, load balancing, and scalability, reducing infrastructure
costs.
Convergence of Technologies Enabling Cloud Computing

1. Virtualization and Multi-core Processors — Enable scalable computing.

2. Utility and Grid Computing — Provide a foundation for cloud computing.

3. SOA, Web 2.0, and Mashups — Facilitate cloud-based service integration.

4. Autonomic Computing and Data Center Automation — Improve efficiency and fault
tolerance.

10. Challenges in Future Parallel and Distributed Systems

1. Energy and Power Efficiency — Reducing power consumption while increasing

performance.

2. Memory and Storage Bottlenecks — Optimizing data movement to avoid bandwidth

limitations.
3. Concurrency and Locality — Improving software and compiler support for parallel
execution.
4. System Resiliency — Ensuring fault tolerance in large-scale computing environments.
4 Compare cluster computing and peer to peer system in the context of cloud architecture

Comparison of Cluster Computing and Peer-to-Peer (P2P) Systems in the Context of Cloud
Architecture

1. Definition

Cluster Computing

A computing cluster consists of interconnected stand-alone computers which work cooperatively

as a single integrated computing resource. In the past, clustered computer systems have
demonstrated impressive results in handling heavy workloads with large data sets.
Peer-to-Peer (P2P) Systems

P2P systems eliminate central coordination, allowing client machines to act as both servers and
clients. They use a decentralized architecture with self-organizing peers, where there is no central
authority and all nodes are independent. Dynamic membership allows peers to join and leave freely.

2. Comparison Table (From Provided Content)

Functionality /
Computer Clusters Peer-to-Peer Networks
Features

Network of compute nodes Flexible network of client machines

Architecture &
interconnected by SAN, LAN, or WAN logically connected by an overlay
Network
hierarchically network

Control & Resource Homogeneous nodes with distributed Autonomous client nodes, free in and
Management control, running UNIX or Linux out, with self-organization

Most appealing to business file

Applications & High-performance computing, search
sharing, content delivery, and social
Services engines, and web services
networking

Representative Google search engine, Sun Blade, IBM Gnutella, eMule, BitTorrent, Napster,
Systems Road Runner, Cray XT4 KaZaA, Skype, JXTA

3. Architecture and Network Connectivity

Cluster Computing

• Server clusters consist of multiple interconnected computers using high-bandwidth, low-

latency networks like Storage Area Networks (SANs), Local Area Networks (LANs), and
InfiniBand.
• These clusters are scalable, allowing thousands of nodes to be connected hierarchically.
 • Clusters are connected to the Internet via a VPN gateway, which assigns an IP address to
locate the cluster.

• Each node operates independently, with its own OS, creating Multiple System Images (MSI).

Peer-to-Peer (P2P) Systems

• P2P systems use logical connections between peers, independent of the physical network,
known as Overlay Networks.

• Two types of overlay networks:

o Unstructured overlays — Randomly connected peers, requiring flooding for data

retrieval (high traffic).

o Structured overlays — Use predefined rules for routing and data lookup, improving
efficiency.

4. Control and Resource Management

Cluster Computing
• Clusters use homogeneous nodes with distributed control.

• An ideal cluster merges multiple system images into a Single-System Image (SSI), where all
nodes appear as a single powerful machine.

• SSI is achieved through middleware or specialized OS support, enabling CPU, memory, and
I/O sharing across all cluster nodes.
• Clusters without SSI function as a collection of independent computers rather than a unified
system.
Peer-to-Peer (P2P) Systems
• P2P systems have no central authority; all nodes are independent.

• They use self-organization — peers manage themselves without centralized control.

• Dynamic membership — peers can join and leave freely without affecting the overall system.

5. Single-System Image (SSI) vs. Decentralization

Cluster Computing — SSI

• SSI is achieved through middleware or specialized OS support.

• Enables CPU, memory, and I/O sharing across all cluster nodes.

• Clusters without SSI function as a collection of independent computers rather than a unified
system.
• Key cluster hardware, software, and middleware support includes:

o Cluster nodes consist of PCs, workstations, or servers, interconnected using Gigabit

Ethernet, Myrinet, or InfiniBand.
 o Linux OS is commonly used for cluster management.

o Message-passing interfaces (MPI, PVM) enable parallel execution across nodes.

o Middleware supports features like high availability (HA), distributed memory

sharing (DSM), and job scheduling.

Peer-to-Peer (P2P) Systems — Decentralization

• No single-system image concept; each peer is fully independent.

• Peers communicate directly with each other through overlay networks.

• No middleware or centralized OS management required.

6. Major Design Issues

Cluster Computing Design Issues

Feature Functional Characterization Feasible Implementations

Availability and Hardware and software support for Failover, failback, checkpointing,
Support sustained HA in cluster rollback recovery, nonstop OS

Hardware Fault Automated failure management to Component redundancy, hot swapping,

Tolerance eliminate all single points of failure RAID, multiple power supplies

Hardware mechanisms or middleware

Single System Image Achieving SSI at functional level with
support to achieve DSM at coherent
(SSI) hardware and software support
cache level

Efficient To reduce message-passing system Fast message passing, active messages,

Communications overhead and hide latencies enhanced MPI library

Using a global job management

Cluster-wide Job Application of single-job management
system with better scheduling and
Management systems such as LSF, Codine
monitoring

Balancing the workload of all Workload monitoring, process

Dynamic Load
processing nodes along with failure migration, job replication and gang
Balancing
recovery scheduling

Adding more servers to a cluster or Use of scalable interconnect,

Scalability and
adding more clusters to a grid as performance monitoring, distributed
Programmability
workload increases execution environment

P2P Computing Challenges

• Heterogeneity — Varying hardware, OS, and network configurations.
• Scalability — Must handle growing workloads and distributed resources efficiently.

• Data Location and Routing — Optimizing data placement for better performance.

• Fault Tolerance and Load Balancing — Peers can fail unpredictably.

 • Security and Privacy — No central control means increased risk of data breaches and
malware.

7. Application Families

Cluster Computing Applications

• High-performance computing

• Search engines
• Web services

P2P Application Families

Category Examples Challenges

File Sharing Napster, BitTorrent, Gnutella Copyright issues, security concerns

Collaboration Platforms Skype, MSN, Multiplayer games Privacy risks, spam, lack of trust

Distributed Computing SETI@Home, Genome@Home Security vulnerabilities, selfish behavior

Open P2P Platforms JXTA, .NET, FightingAID@Home Lack of standardization and security

8. Scalability and Availability

Cluster Computing

• Clusters scale by adding more servers or adding more clusters to a grid as the workload or
data set increases.

• Middleware solutions provide necessary functionalities like scalability, fault tolerance, and
job management.

• Virtual clusters can be dynamically created using virtualization, optimizing resource

allocation on demand.

Peer-to-Peer Systems

• P2P networks are highly scalable but have low availability due to frequent peer failures.
• P2P networks scale to millions of nodes with independent OS images.

9. Security

Cluster Computing

• Clusters use authenticated security with centralized control.

• Security is managed through middleware tools including high availability and fault tolerance
mechanisms.

Peer-to-Peer Systems
 • No central control means increased risk of data breaches and malware.

• Collusive piracy in P2P networks allows unauthorized file sharing.

• Content poisoning and timestamped tokens help detect piracy and protect digital rights.

• P2P networks are suitable only for low-security applications like file sharing and
collaborative tools.

10. Grid Standards Comparison (Computational Grids vs. P2P Grids)

Design Issues Computational and Data Grids P2P Grids

Distributed supercomputing, National Open grid with P2P flexibility, all

Grid Applications
Grid initiatives resources from client machines

Representative TeraGrid (US), ChinaGrid, e-Science JXTA, FightAid@home,

Systems Grid (UK) SETI@home

Development Lessons Restricted user groups, middleware Unreliable user-contributed

Learned bugs, protocols to acquire resources resources, limited to a few apps

.
8 Describe the software environment used for distributed systems and cloud platforms

Software Environments for Distributed Systems and Cloud Platforms

Overview
This section introduces Service-Oriented Architecture (SOA) and other key software
environments that enable distributed and cloud computing systems. These environments define
how applications, services, and data interact within grids, clouds, and P2P networks.

1. Service-Oriented Architecture (SOA)

SOA enables modular, scalable, and reusable software components that communicate over
a network. It underpins web services, grids, and cloud computing environments.

1.1 Layered Architecture for Web Services and Grids

• Distributed computing builds on the OSI model, adding layers for service interfaces,
workflows, and management.
• Communication standards include:
Standard Description

SOAP (Simple Object Access Protocol) Used in web services

RMI (Remote Method Invocation) Java-based communication

IIOP (Internet Inter-ORB Protocol) Used in CORBA-based systems

• Middleware tools (e.g., WebSphere MQ, Java Message Service) manage messaging, security,
and fault tolerance.

1.2 Web Services and Tools

SOA is implemented via two main approaches:
1. Web Services (SOAP-based) — Fully specified service definitions, enabling distributed OS-
like environments.
2. REST (Representational State Transfer) — Simpler, lightweight alternative for web
applications and APIs.
• Web Services provide structured, standardized communication but face challenges in protocol
agreement and efficiency.
• REST is flexible and scalable, better suited for fast-evolving environments.
• Integration of Services — Distributed systems use Remote Method Invocation (RMI) or
RPCs to link services into larger applications.

1.3 The Evolution of SOA

SOA has expanded from basic web services to complex multi-layered ecosystems:
• Sensor Services (SS) — Devices like ZigBee, Bluetooth, GPS, and WiFi collect raw data.
• Filter Services (FS) — Process data before feeding into computing, storage, or discovery
clouds.
• Cloud Ecosystem — Integrates compute clouds, storage clouds, and discovery clouds for
managing large-scale applications.
SOA enables data transformation from:
 Raw Data → Useful Information → Knowledge → Wisdom → Intelligent Decisions

1.4 Grids vs. Clouds

Feature Grids Clouds

Resources Static resources Elastic, on-demand resources via virtualization

Negotiated resource Automation and scalability

Focus
allocation
Structured Hybrid models — clouds of grids, grids of
Models clouds, and inter-cloud architectures
computing

2. Trends Toward Distributed Operating Systems

Traditional distributed systems run independent OS instances on each node, resulting in
multiple system images. A distributed OS manages all resources coherently and efficiently
across nodes.

2.1 Distributed OS Approaches (Tanenbaum's Models)

1. Network OS — Basic resource sharing via file systems (low transparency).
2. Middleware-based OS — Limited resource sharing through middleware extensions (e.g.,
MOSIX for Linux clusters).
3. Truly Distributed OS — Provides Single-System Image (SSI) with full transparency across
resources.

2.2 Amoeba vs. DCE

System Approach Description

Microkernel Offers a lightweight distributed OS model

Amoeba
approach
Middleware Extends UNIX for RPC-based distributed
DCE computing
approach
Enables process migration across Linux-based
MOSIX2 Linux-based clusters and clouds

2.3 MOSIX2 for Linux Clusters

• Supports virtualization for seamless process migration across multiple clusters and clouds.
• Enhances parallel computing by dynamically balancing workloads across Linux nodes.

2.4 Transparency in Programming Environments

• Cloud computing separates user data, applications, OS, and hardware for flexible
computing.
• Users can switch between OS platforms and cloud services without being locked into
specific applications.

3. Parallel and Distributed Programming Models

 Distributed computing requires efficient parallel execution models to process large-scale
workloads.
Model Description Key Features

MPI (Message Standard for writing parallel Explicit communication

Passing applications on distributed between processes via
Interface) systems message-passing

Web programming model for Map function generates key-

MapReduce scalable data processing on value pairs; Reduce function
large clusters merges values

Open-source framework for Distributed storage (HDFS)

processing vast datasets in and MapReduce-based
Hadoop
business and cloud computing
applications

3.1 Message-Passing Interface (MPI)

• Used for High-Performance Computing (HPC).
• Programs explicitly send and receive messages for inter-process communication.

3.2 MapReduce
• Highly scalable parallel model, used in big data processing and search engines.
• Splits workloads into:
o Map — Processing tasks.
o Reduce — Aggregation tasks.
• Google executes thousands of MapReduce jobs daily for large-scale data analysis.

3.3 Hadoop
• Open-source alternative to MapReduce, used for processing petabytes of data.
• Scalable, cost-effective, and fault-tolerant, making it ideal for cloud services.

4. Grid Standards and Toolkits

Grids use standardized middleware to manage resource sharing and security.
Standard Function Key Features

OGSA (Open Grid Supports heterogeneous

Defines common grid computing, security policies,
Services
services and resource allocation
Architecture)

Middleware for Uses PKI authentication,

Globus Toolkit (GT4) resource discovery and Kerberos, SSL, and delegation
security policies

Grid computing Supports autonomic computing

IBM Grid Toolbox framework for and security management
AIX/Linux clusters

5. The Rise of Data-Intensive Computing

• Scientific research, business, and web applications generate vast amounts of data.
• Cloud computing and parallel computing address the data deluge challenge.
• MapReduce and Iterative MapReduce enable scalable data processing for big data and
machine learning applications.
• The convergence of data-intensive computing, cloud platforms, and multicore
architectures is shaping the next generation of distributed computing.