Cloud hardware and
software
Module 2
Chapter Focus:
Module-2 Cloud hardware and software (8 hours)
○ Cloud infrastructure challenges, Cloud
hardware-warehouse-scale computer (WSC), WSC
performance, Hypervisors, Cluster management with
Borg, Containers; Docker containers, Kubernetes
○ Text Book 1: Chapter 4.1 to 4.4, 4.7, 4.13 to 4.14
Cloud Infrastructure
Cloud Service Providers (CSPs)
● Deliver reliable, scalable, and low-cost cloud services.
● Use modern computing, networking, and software technologies.
Cloud Hardware
● Built using millions of low-cost commodity servers.
● Provides massive computing power at lower cost.
● Easily scalable by adding more servers.
Cloud System Software
● Manages hardware resources.
● Schedules applications.
● Supports multiple workloads efficiently.
Evolution of Computing
Single Processor → Multiprocessor → Multicore → Cluster → Warehouse-Scale
Computers (WSCs)
A Warehouse-Scale Computer (WSC) contain hundreds of thousands of
servers that work together like one giant computer.
Major Challenges
● Controlled by complex software stacks.
a. To manage resource allocation, Job scheduling, Failure recovery, Monitoring
● Huge number of components → harder to ensure efficiency and reliability.
● Using cheap, off-the-shelf parts means higher risk of failure → harder to maintain
availability.
a. High availability = lower downtime
● Needs both hardware redundancy and strong software systems for fault detection
and recovery.
Virtualization
Virtualization hides the complexity of physical hardware.
Users see virtual resources, not actual servers.
Benefits:
● Better resource utilization
● Easier management
● Improved reliability
● Portability
Virtualization in Cloud Computing
Virtual Machines (VMs)
● Virtual computers running on physical servers.
● Each VM has its own operating system.
● Multiple VMs share the same hardware.
Why do Cloud Providers Use VMs?
For Users
● Better isolation between applications.
● Can choose their preferred operating system.
● Easy to move applications.
For Providers
● Higher server utilization.
● Lower operating costs.
● Easier resource management.
VM Migration
One major advantage of virtualization is VM migration.
If one server becomes overloaded,
the VM can be moved to another server.
Two methods:
Cold Migration Live Migration
● Stop VM ● Move VM while it is still running.
● Copy VM ● Users usually do not notice any interruption.
● Restart VM
Containers
● OS-level virtualization.
● Package applications with their dependencies.
● Share the host operating system.
● Lightweight, portable, and fast.
Benefits of Virtualization
● Lightweight and portable
● Faster startup
● Better resource utilization
● Easy deployment and scaling
● Predictable application performance
● Simplified Cloud Management
Challenge of Virtualization
Virtualization introduces another software layer called the hypervisor.
The hypervisor: Container Management
● Creates VMs. ● Docker – Creates and packages
containers.
● Shares CPU and memory.
● Kubernetes – Automates deployment,
● Controls hardware access. scaling, and management.
Although useful, virtualization: ● Apache Mesos – Manages resources
across distributed clusters.
● Increases software complexity.
● Adds some performance overhead.
● Introduces additional security concerns.
Warehouse Scale Computers
■ WSCs form the backbone of the cloud infrastructure.
It has has 50,000 - 100,000 processors.
■ The cost of a WSC is of the order of $150 million.
Data Center (WSC) → Cells → Racks→ Servers
Servers: Individual computers that execute applications.
Racks: contains about 48 servers connected by a high-speed switch.
Cells (Arrays): Multiple racks are connected together. Each cell may contain up to
30 racks.
Warehouse-Scale
Computer
↓
Cells
↓
Racks
↓
Servers
Communication speed depends on where the servers are located.
● Same Server Fastest communication.
● Same Rack Very fast.
● Different Rack Slower.
● Different Cell Slowest.
So, cloud schedulers try to place related applications close together to reduce
communication time.
■
■ Two basic groups of multicore processors:
◻ brawny - single core performance is impressive,
■ High performance, Fast execution, Fewer cores needed
■ High power consumption
◻ wimpy - less powerful
■ Lower performance, Energy efficient, Many cores needed
■ Higher synchronization overhead, Increased scheduling complexity
Brawny Core Wimpy Core
● High performance ● Lower performance
● Higher power consumption ● Energy efficient
● Fewer threads required ● More threads required
● Lower scheduling overhead ● Higher scheduling overhead
■ When running on wimpy cores a task needs to spawn a larger
number of threads. Major implications:
◻ It complicates the software development process as it
requires an explicit parallelization of the application thus,
increasing the cost of application development.
◻ Running a larger number of threads increases the response
time. Very often all threads have to finish before the next step of
an algorithm, the well known problem posed by
barrier-synchronization.
WSC (Warehouse-Scale Computer) Performance
● Millions of users use it simultaneously.
● Thousands of different applications run together.
● Hardware constantly fails.
● Data is spread across many servers.
There is no single application that represents all workloads. WSCs
workload is diverse, there are no ``killer'' applications that would drive the
design decisions. Therefore, no guaranteed optimal performance for such
workloads.
■ Solution - profile realistic workloads and analyze data collected
during production runs.
■ Google uses a tool called Google-Wide Profiling (GWP) to measure
performance Google collects performance information while the system is
running.
■ It works by
● selecting random servers
● monitoring them briefly
● collecting performance statistics
● storing results for analysis
This causes very little overhead
■ Only data for C++ codes was analyzed because C++ codes dominate the
CPU cycle consumption though, the majority of codes are written in Java,
Python, and Go.
■ Data was collected from some 20,000 servers built with Intel Ivy Bridge
processors.
WSC Architecture Tax
A large portion of CPU time is not spent doing useful application work.
Instead, it is spent moving data around. 22–27% of CPU cycles are wasted on common
system activities. These activities include:
● Remote Procedure Calls (RPC): When one server requests another server to
perform work.
● Serialization: Converting objects into bytes before sending them across the network.
● Deserialization : Converting received bytes back into objects.
● Compression: Reducing message size before transmission.
● Memory copy: copy large blocks of memory. These alone consume 4–5% CPU time
(memmove, memcpy)
● Memory allocation: Creating and freeing memory repeatedly.
● Operating System: Scheduling, Process management, Kernel functions consume
another large fraction of CPU time.
■ Steps of a typical communication pattern:
1. serialize the data to the protocol buffer;
2. execute an RPC and pass the buffer to the callee; and
3. caller deserializes the buffers received in response to the RPC.
Cache and memory management.
Memory is another bottleneck
Cloud applications are huge (hundreds of MB/GB)
Stall Cycles: A CPU sometimes has nothing to execute because it is waiting for
data. → performance reduces
Large programs cause
● cache misses (data not in cache so needs to be fetched from RAM): 50–60%
of CPU stall cycles
● memory latency : time taken for the data to arrive
● poor CPU utilization - CPU is often waiting for memory, communication,
synchronization.
Improving WSC Performance
● Optimize common communication routines
● Reduce memory latency and cache misses
● Improve Instruction-Level Parallelism (ILP)
● Use Simultaneous Multithreading (SMT) to hide memory delays
● Balance cache size and speed
● Small efficiency gains can save millions of dollars in large cloud data centers
Note:
SMT lets another thread use the CPU. Example:
Thread A waits for memory.
↓
Thread B executes immediately.
↓
CPU stays busy.
Microarchitecture of a Modern CPU Core Program
│
● Instruction Fetch (Front-End)
▼
○ The CPU fetches instructions from memory
Instruction Fetch
● Instruction Decode │
○ The CPU translates machine instructions into simpler internal ▼
operations called micro-op Instruction Decode
● μop Queue : Micro-operation Queue (μops). │
○ Decoded μops wait here before execution. ▼
● Instruction Scheduler (Back-End) Micro-operation Queue (μop Queue)
○ This is the brain of the CPU. It decides │
■ which instruction is ready ▼
■ which execution unit is free Instruction Scheduler
│
■ which instruction should run first
┌─────┼─────┐
○ Instead of executing instructions strictly in program order, the
▼ ▼ ▼
scheduler often uses out-of-order execution to improve ALU Load/Store FPU
performance. │ │ │
● Execution Units └──────┼──────┘
○ ALU (Arithmetic Logic Unit), Load/Store Unit, Floating Point ▼
Unit (FPU) Write Back
● Write Back │
○ After execution, the result is written back into CPU registers. ▼
Program Complete
Modern
processors core
microarchitecture.
1. Front-End (Instruction Preparation)
● L1 Instruction Cache → stores recently used instructions.
● Pre-decode & Instruction Queue → prepares and organizes instructions.
● Branch Predictor → guesses the next instruction to avoid waiting.
● Load/Store Buffers & Reorder Buffers → handle memory access and keep
track of execution order.
● Micro-op Cache → stores small decoded instructions for faster reuse.
● Allocate/Rename Unit → prevents conflicts by renaming registers.
Purpose: Fetch and prepare instructions for execution.
2. Scheduler (Instruction Dispatcher)
● Distributes instructions to different CPU units.
● Has 5 Ports for different tasks:
○ Port 0 & 1 → Arithmetic Logic Units (ALUs) for add, multiply, vector ops,
floating-point ops.
○ Port 5 → Jump instructions, floating-point operations.
○ Port 2 & 3 → Load operations (reading from memory).
○ Port 4 → Store operations (writing to memory).
● Purpose: Send instructions to correct execution units.
3. Back-End (Execution & Memory)
● ALUs (Arithmetic Logic Units) → perform math and logic.
● FP Units (Floating Point) → handle decimal and scientific calculations.
● Memory Control → manages data movement between:
○ L1 Data Cache (fast, small memory).
○ L2 Cache (slower, larger memory).
● Execution can be in-order or out-of-order (reordering for efficiency).
● Purpose: Actually run the instructions & access memory.
Hypervisors
● A hypervisor is software that divides a computer’s resources into one or more
virtual machines (VMs).
● A guest OS is the operating system running inside a VM (not directly on the
hardware).
● The Guest OS doesn't directly control the physical hardware. It asks the
hypervisor to provide access to CPU, Memory, Storage, Network
● The hypervisor sits between hardware and guest OS, making multiple OS
run on the same machine while keeping them isolated and secure.
Functions and Benefits of Hypervisors
1. Isolation & Security
● Keeps VMs isolated from each other.
● Problems in one VM normally do not affect other VMs.
● Prevents unauthorized access to physical resources.
2. Resource Management
● Controls how each VM uses CPU, memory and other resources.
● Monitors performance and prevents resource conflicts.
Functions and Benefits of Hypervisors
3. Resource Sharing
● Allows multiple services/applications to share the same physical server.
● Improves hardware utilization.
4. Live Migration
● Moves a running VM from one physical server to another.
● Live Migration is useful for:
○ Load balancing
○ Maintenance
○ Fault recovery
Hypervisor
Hypervisor runs with higher privilege than the Guest OS.
Guest OS requests access to hardware through the hypervisor.
Privileged instructions from a VM are intercepted (trapped) by the hypervisor.
Hypervisor checks the operation and allows or blocks it.
Hypervisor virtualizes:
● CPU → allocates processor time to VMs
● Memory → maps virtual memory to physical memory
● Interrupts → delivers hardware events to the correct VM
Hypervisor
Memory Virtualization:
● Each VM sees its own virtual memory.
● Hypervisor maintains mappings between:
Virtual Address → Physical Memory
● Hardware MMU (Memory Management Unit) performs address
translation.
Hypervisor: Memory Management & Performance
Memory management challenges
● Multiple VMs compete for limited physical memory.
● Hypervisor decides which memory pages should remain in RAM or be moved to disk.
● Excessive swapping can cause performance degradation (thrashing).
Optimization techniques
● Page sharing: Identical pages used by multiple VMs can share one physical copy.
● Copy-on-write: A shared page is copied only when a VM modifies it.
● Ballooning: Technique where hypervisor can reclaim unused RAM from virtual
machines (VMs) and reallocate it where it is needed
Resource Hypervisor's job
CPU Allocate CPU time
Memory Allocate and map RAM
Storage Provide virtual disks
Network Provide virtual network access
Security Isolate VMs
Performance Monitor and manage resource usage
Migration Move VMs between servers
Cluster Management with Borg
A computer cluster is a tight group of connected computers working together
A large cluster may contain tens of thousands of processors/machines.
Different machines may have different:
● CPU types and speeds
● Number of CPU cores
● RAM capacity
● Storage
● Network capabilities
Therefore, managing applications manually becomes very difficult.
Application Developer wants to: Cloud/Service Provider should provide
1. Easily find computing resources → High system availability
2. Deploy applications easily → High reliability
3. Control application resources → Efficient resource utilization
4. Avoid dealing with hardware details → Handle machine/application failures
Cluster Management with Borg
● Borg is Google's cluster management system. It manages thousands of
machines and applications in a data center.
● Instead of developers manually deciding which machine should run which
application, Borg automatically handles resource allocation, scheduling,
failures, and monitoring
● Borg hides the complexity of the infrastructure from developers.
Borg was designed to:
● Manage applications across large numbers of machines
● Provide high availability and reliability
● Automatically handle resource allocation
● Automatically handle failures
● Support:
○ Long-running applications
○ Interactive production jobs
○ Batch jobs
● Allow developers to focus on application development rather than
infrastructure management
BORG ARCHITECTURE
A Borg-managed cluster is
called a cell.
A cell can contain tens of
thousands of machines.
Communications manager
Borg Architecture :
Cells
● Each job runs in one Borg cell, a set of machines that are managed as
a unit.
● Each cell belongs to a cluster of heterogenous machines, generally a data
centre.
● A cluster contains a single cell plus other smaller cells reserved for testing or
other purposes.
Borg Architecture:
Cells
● Borg cells run a heterogeneous workload with two main parts:
○ A long-running process/service which is meant for end-user-facing
products.
○ Batch tasks are relatively short-lived and less sensitive to performance
fluctuations.
● Borg masks this heterogeneity of machines by determining where a task can
run in a particular cell, allocating it and its required resources.
● Borg monitors the health of the task and attempts restarts in case of failures.
BorgMaster
● Centralized controller of the cluster
● Maintains information, health and state about machines, tasks and resources
● Receives requests from users, coordinates the execution of jobs
● Assigns and manages tasks
● Communicates with Borglets
● A borglet is a machine-local agent process
● runs on every physical machine
● It acts as the worker-node counterpart to the centralized controller known as the
Borgmaster.
● Starts, stops and restarts tasks
● Manages local resources
● Reports machine/task status to BorgMaster
● The Borgmaster polls each borglet every few seconds.
● This polling mechanism retrieves status updates and hands off new task execution
instructions.
● If a borglet stops responding to these health checks, the Borgmaster marks the
underlying machine as down and reschedules its critical workloads elsewhere
Jobs and tasks
● Users submit their work to Borg in the form of jobs
● A Borg job can contain multiple tasks.
● A Borg job’s properties include its name, owner, and the number of tasks.
● It can also have constraints on its resources which determine its required
attributes, such as OS version and processor type.
● Tasks can also have attributes, such as resource requirements, hierarchy
within the job etc.
Borg Scheduler
Maintains a queue of pending tasks.
Examines tasks periodically in round-robin order.
Finds machines that can satisfy the task's requirements and selects the best
machine(s) for execution.
The scheduler uses two important stages:
A. Feasibility
● Finds machines where the task can run, by checking resources (CPU RAM etc)
B. Scoring
● Among the feasible machines, determines which machine is the best choice.
● Considers resource utilization and workload balancing.
● Borg caches the previous score and reuses it until the machine or task changes
significantly.
Not all jobs are equally important.
Borg assigns different priority levels, for example:
Monitoring , Production, Batch, Testing
High-priority jobs receive resources preferentially.
A quota specifies how much CPU, RAM, disk, etc. a job can use during a particular
period.
Higher-priority quotas generally have a higher cost.
The scheduler tries to allocate resources while:
● Balancing workload
● Avoiding resource waste
● Leaving some capacity for sudden demand spikes
Alloc and Alloc Set
Alloc: Reserves resources on one machine for a task.
Alloc Set: Reserves resources across multiple machines.
A Borg cell can contain tens of thousands of machines, so scheduling must
itself be efficient.
● Borg uses multiple scheduler processes working concurrently.
● They interact with BorgMaster and Borglets.
● They use cached copies of cluster state to reduce overhead.
● This allows Borg to make scheduling decisions without constantly querying
every machine.
The states of a Borg task:
Task state changes as a result of
either user requests or system actions
Users can send requests such as:
● Submit → start a new task
● Update → modify a running task
● Kill → terminate a task
System-generated state changes include:
● Reject → task cannot be started
● Evict → task is removed to free
resources
● Lost → task/machine becomes
unavailable
Container
A container is a lightweight, isolated environment used to run an application along
with everything it needs.
a container shares the host operating system's kernel while keeping applications
isolated.
Containers use Linux technologies such as:
● Namespaces → isolate processes, networks, filesystems, etc.
● cgroups (Control Groups) →
a. control and limit resource usage including CPU usage, Disk I/O, Network
Resources
b. They can also prioritize certain processes so that important applications
receive more resources.
Docker
Docker is a platform that makes it easy to create, package, distribute, and
run containers.
A Docker container packages:
Application + Code + Runtime + Libraries + Dependencies
Therefore, the application can run consistently across different environments.
● Portable – same container can run in different environments
● Lightweight – shares the host OS kernel
● Fast startup – starts much faster than a VM
● Lower resource usage
● Application isolation
● Easy scaling – containers can be quickly created or removed
Docker Ecosystem
Image
● A blueprint/template for an application. Base image
● Contains the application and its dependencies.
Two basic types:
Base Image
● Starting point for creating an image.
● Ex: Ubuntu, scratch, alpine, Python.
Child Image
● Built on top of another image.
● Adds additional application software or configuration.
Container
● A running instance of an image.
Docker Daemon
● Background service that manages Docker containers and images.
Docker Client
● Tool through which users communicate with the Docker daemon.
● Example: Docker CLI.
Docker Hub
● Registry/repository for storing and sharing Docker images.
Dockerfile
● A Dockerfile is a text file containing instructions for creating a Docker image.
Containers are widely used in cloud computing because they:
● Allow efficient application deployment
● Support rapid scaling
● Improve resource utilization
● Provide portability between environments
● Simplify application management
Amazon ECS
ECS = Amazon Elastic Container Service
● AWS service for creating and managing container clusters.
● Container hosts are usually Amazon EC2 instances.
● Uses an AWS-managed scheduler/cluster manager to place containers on
hosts.
● Container images must be available through an accessible container
registry such as Docker Hub or an AWS registry.
Integration with AWS services:
● IAM → controls permissions and access.
● CloudTrail → records and monitors resource/API activity.
● CloudFormation → manages ECS infrastructure.
Google Kubernetes Engine (GKE)
GKE = Google Kubernetes Engine
● Managed container service based on Kubernetes.
● Kubernetes is an open-source container orchestration platform.
● Automates:
○ Container deployment
○ Scaling
○ Starting/stopping containers
○ Scheduling containers across machines
● Google emphasizes performance and large-scale container
management.
● Integrates with services such as Google Cloud Logging.
● Supports private container registries and declarative configuration.
Microsoft Azure Container Service
● Azure provides container management through Azure Resource
Manager.
● Supports multiple orchestration technologies, including:
○ Docker
○ Apache Mesos
● Provides a way to manage container-based applications within Azure
infrastructure.
Open Container Initiative (OCI)
OCI was created to develop common industry standards for containers.
Its goals include:
● Standardize container image formats
● Standardize container runtime
● Improve compatibility between container platforms
● Make containers more portable across cloud providers
Kubernetes
Kubernetes (K8s) is an open-source container orchestration system
originally developed at Google.
If Docker is used to create and run containers, Kubernetes is used to
manage many containers across many machines.
For example, suppose an application needs 20 containers:
Kubernetes decides:
● Where containers should run
● How many containers are required
● When to start/stop containers
● What to do when a container fails
● How to distribute requests
● How to scale the application
Kubernetes is an "open-ended" system:
● Kubernetes does not force users to use one particular technology for
everything.
It provides the basic infrastructure for:
● Deployment, Scaling, Load balancing, Monitoring, Logging
But other components can be added or replaced.
● It also allows developers to create their own:
a. Controllers
b. Schedulers
c. Extensions
Kubernetes Architecture Control Plane / Master → control component
Worker Nodes / Minions → worker machines
Kubernetes: Control Plane / Master
The Master/Control Plane manages the cluster.
Important components
API Server
● Main communication interface
● Users and other components communicate with Kubernetes through it.
● Provides APIs for managing the cluster.
Scheduler
● Decides where a Pod should run.
● Looks at available CPU, memory and other resources.
etcd
● Distributed key-value database.
● Stores important Kubernetes cluster information and configuration.
Kubernetes: Worker Nodes / Minions
Worker nodes are the machines that actually run applications.
Kubelet
● Runs on each worker node.
● Communicates with the control plane.
● Ensures that the required containers are running.
● Reports the node's status.
Container Runtime
● Actually runs the containers.
● Container runtimes can be Docker; modern Kubernetes can use other container
runtimes as well.
Proxy / Networking component
● Helps provide communication and networking between services and containers.
Kubernetes: Pod
● This is one of the most important Kubernetes
(example)
concepts.
● A Pod is the smallest deployable unit in
Kubernetes.
● A Pod is configured through a YAML manifest and
typically includes the following shared layers:
● A Pod can contain: One or more related
application/helper containers
● The pods can share:
● Network, IP address, Port space, Storage
volumes
● Therefore, containers inside the same Pod can
communicate using: localhost
One of Kubernetes' major advantages is self-healing.
Assume there are n number of pods running. If one of them fails, Kubernetes detects
that the actual state does not match the desired state and creates a replacement
Labels: Kubernetes can use labels to identify and group Pods.
Services: A Kubernetes Service provides a stable way to access a group of Pods.