Module 2

Levels of Virtualization and their role in AI clusters,

Hardware-Level Virtualization (Machine-Level)

OS-Level Virtualization (Container-Level)

Application-Level Virtualization

Network Virtualization

Storage Virtualization
Hardware-Level Virtualization (Machine-Level)
Hardware-Level Virtualization (Machine-Level)

Virtualization implemented directly on physical hardware using a hypervisor

(Type-1 or Type-2).

Creates multiple Virtual Machines (VMs), each with its own OS.
Role in AI Clusters
● Isolation of AI workloads: Each AI experiment/model runs in a separate VM.

● GPU passthrough & vGPU: Enables controlled sharing of expensive GPUs across users.

● Security & stability: Faults in one VM do not affect others.

● Elastic scaling: AI nodes can be dynamically provisioned or decommissioned.

AI Use Case

● University or enterprise AI clusters where multiple teams train models simultaneously on shared
hardware.
Examples

● VMware ESXi running multiple AI virtual machines on a single GPU server

● KVM hypervisor hosting separate VMs for different AI research teams

● Microsoft Hyper-V creating isolated VMs for AI training and testing

● GPU Passthrough: One NVIDIA GPU assigned to a single VM for deep learning

AI Cluster Example

A college AI lab runs separate VMs for each student group, each VM training its own
ML model without interfering with others.
2. OS-Level Virtualization (Container-Level)
Virtualization at the operating system level using containers (e.g., Docker).

Containers share the host OS kernel but have isolated environments.

2. OS-Level Virtualization (Container-Level)
Role in AI Clusters

● Fast startup: Containers launch in seconds—ideal for AI pipelines.

● Environment consistency: Same CUDA, TensorFlow, PyTorch versions everywhere.

● High resource efficiency: No extra OS overhead → better GPU utilization.

● Orchestration with Kubernetes: Automates training jobs across nodes.

Examples

● Docker containers running TensorFlow or PyTorch environments

● Kubernetes cluster scheduling AI training jobs across nodes

● NVIDIA CUDA containers for GPU-accelerated deep learning

● Kubeflow pipelines for ML model training and deployment

AI Cluster Example

An AI cluster runs 100s of Docker containers, each container executing a different deep-learning
experiment on shared GPUs.
3. Application-Level Virtualization
Individual applications are virtualized and run independently of the
underlying OS.

Includes packaged ML apps, virtual environments, or runtime

sandboxes.
Role in AI Clusters
● Portable AI applications: Same model runs across dev, test, and production.

● Simplified deployment: No dependency conflicts.

● Model serving isolation: Each inference service runs independently.

AI Use Case
● Deploying trained AI models as APIs or microservices.
Examples

● A virtual Python environment (venv / Conda) for ML libraries

● Packaged AI inference service running independently of OS

● Model-as-a-Service (MaaS) deployed as a standalone application

● Jupyter Notebook servers running as isolated AI applications

AI Cluster Example

A trained AI model is deployed as a REST API, allowing applications to use predictions without knowing system
details.
4. Network Virtualization
 ● Virtual networks are created on top of physical networks using SDN and overlays.

Role in AI Clusters
● High-speed communication between distributed training nodes.

● Traffic isolation for different AI workloads.

● Optimized data flow for parameter servers and GPU collectives.

● Scalable cluster networking without rewiring hardware.

Examples

● Software Defined Networking (SDN) for AI data traffic control

● Virtual Private Cloud (VPC) connecting AI nodes securely

● Overlay networks in Kubernetes (Flannel, Calico)

● Virtual LANs (VLANs) separating training and inference traffic

AI Cluster Example

Distributed AI training nodes communicate over a virtual high-speed network,

optimizing gradient sharing between GPUs.
5. Storage Virtualization
● Abstracts multiple physical storage devices into a unified storage pool.

Role in AI Clusters

● High-throughput data access for massive datasets.

● Shared datasets across nodes without duplication.

● Fault tolerance and replication.

● Supports object, block, and file storage for AI workloads.

 Virtualization Level Key Contribution to AI Clusters

Hardware-Level Secure isolation, VM-based GPU usage

OS-Level (Containers) Fast, scalable AI workloads

Application-Level Portable AI apps & model serving

Network Virtualization High-speed distributed training

Storage Virtualization Shared, scalable data access

GPU Virtualization Efficient accelerator sharing

Examples

● Distributed file systems (Ceph, GlusterFS)

● Object storage (S3-compatible storage for datasets)

● Network Attached Storage (NAS) shared across AI nodes

● Logical Volume Management (LVM) pooling disks for AI data

AI Cluster Example

All AI nodes access a shared dataset repository without copying data locally, saving storage and time.
Virtual Machine Monitors (VMM) / Hypervisors
It is a software layer that enables virtualization by allowing multiple
Virtual Machines (VMs)—each with its own operating system—to run
simultaneously on a single physical machine.
The VMM manages and allocates CPU, memory, storage, network, and
GPU resources among VMs while ensuring isolation and security.
 Core Functions of a VMM
CPU Virtualization

● Schedules virtual CPUs (vCPUs) onto physical CPUs

● Supports context switching between VMs

Memory Virtualization

● Maps virtual memory of VMs to physical RAM

● Ensures memory isolation between VMs

I
/O Virtualization

● Manages disk, network, and device access

● Uses device emulation or direct passthrough

Isolation & Security

● Prevents one VM from accessing another VM’s resources

● Fault containment (VM crash ≠ host crash)

Resource Management

● Dynamic allocation of CPU, RAM, storage, and GPUs

● Enables load balancing and scalability

Types of Virtual Machine Monitors
Type-1 VMM (Bare-Metal Hypervisor)

Examples
● VMware ESXi
● Xen
● Microsoft Hyper-V
2. Type-2 VMM (Hosted Hypervisor)
Examples

● Oracle VirtualBox

● VMware Workstation
Full Virtualization and Para Virtualization
Full Virtualization is a virtualization technique that simulates an entire physical
computer, including its hardware components, to create multiple virtual machines (VMs)

n this approach, the guest operating system is unaware that it's running in a virtualized
environment, as it interacts with virtualized hardware that emulates real hardware.
How Full Virtualization Works?

1. Hypervisor Layer:
Full virtualization relies on a hypervisor, also known as a Virtual Machine Monitor
(VMM), which sits between the physical hardware and guest operating systems.
The hypervisor manages and controls the allocation of physical resources to
virtual machines.
2. Hardware Virtualization:
Full virtualization uses hardware-assisted virtualization technologies like Intel VT-x
or AMD-V to enhance performance. These technologies allow the hypervisor to
run guest OSes directly on the physical CPU without significant performance
overhead.
1. Isolation:
VMs created through full virtualization are
completely isolated from each other. Each VM
runs its own instance of the guest operating
system, which cannot interfere with other VMs.
2. Examples:
Popular hypervisors for full virtualization include
VMware vSphere/ESXi, Microsoft Hyper-V, and
Oracle VirtualBox.
Para Virtualization

It involves modifying the guest operating systems to be aware of the virtualized

environment.

paravirtualization requires guest OSes to use a specific set of APIs to interact with the
virtualization layer.
How Para Virtualization Works?

1. Hypervisor Layer:
Similar to full virtualization, paravirtualization also employs a hypervisor, but here,
the guest operating systems are aware of it. The hypervisor provides a set of APIs
that guest OSes must use to communicate with the underlying hardware.
2. Guest OS Modifications:
Guest operating systems must be modified to replace certain hardware-related
instructions with hypercalls, which are calls to the hypervisor. These hypercalls
allow the guest OS to request services from the hypervisor, such as memory
management or CPU scheduling.
1. Performance Benefits:
Since para virtualization avoids the
overhead of emulating complete
hardware, it often offers better
performance than full virtualization. Guest
OSes can communicate more directly with
the hypervisor, resulting in improved
efficiency.
2. Examples:
Xen is a widely-used hypervisor that
supports para virtualization. It is known for
its performance and scalability in
virtualized environments.
Xen Server Architecture
Xen is a Type-1 (bare-metal) hypervisor
that runs directly on hardware and
manages multiple virtual machines.

Its architecture is distinctive because it

uses a privileged control domain
(Dom0) to manage guest VMs (DomU).
Architectural Layers

1. Hardware Layer
Physical CPU, RAM, disk, NIC, GPU

Supports virtualization extensions (Intel VT-x / AMD-V)

2. Xen Hypervisor (Core Layer)
Thin microkernel layer Runs directly on hardware

Responsible for: CPU scheduling Memory management Interrupt

handling Isolation between VMs
3. Domain 0 (Dom0) – Control Domain
● First VM started by Xen at boot

● Privileged domain

● Has direct access to hardware devices

● Contains:

○ Device drivers

○ Management tools (xl, xapi, etc.)

○ Backend drivers for I/O virtualization

Functions:

● Creates and destroys guest VMs

● Allocates CPU and memory

● Handles disk and network I/O for guests

4. Domain U (DomU) – Guest VMs
Unprivileged guest virtual machines

Run user operating systems (Linux, Windows)

Use frontend drivers to communicate with Dom0

No direct hardware access

Rings in Xen Architecture

Ring Component

Ring 0 Xen Hypervisor

Ring 1 Guest OS Kernel (DomU)

Ring 3 User Applications

Xen Architecture Flow (Stepwise)
● Hardware boots Xen hypervisor

● Xen starts Dom0

● Dom0 initializes drivers

● Dom0 creates DomU guest VMs

● Guests communicate via frontend → backend drivers

● Hypervisor enforces isolation

summary
Xen architecture consists of a minimal bare-metal hypervisor, a privileged control
domain (Dom0) that manages hardware and virtual machines, and multiple
unprivileged guest domains (DomU) that run user operating systems with isolated
resource access.
Containerization using Docker
Containerization A OS-level virtualization

Containerization is an OS-level
virtualization technique that packages
an application along with its
dependencies (libraries, runtime,
binaries, configuration files) into a
single lightweight unit called a
container.

Unlike traditional virtual machines (VMs), containers:

● Share the host OS kernel

● Are lightweight and fast to start

● Provide process-level isolation

Why Containerization?

Traditional Deployment Problem Docker Solution

“Works on my machine” issue Consistent environment everywhere

Dependency conflicts Isolated containers

Heavy virtual machines Lightweight containers

Slow deployment Rapid startup

 Feature Docker Containers Virtual Machines

OS Share Host OS Separate OS

Size MBs GBs

Boot Time Seconds Minutes

Performance Near-native Slightly slower

Isolation Process level Full OS level

Docker Architecture
Components:
1. Docker Client

○ Runs commands (docker build, docker run)

○ Communicates with Docker Daemon

2. Docker Daemon (dockerd)

○ Manages images, containers, networks, volumes

1. Docker Images

○ Read-only templates

○ Blueprint of container

2. Docker Containers

○ Running instance of an image

3. Docker Registry

○ Stores Docker images

○ Example: Docker Inc.’s Docker Hub

Important Docker Concepts
3. Container
1. Dockerfile
Running instance of image.
Script that defines how an image is built.
4. Volumes
2. Image
Persistent storage mechanism.
Immutable template for containers.
5. Networks

Container-to-container communication.
Advantages of Docker

● Portability across environments

● Faster deployment cycles

● Efficient resource utilization

● Scalability (especially with orchestration tools like Kubernetes)

● Microservices-friendly architecture
Kubernetes
Kubernetes (K8s) is an open-source container orchestration platform used to automate:

● Deployment

● Scaling

● Load balancing

● Self-healing

● Rolling updates
Why Kubernetes?
Problem Kubernetes Solution
When applications scale
Multiple containers to manage Centralized orchestration
beyond a single container or
server, manual management Traffic distribution Built-in Service load
balancing
becomes impractical.
Failure recovery Self-healing (auto-restart)

Scaling based on demand Horizontal Pod Autoscaling

Rolling updates without Rolling deployment strategy

downtime
 Feature Docker Kubernetes

Purpose Containerization Orchestration

Scope Single host Cluster-wide

Scaling Manual Automatic

Self-healing No Yes
Auto-Scaling

Auto-Scaling and Load Balancing (Cloud & Kubernetes

Context)

These are two foundational mechanisms in distributed

systems and cloud-native architecture to ensure
performance, availability, and cost-efficiency.
Auto-Scaling
Auto-scaling is the dynamic adjustment of compute resources (containers, VMs, pods, instances)
based on workload demand.

It ensures:

● No resource underutilization

● No performance degradation during traffic spikes

● Cost optimization
Types of Auto-Scaling
A. Horizontal Scaling (Scale Out / B. Vertical Scaling (Scale Up / Scale
Scale In) Down)
● Increase or decrease number of ● Increase CPU/RAM of existing instance
instances

● Example: 3 pods → 10 pods ● Example: 4GB RAM → 16GB RAM

Used in:
Limitation:
● Kubernetes (Horizontal Pod
Autoscaler) ● Hardware bound
● Amazon Web Services EC2 Auto
Scaling ● May require restart
Load Balancing
Load balancing distributes incoming network traffic across multiple
servers/pods to:

● Prevent overload

● Improve response time

● Increase reliability
Types of Load Balancing
A. Layer 4 (Transport
Layer) B. Layer 7 (Application
Layer)
● Based on IP + Port
● Based on URL, HTTP
● TCP/UDP level header

● Faster ● Content-aware routing

● Used in microservices
 Feature Auto-Scaling Load Balancing

Purpose Adjust capacity Distribute traffic

Trigger Resource metrics Incoming requests

Function Adds/removes instances Routes traffic

Goal Performance + Cost optimization Availability + Throughput

GPU Virtualization
GPU virtualization enables multiple virtual machines or workloads to share a single
physical GPU securely and efficiently.

It improves:

● Resource utilization

● Multi-tenant isolation

● Cost efficiency in AI clusters

Types of GPU Virtualization
A. Pass-Through (Direct Assignment)

Entire GPU assigned to one VM

Near-native performance

No sharing between VMs

Uses PCIe passthrough (VFIO)

B. vGPU (Virtual GPU)

Physical GPU divided into multiple virtual GPUs

Each VM gets dedicated memory slice

Managed by hypervisor
. API Remoting / GPU Sharing
C

● Applications share GPU via driver layer

● Used in cloud environments

● Less isolation compared to vGPU

Role of cuda

Homework