Chapter 5 – Cloud Resource

Virtualization
Contents
1. Virtualization.
2. Layering and virtualization.
3. Virtual machine monitor.
4. Virtual machine.
5. Performance and security isolation.
6. Architectural support for virtualization.
7. x86 support for virtualization.
8. Full and paravirtualization.
9. Xen 1.0 and Xen 2.0.
10. Performance comparison of virtual machine monitors.
11. The darker side of virtualization.
12. Software fault isolation.
Dan Marinescu Cloud Computing Third Edition - Chapter 5 2
Motivation
 There are many physical realizations of the fundamental
abstractions necessary to describe the operation of a computing
systems.
 Interpreters.
 Memory.
 Communications links.
 Virtualization is a basic tenet of cloud computing, it simplifies the
management of physical resources for the three abstractions.
 The state of a virtual machine (VM) running under a virtual machine
monitor (VMM) can de saved and migrated to another server to
balance the load.
 Virtualization allows users to operate in environments they are
familiar with, rather than forcing them to idiosyncratic ones.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 3

Motivation (cont’d)

 Cloud resource virtualization is important for:

 System security, as it allows isolation of services running on
the same hardware.
 Performance and reliability, as it allows applications to migrate
from one platform to another.
 The development and management of services offered by a
provider.
 Performance isolation.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 4

Virtualization
 Simulates the interface to a physical object by:
 Multiplexing: creates multiple virtual objects from one instance
of a physical object. Example - a processor is multiplexed
among a number of processes or threads.
 Aggregation: creates one virtual object from multiple physical
objects. Example - a number of physical disks are aggregated
into a RAID disk.
 Emulation: constructs a virtual object from a different type of a
physical object. Example - a physical disk emulates a Random
Access Memory (RAM).
 Multiplexing and emulation. Examples - virtual memory with
paging multiplexes real memory and disk; a virtual address
emulates a real address.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 5

Layering
 Layering – a common approach to manage system complexity.
 Minimizes the interactions among the subsystems of a complex
system.
 Simplifies the description of the subsystems; each subsystem is
abstracted through its interfaces with the other subsystems.
 We are able to design, implement, and modify the individual
subsystems independently.
 Layering in a computer system.
 Hardware.
 Software.
 Operating system.

 Libraries.

 Applications.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 6

Interfaces
 Instruction Set Architecture (ISA) – at the boundary between
hardware and software.

 Application Binary Interface (ABI) – allows the ensemble consisting

of the application and the library modules to access the hardware;
the ABI does not include privileged system instructions, instead it
invokes system calls.

 Application Program Interface (API) - defines the set of instructions

the hardware was designed to execute and gives the application
access to the ISA; it includes HLL library calls which often invoke
system calls.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 7

 A1 Applications

API

Libraries A2

ABI
System calls

Operating System A3

ISA
System ISA User ISA
Hardware

Application Programming Interface, Application Binary Interface,

and Instruction Set Architecture . An application uses library
functions (A1), makes system calls (A2), and executes machine
instructions (A3).
Cloud Computing Third Edition - Chapter
Dan Marinescu 5 8
Code portability
 Binaries created by a compiler for a specific ISA and a specific
operating systems are not portable.

 It is possible, though, to compile a HLL program for a virtual

machine (VM) environment where portable code is produced
and distributed and then converted by binary translators to the
ISA of the host system.
A dynamic binary translation converts blocks of guest
instructions from the portable code to the host instruction and
leads to a significant performance improvement, as such blocks
are cached and reused

Dan Marinescu Cloud Computing Third Edition - Chapter 5 9

 HLL code

Compiler front-end Compiler

Intermediate Portable
code code

Compiler back-end VM loader

Object code VM image

VM compiler/ VM compiler/
Loader
interpreter interpreter

Memory Memory Memory

image image ISA-1 image ISA-2

Cloud Computing Third Edition - Chapter

Dan Marinescu 5 10
Virtual machine monitor (VMM / hypervisor)
 Partitions the resources of computer system into one or more virtual
machines (VMs). Allows several operating systems to run
concurrently on a single hardware platform.

 A VMM allows
 Multiple services to share the same platform.
 Live migration - the movement of a server from one platform to
another.
 System modification while maintaining backward compatibility
with the original system.
 Enforces isolation among the systems, thus security.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 11

VMM virtualizes the CPU and the memory
 A VMM
 Traps the privileged instructions executed by a guest OS and
enforces the correctness and safety of the operation.
 Traps interrupts and dispatches them to the individual guest
operating systems.
 Controls the virtual memory management.
 Maintains a shadow page table for each guest OS and replicates
any modification made by the guest OS in its own shadow page
table. This shadow page table points to the actual page frame
and it is used by the Memory Management Unit (MMU) for
dynamic address translation.
 Monitors the system performance and takes corrective actions to
avoid performance degradation. For example, the VMM may
swap out a Virtual Machine to avoid thrashing.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 12

Virtual machines (VMs)

 VM - isolated environment that appears to be a whole computer,

but actually only has access to a portion of the computer
resources.
 Process VM - a virtual platform created for an individual process
and destroyed once the process terminates.
 System VM - supports an operating system together with many
user processes.
 Traditional VM - supports multiple virtual machines and runs
directly on the hardware.
 Hybrid VM - shares the hardware with a host operating system
and supports multiple virtual machines.
 Hosted VM - runs under a host operating system.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 13

Traditional, hybrid, and hosted VMs
Process VMs System VMs

Same ISA Different ISA Same ISA Different ISA

Application Application

Multi Dynamic Traditional Whole Guest Guest

program translators VM system VM OS -1 OS -n

Binary
optimizers HLL VMs
Hybrid VM
Codesigned VM-1 VM-n
VM

Virtual Machine Monitor

Hosted VM

Hardware

(a) (b)

Application Application
Application
Guest OS -1 Guest OS -n
Application
Application
Application

VM-1 VM-n
Guest OS
Virtual Machine Monitor

Host OS VMM Host OS

Hardware Hardware

(c) (d)

Dan Marinescu Cloud Computing Third Edition - Chapter 5 14

Dan Marinescu Cloud Computing Third Edition - Chapter 5 15
Performance and security isolation
 The run-time behavior of an application is affected by other
applications running concurrently on the same platform and
competing for CPU cycles, cache, main memory, disk and network
access. Thus, it is difficult to predict the completion time!

 Performance isolation - a critical condition for QoS guarantees in

shared computing environments.

 A VMM is a much simpler and better specified system than a

traditional operating system. Example - Xen has approximately
60,000 lines of code; Denali has only about half, 30,000.

 The security vulnerability of VMMs is considerably reduced as the

systems expose a much smaller number of privileged functions.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 16

Computer architecture and virtualization
 Conditions for efficient virtualization:
 A program running under the VMM should exhibit a behavior
essentially identical to that demonstrated when running on an
equivalent machine directly.
 The VMM should be in complete control of the virtualized resources.
 A statistically significant fraction of machine instructions must be
executed without the intervention of the VMM.
 Two classes of machine instructions:
 Sensitive - require special precautions at execution time:
 Control sensitive - instructions that attempt to change either the

memory allocation or the privileged mode.

 Mode sensitive - instructions whose behavior is different in the

privileged mode.
 Innocuous - not sensitive.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 17

Full virtualization and paravirtualization
 Full virtualization – a guest OS can run unchanged under the VMM
as if it was running directly on the hardware platform.
 Requires a virtualizable architecture.
 Examples: Vmware.

 Paravirtualization - a guest operating system is modified to use only

instructions that can be virtualized. Reasons for paravirtualization:
 Some aspects of the hardware cannot be virtualized.
 Improved performance.
 Present a simpler interface.

Examples: Xen, Denaly

Dan Marinescu Cloud Computing Third Edition - Chapter 5 18

Full virtualization and paravirtualization

Guest OS Guest OS
Hardware Hardware
abstraction abstraction
layer layer

Hypervisor Hypervisor

Hardware Hardware

(a) Full virtualization (b) Paravirtualization

Dan Marinescu Cloud Computing Third Edition - Chapter 5 19

Virtualization of x86 architecture
 Ring de-privileging - a VMMs forces the operating system and the
applications to run at a privilege level greater than 0.
 Ring aliasing - a guest OS is forced to run at a privilege level other
than that it was originally designed for.
 Address space compression - a VMM uses parts of the guest
address space to store several system data structures.
 Non-faulting access to privileged state - several store instructions
can only be executed at privileged level 0 because they operate on
data structures that control the CPU operation. They fail silently
when executed at a privilege level other than 0.
 Guest system calls which cause transitions to/from privilege level 0
must be emulated by the VMM.
 Interrupt virtualization - in response to a physical interrupt, the VMM
generates a ``virtual interrupt'' and delivers it later to the target guest
OS which can mask interrupts.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 20

Virtualization of x86 architecture (cont’d)
 Access to hidden state - elements of the system state, e.g.,
descriptor caches for segment registers, are hidden; there is no
mechanism for saving and restoring the hidden components when
there is a context switch from one VM to another.
 Ring compression - paging and segmentation protect VMM code
from being overwritten by guest OS and applications. Systems
running in 64-bit mode can only use paging, but paging does not
distinguish between privilege levels 0, 1, and 2, thus the guest OS
must run at privilege level 3, the so called (0/3/3) mode. Privilege
levels 1 and 2 cannot be used thus, the name ring compression.
 The task-priority register is frequently used by a guest OS; the
VMM must protect the access to this register and trap all attempts
to access it. This can cause a significant performance degradation.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 21

VT-x, a major architectural enhancement

 Supports two modes of operations:

 VMX root - for VMM operations.
 VMX non-root - support a VM.

 The Virtual Machine Control Structure including host-state and

guest-state areas.
 VM entry - the processor state is loaded from the guest-state of the VM
scheduled to run; then the control is transferred from VMM to the VM.
 VM exit - saves the processor state in the guest-state area of the
running VM; then it loads the processor state from the host-state area,
finally transfers control to the VMM.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 22

 VT- x

Virtual-machine control structure

VM entry
host-state
VMX root VMX non-root
guest-state
VM exit

(a) (b)

Dan Marinescu Cloud Computing Third Edition - Chapter 5 23

VT-d, a new virtualization architecture

 I/O MMU virtualization gives VMs direct access to

peripheral devices.
 VT-d supports:
 DMA address remapping, address translation for device DMA
transfers.
 Interrupt remapping, isolation of device interrupts and VM
routing.
 I/O device assignment, the devices can be assigned by an
administrator to a VM in any configurations.
 Reliability features, it reports and records DMA and interrupt
errors that my otherwise corrupt memory and impact VM
isolation.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 24

Layering
 Layering – a common approach to manage system complexity.
 Minimizes the interactions among the subsystems of a complex
system.
 Simplifies the description of the subsystems; each subsystem is
abstracted through its interfaces with the other subsystems.
 We are able to design, implement, and modify the individual
subsystems independently.
 Layering in a computer system.
 Hardware.
 Software.
 Operating system.
 Libraries.

Applications.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 25

Interfaces
 Instruction Set Architecture (ISA) – at the boundary between
hardware and software.

 Application Binary Interface (ABI) – allows the ensemble consisting

of the application and the library modules to access the hardware;
the ABI does not include privileged system instructions, instead it
invokes system calls.

 Application Program Interface (API) - defines the set of instructions

the hardware was designed to execute and gives the application
access to the ISA; it includes HLL library calls which often invoke
system calls.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 26

 A1 Applications

API

Libraries A2

ABI
System calls

Operating System A3

ISA
System ISA User ISA
Hardware

Application Programming Interface, Application Binary Interface,

and Instruction Set Architecture . An application uses library
functions (A1), makes system calls (A2), and executes machine
instructions (A3).

Dan Marinescu Cloud Computing Third Edition - Chapter 5 27

Code portability

 Binaries created by a compiler for a specific ISA and a specific

operating systems are not portable.
 It is possible, though, to compile a HLL program for a virtual
machine (VM) environment where portable code is produced
and distributed and then converted by binary translators to the
ISA of the host system.
A dynamic binary translation converts blocks of guest
instructions from the portable code to the host instruction and
leads to a significant performance improvement, as such blocks
are cached and reused

Dan Marinescu Cloud Computing Third Edition - Chapter 5 28

 HLL code

Compiler front-end Compiler

Intermediate Portable
code code

Compiler back-end VM loader

Object code VM image

VM compiler/ VM compiler/
Loader
interpreter interpreter

Memory Memory Memory

image image ISA-1 image ISA-2

Dan Marinescu Cloud Computing Third Edition - Chapter 5 29

Hypervisor or virtual machine monitor (VMM )

 Partitions the resources of computer system into one or more virtual

machines (VMs).
 Allows several operating systems to run concurrently on a single
hardware platform.
 A hypervisor allows
 Multiple services to share the same platform.
 Live migration - the movement of a server from one platform to another.
 System modification while maintaining backward compatibility with the
original system.
 Enforces isolation among the systems, thus security.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 30

A hypervisor virtualizes CPU and memory
 A hypervisor:
 Traps privileged instructions executed by a guest OS and
enforces the correctness and safety of the operation.
 Traps interrupts and dispatches them to the individual guest
operating systems.
 Controls the virtual memory management.
 Maintains a shadow page table for each guest OS and replicates
any modification made by the guest OS in its own shadow page
table. This shadow page table points to the actual page frame
and it is used by the Memory Management Unit (MMU) for
dynamic address translation.
 Monitors the system performance and takes corrective actions to
avoid performance degradation. For example, the VMM may
swap out a Virtual Machine to avoid thrashing.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 31

Virtual machines (VMs)

 VM - isolated environment that appears to be a whole computer,

but actually only has access to a portion of the computer
resources.
 Process VM - a virtual platform created for an individual process
and destroyed once the process terminates.
 System VM - supports an operating system together with many
user processes.
 Traditional VM - supports multiple virtual machines and runs
directly on the hardware.
 Hybrid VM - shares the hardware with a host operating system
and supports multiple virtual machines.
 Hosted VM - runs under a host operating system.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 32

Traditional, hybrid, and hosted VMs
Process VMs System VMs

Same ISA Different ISA Same ISA Different ISA

Application Application

Multi Dynamic Traditional Whole Guest Guest

program translators VM system VM OS -1 OS -n

Binary
optimizers HLL VMs
Hybrid VM
Codesigned VM-1 VM-n
VM

Virtual Machine Monitor

Hosted VM

Hardware

(a) (b)

Application Application
Application
Guest OS -1 Guest OS -n
Application
Application
Application

VM-1 VM-n
Guest OS
Virtual Machine Monitor

Host OS VMM Host OS

Hardware Hardware

(c) (d)

Dan Marinescu Cloud Computing Third Edition - Chapter 5 33

Dan Marinescu Cloud Computing Third Edition - Chapter 5 34
Performance and security isolation
 The run-time behavior of an application is affected by other
applications running concurrently on the same platform and
competing for CPU cycles, cache, main memory, disk and network
access. Thus, it is difficult to predict the completion time!
 Performance isolation - a critical condition for QoS guarantees in
shared computing environments.
 A VMM is a much simpler and better specified system than a
traditional operating system. Example - Xen has approximately
60,000 lines of code; Denali has only about half, 30,000.
 The security vulnerability of VMMs is considerably reduced as the
systems expose a much smaller number of privileged functions.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 35

Architectural support for virtualization
 Conditions for efficient virtualization:
 A program running under the VMM should exhibit a behavior
essentially identical to that demonstrated when running on an
equivalent machine directly.
 The VMM should be in complete control of the virtualized resources.
 A statistically significant fraction of machine instructions must be
executed without the intervention of the VMM.
 Two classes of machine instructions:
 Sensitive - require special precautions at execution time:
 Control sensitive - instructions that attempt to change either the

memory allocation or the privileged mode.

 Mode sensitive - instructions whose behavior is different in the

privileged mode.
 Innocuous - not sensitive.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 36

Full virtualization and paravirtualization

 Full virtualization – a guest OS can run unchanged under the VMM

as if it was running directly on the hardware platform.
 Requires a virtualizable architecture.
 Examples: Vmware.

 Paravirtualization - a guest operating system is modified to use only

instructions that can be virtualized. Reasons for paravirtualization:
 Some aspects of the hardware cannot be virtualized.
 Improved performance.
 Present a simpler interface.

Examples: Xen, Denaly

Dan Marinescu Cloud Computing Third Edition - Chapter 5 37

Full virtualization and paravirtualization

Guest OS Guest OS
Hardware Hardware
abstraction abstraction
layer layer

Hypervisor Hypervisor

Hardware Hardware

(a) Full virtualization (b) Paravirtualization

Dan Marinescu Cloud Computing Third Edition - Chapter 5 38

Virtualization of x86 architecture
 Ring de-privileging - a VMMs forces the operating system and the
applications to run at a privilege level greater than 0.
 Ring aliasing - a guest OS is forced to run at a privilege level other
than that it was originally designed for.
 Address space compression - a VMM uses parts of the guest
address space to store several system data structures.
 Non-faulting access to privileged state - several store instructions
can only be executed at privileged level 0 because they operate on
data structures that control the CPU operation. They fail silently
when executed at a privilege level other than 0.
 Guest system calls which cause transitions to/from privilege level 0
must be emulated by the VMM.
 Interrupt virtualization - in response to a physical interrupt, the VMM
generates a ``virtual interrupt'' and delivers it later to the target guest
OS which can mask interrupts.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 39

Virtualization of x86 architecture (Cont’d)
 Access to hidden state - elements of the system state, e.g.,
descriptor caches for segment registers, are hidden; there is no
mechanism for saving and restoring the hidden components when
there is a context switch from one VM to another.
 Ring compression - paging and segmentation protect VMM code
from being overwritten by guest OS and applications. Systems
running in 64-bit mode can only use paging, but paging does not
distinguish between privilege levels 0, 1, and 2, thus the guest OS
must run at privilege level 3, the so called (0/3/3) mode. Privilege
levels 1 and 2 cannot be used thus, the name ring compression.
 The task-priority register is frequently used by a guest OS; the
VMM must protect the access to this register and trap all attempts
to access it. This can cause a significant performance degradation.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 40

VT-x, a major architectural enhancement

 Supports two modes of operations:

 VMX root - for VMM operations.
 VMX non-root - support a VM.

 The Virtual Machine Control Structure including host-state and

guest-state areas.
 VM entry - the processor state is loaded from the guest-state of the VM
scheduled to run; then the control is transferred from VMM to the VM.
 VM exit - saves the processor state in the guest-state area of the
running VM; then it loads the processor state from the host-state area,
finally transfers control to the VMM.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 41

 VT- x

Virtual-machine control structure

VM entry
host-state
VMX root VMX non-root
guest-state
VM exit

(a) (b)

Dan Marinescu Cloud Computing Third Edition - Chapter 5 42

VT-d, a new virtualization architecture

 I/O MMU virtualization gives VMs direct access to

peripheral devices.
 VT-d supports:
 DMA address remapping, address translation for device DMA
transfers.
 Interrupt remapping, isolation of device interrupts and VM
routing.
 I/O device assignment, the devices can be assigned by an
administrator to a VM in any configurations.
 Reliability features, it reports and records DMA and interrupt
errors that my otherwise corrupt memory and impact VM
isolation.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 43

Xen - a VMM based on paravirtualization
 The goal of the Cambridge group - design a VMM capable of scaling
to about 100 VMs running standard applications and services
without any modifications to the Application Binary Interface (ABI).
 Linux, Minix, NetBSD, FreeBSD, NetWare, and OZONE can operate
as paravirtualized Xen guest OS running on x86, x86-64, Itanium,
and ARM architectures.
 Xen domain - ensemble of address spaces hosting a guest OS and
applications running under the guest OS. Runs on a virtual CPU.
 Dom0 - dedicated to execution of Xen control functions and privileged
instructions.
 DomU - a user domain.

 Applications make system calls using hypercalls processed

by Xen; privileged instructions issued by a guest OS are
paravirtualized and must be validated by Xen.

Dan Marinescu Cloud Computing Third Edition - Chapte 44

r5
 Xen
Management
OS Application Application Application

Guest OS Guest OS Guest OS

Xen-aware
device drivers
Xen-aware Xen-aware Xen-aware
device drivers device drivers device drivers

Xen
Domain0 control Virtual x86 Virtual physical Virtual block
interface Virtual network
CPU memory devices

X86 hardware

Dan Marinescu Cloud Computing Third Edition - Chapter 5 45

Xen implementation on x86 architecture
 Xen runs at privilege Level 0, the guest OS at Level 1, and
applications at Level 3.
 The x86 architecture does not support either the tagging of TLB
entries or the software management of the TLB. Thus, address
space switching, when the VMM activates a different OS, requires a
complete TLB flush; this has a negative impact on the performance.
 Solution - load Xen in a 64 MB segment at the top of each address
space and delegate the management of hardware page tables to
the guest OS with minimal intervention from Xen. This region is not
accessible or re-mappable by the guest OS.
 Xen schedules individual domains using the Borrowed Virtual Time
(BVT) scheduling algorithm.
 A guest OS must register with Xen a description table with the
addresses of exception handlers for validation.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 46

Dom0 components
 XenStore – a Dom0 process.
 Supports a system-wide registry and naming service.
 Implemented as a hierarchical key-value storage.
 A watch function informs listeners of changes of the key in storage
they have subscribed to.
 Communicates with guest VMs via shared memory using Dom0
privileges.
 Toolstack - responsible for creating, destroying, and managing the
resources and privileges of VMs.
 To create a new VM, a user provides a configuration file describing
memory and CPU allocations and device configurations.
 Toolstack parses this file and writes this information in XenStore.
 Takes advantage of Dom0 privileges to map guest memory, to load a
kernel and virtual BIOS and to set up initial communication channels
with XenStore and with the virtual console when a new VM is created.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 47

Strategies for virtual memory management, CPU multiplexing, and
I/O devices

Dan Marinescu Cloud Computing Third Edition - Chapter 5 48

Xen abstractions for networking and I/O
 Each domain has one or more Virtual Network Interfaces (VIFs)
which support the functionality of a network interface card. A VIF is
attached to a Virtual Firewall-Router (VFR).
 Split drivers have a front-end in the DomU and the back-end in
Dom0; the two communicate via a ring in shared memory.
 Ring - a circular queue of descriptors allocated by a domain and
accessible within Xen. Descriptors do not contain data, the data
buffers are allocated off-band by the guest OS.
 Two rings of buffer descriptors, one for packet sending and one for
packet receiving, are supported.
 To transmit a packet:
 a guest OS enqueues a buffer descriptor to the send ring,
 then Xen copies the descriptor and checks safety,
 copies only the packet header, not the payload, and
 executes the matching rules.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 49

 I/O channel
Driver domain Guest domain
Bridge

Backend Frontend

Network
interface

Event channel

XEN

NIC
(a)

Request queue

Consumer Request Producer Request

(private pointer in Xen) (shared pointer updated
by the guest OS)

Outstanding Unused
descriptors descriptors

Consumer Response
Producer Response
(private pointer maintained by
(shared pointer updated Response queue the guest OS)
by Xen)
(b)

Xen zero-copy semantics for data transfer using I/O rings. (a) The communication
between a guest domain and the driver domain over an I/O and an event channel;
NIC is the Network Interface Controller. (b) the circular ring of buffers.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 50

Xen 2.0
 Optimization of:
 Virtual interface - takes advantage of the capabilities of some
physical NICs, such as checksum offload.
 I/O channel - rather than copying a data buffer holding a packet,
each packet is allocated in a new page and then the physical
page containing the packet is re-mapped into the target
domain.
 Virtual memory - takes advantage of the superpage and global
page mapping hardware on Pentium and Pentium Pro
processors. A superpage entry covers 1,024 pages of physical
memory and the address translation mechanism maps a set of
contiguous pages to a set of contiguous physical pages. This
helps reduce the number of TLB misses.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 51

 Driver domain Guest domain Driver domain Guest domain

Bridge Bridge

Offload
I/O
Driver I/O
channel channel High Level
NIC Backend Virtual NIC Backend Virtual
Driver Interface Interface Driver Interface
Interface

Physical Xen VMM Physical Xen VMM

NIC NIC

(a) (b)

Xen network architecture .(a) The original architecture;

(b) The optimized architecture

Dan Marinescu Cloud Computing Third Edition - Chapter 5 52

 A comparison of send and receive data rates for a native Linux system, the Xen
driver domain, an original Xen guest domain, and an optimized Xen guest domain.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 53

Performance comparison of virtual machines
 Compare the performance of Xen and OpenVZwith, a standard
operating system, a plain vanilla Linux.
 The questions examined are:
 How the performance scales up with the load?
 What is the impact of a mix of applications?
 What are the implications of the load assignment on individual
servers?
 The main conclusions:
 The virtualization overhead of Xen is considerably higher than that of
OpenVZ and that this is due primarily to L2-cache misses.
 The performance degradation when the workload increases is also
noticeable for Xen.
 Hosting multiple tiers of the same application on the same server is
not an optimal solution.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 54

 Linux OpenVZ Xen

Web Web Web

server server server

MySQL MySQL MySQL

server server server

(a)

Linux OpenVZ Xen

Web MySQL Web MySQL Web MySQL

server server server server server server

(b)

Linux OpenVZ Xen

Web MySQL Web MySQL Web MySQL
server server server server server server
Web MySQL Web MySQL Web MySQL
server server server server server server
Web MySQL Web MySQL Web MySQL
server server server server server server
Web MySQL Web MySQL Web MySQL
server server server server server server

(c)

The setup for the performance comparison of a native Linux system with OpenVZ, and
the Xen systems. The applications are a web server and a MySQL database server. (a)
The first experiment, the web and the DB, share a single system; (b) The second
experiment, the web and the DB, run on two different systems; (c) The third experiment,
the web and the DB, run on two different systems and each has four instances.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 55

The darker side of virtualization
 In a layered structure, a defense mechanism at some layer can be
disabled by malware running at a layer below it.
 It is feasible to insert a rogue VMM, a Virtual-Machine Based
Rootkit (VMBR) between the physical hardware and an operating
system.
 Rootkit - malware with a privileged access to a system.
 The VMBR can enable a separate malicious OS to run
surreptitiously and make this malicious OS invisible to the guest OS
and to the application running under it.
 Under the protection of the VMBR, the malicious OS could:
 observe the data, the events, or the state of the target system.
 run services, such as spam relays or distributed denial-of-service
attacks.
 interfere with the application.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 56

 Application

Application
Malicious Guest OS
OS
Operating
Malicious system (OS)
OS Virtual machine monitor

Virtual machine based rootkit Virtual machine based rootkit

Hardware Hardware

(a) (b)
The insertion of a Virtual-Machine Based Rootkit (VMBR) as the lowest
layer of the software stack running on the physical hardware; (a) below an
operating system; (b) below a legitimate virtual machine monitor. The
VMBR enables a malicious OS to run surreptitiously and makes it invisible
to the genuine or the guest OS and to the application.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 57

The features of the SFI for the Native Client on the x86-32, x86-64 , and ARM.

Cloud Computing Third Edition - Chapter

Dan Marinescu 5 58
Virtualization
 Simulates the interface to a physical object by:
 Multiplexing: creates multiple virtual objects from one instance
of a physical object. Example - a processor is multiplexed
among a number of processes or threads.
 Aggregation: creates one virtual object from multiple physical
objects. Example - a number of physical disks are aggregated
into a RAID disk.
 Emulation: constructs a virtual object from a different type of a
physical object. Example - a physical disk emulates a Random
Access Memory (RAM).
 Multiplexing and emulation. Examples - virtual memory with
paging multiplexes real memory and disk; a virtual address
emulates a real address.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 59

Virtualization (Cont’d)
 Three fundamental abstractions necessary to describe the
operation of a computing systems.
 Interpreters – active hardware and software components
which transform data. E.g. processors, programs in
execution.
 Memory – repository of data.
 Communications links – allow hardware and software
components to interact with one anpther .
 Virtualization is a basic tenet of cloud computing, it simplifies
the management of physical resources for the three
abstractions.
 Provides benefits and creates problems for cloud users and for
cloud service providers.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 60

Virtualization  user benefits versus concerns
 Users operate in environments they are familiar with, rather than
forcing them to idiosyncratic ones.
 Applications can migrate from one platform to another.
 Support performance isolation important for application optimization
and QoS (Quality of Service) assurance.
 Adds overhead and increases the execution time. The hypervisor is
invoked by the OS when applications make systems calls.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 61

Virtualization  system benefits versus concerns
 Simplifies the development and management of services offered by
a CSP.
 Allows isolation of services running on the same hardware.
 Important for load balancing. The state of a virtual machine (VM)
running under a hypervisor can de saved and migrated to another
server to balance the load.
 Increases the size of software stack.
 Complicates software maintenance. Saved VMs are not updated
when OS and other system software patches are applied.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 62

Virtual machines (VMs)

 VM - isolated environment that appears to be a whole computer,

but actually only has access to a portion of the computer
resources.
 Process VM - a virtual platform created for an individual process
and destroyed once the process terminates.
 System VM - supports an operating system together with many
user processes.
 Traditional VM - supports multiple virtual machines and runs
directly on the hardware.
 Hybrid VM - shares the hardware with a host operating system
and supports multiple virtual machines.
 Hosted VM - runs under a host operating system.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 63

Traditional, hybrid, and hosted VMs
Process VMs System VMs

Same ISA Different ISA Same ISA Different ISA

Application Application

Multi Dynamic Traditional Whole Guest Guest

program translators VM system VM OS -1 OS -n

Binary
optimizers HLL VMs
Hybrid VM
Codesigned VM-1 VM-n
VM

Virtual Machine Monitor

Hosted VM

Hardware

(a) (b)

Application Application
Application
Guest OS -1 Guest OS -n
Application
Application
Application

VM-1 VM-n
Guest OS
Virtual Machine Monitor

Host OS VMM Host OS

Hardware Hardware

(c) (d)

Dan Marinescu Cloud Computing Third Edition - Chapter 5 64

Hypervisors –CPU and memory virtualization
 A hypervisor:
 Traps the privileged instructions executed by a guest OS and
enforces the correctness and safety of the operation.
 Traps interrupts and dispatches them to the individual guest
operating systems.
 Controls the virtual memory management.
 Maintains a shadow page table for each guest OS and replicates
any modification made by the guest OS in its own shadow page
table. This shadow page table points to the actual page frame
and it is used by the Memory Management Unit (MMU) for
dynamic address translation.
 Monitors the system performance and takes corrective actions to
avoid performance degradation. For example, the VMM may
swap out a Virtual Machine to avoid thrashing.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 65

Xen - a VMM based on paravirtualization
 The goal of the Cambridge group - design a VMM capable of scaling
to about 100 VMs running standard applications and services
without any modifications to the Application Binary Interface (ABI).
 Linux, Minix, NetBSD, FreeBSD, NetWare, and OZONE can operate
as paravirtualized Xen guest OS running on x86, x86-64, Itanium,
and ARM architectures.
 Xen domain - ensemble of address spaces hosting a guest OS and
applications running under the guest OS. Runs on a virtual CPU.
 Dom0 - dedicated to execution of Xen control functions and privileged
instructions.
 DomU - a user domain.

 Applications make system calls using hypercalls processed

by Xen; privileged instructions issued by a guest OS are
paravirtualized and must be validated by Xen.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 66

 Virtualization under Xen
Management
OS Application Application Application

Guest OS Guest OS Guest OS

Xen-aware
device drivers
Xen-aware Xen-aware Xen-aware
device drivers device drivers device drivers

Xen
Domain0 control Virtual x86 Virtual physical Virtual block
interface Virtual network
CPU memory devices

X86 hardware

Dan Marinescu Cloud Computing Third Edition - Chapter 5 67

Dom0 components
 XenStore – a Dom0 process.
 Supports a system-wide registry and naming service.
 Implemented as a hierarchical key-value storage.
 A watch function informs listeners of changes of the key in storage
they have subscribed to.
 Communicates with guest VMs via shared memory using Dom0
privileges.
 Toolstack - responsible for creating, destroying, and managing the
resources and privileges of VMs.
 To create a new VM a user provides a configuration file describing
memory and CPU allocations and device configurations.
 Toolstack parses this file and writes this information in XenStore.
 Takes advantage of Dom0 privileges to map guest memory, to load a
kernel and virtual BIOS and to set up initial communication channels
with XenStore and with the virtual console when a new VM is created.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 68

Dan Marinescu Cloud Computing Third Edition - Chapter 5 69
Xen abstractions for networking and I/O
 Each domain has one or more Virtual Network Interfaces (VIFs)
which support the functionality of a network interface card. A VIF is
attached to a Virtual Firewall-Router (VFR).
 Split drivers have a front-end in the DomU and the back-end in
Dom0; the two communicate via a ring in shared memory.
 Ring - a circular queue of descriptors allocated by a domain and
accessible within Xen. Descriptors do not contain data, the data
buffers are allocated off-band by the guest OS.
 Two rings of buffer descriptors, one for packet sending and one for
packet receiving, are supported.
 To transmit a packet:
 a guest OS enqueues a buffer descriptor to the send ring,
 then Xen copies the descriptor and checks safety,
 copies only the packet header, not the payload, and
 executes the matching rules.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 70

 I/O channel
Driver domain Guest domain
Bridge

Backend Frontend

Network
interface

Event channel

XEN

NIC
(a)

Request queue

Consumer Request Producer Request

(private pointer in Xen) (shared pointer updated
by the guest OS)

Outstanding Unused
descriptors descriptors

Consumer Response
Producer Response
(private pointer maintained by
(shared pointer updated Response queue the guest OS)
by Xen)
(b)

Xen zero-copy semantics for data transfer using I/O rings. (a) The communication
between a guest domain and the driver domain over an I/O and an event channel;
NIC is the Network Interface Controller. (b) the circular ring of buffers.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 71

Xen 2.0 Optimizations
 Virtual interface - takes advantage of the capabilities of some
physical NICs, such as checksum offload.
 I/O channel - rather than copying a data buffer holding a packet,
each packet is allocated in a new page and then the physical page
containing the packet is re-mapped into the target domain.
 Virtual memory - takes advantage of the superpage and global
page mapping hardware on Pentium and Pentium Pro processors.
 A superpage entry covers 1,024 pages of physical memory and the
address translation mechanism maps a set of contiguous pages to
a set of contiguous physical pages. This helps reduce the number
of TLB misses.

Cloud Computing Third Edition - Chapter

Dan Marinescu 5 72
 Driver domain Guest domain Driver domain Guest domain

Bridge Bridge

Offload
I/O
Driver I/O
channel channel High Level
NIC Backend Virtual NIC Backend Virtual
Driver Interface Interface Driver Interface
Interface

Physical Xen VMM Physical Xen VMM

NIC NIC

(a) (b)

Xen network architecture. (a) Original architecture;

(b) Optimized architecture

Dan Marinescu Cloud Computing Third Edition - Chapter 5 73

A comparison of send and receive data rates for a native Linux
system, the Xen driver domain, an original Xen guest domain, and
an optimized Xen guest domain.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 74

Porting commodity operating systems to Xen

 The number of lines of code that had to be modified to port an OS to

Xen:
 3,000 lines of Linux code, or 1.36%..
 4,620 lines of Windows XP code is or about 0.04%

Dan Marinescu Cloud Computing Third Edition - Chapter 5 75

KVM – kernel VM
 Kernel-based Virtual Machine (KVM) is a virtualization infrastructure of
the Linux released as part of the 2.6.20 Linux kernel in 2007.
 KVM runs inside Linux as a driver handling the new virtualization
instructions exposed by hardware.
 KVM inherits all the new features of Linux in scheduling, memory
management, power management, and so on.
 KVM components:
 Generic host kernel module exposing architecture-independent functionality.
 Architecture-specific kernel module for the host system.
 User-space emulation of the VM hardware that the guest OS runs on.
 Guest OS performance optimization additions.
 When the guest OS performs a privileged operation the CPU exits and
KVM takes over. If KVM itself can service the request it then gives
control back to the guest.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 76

Dan Marinescu Cloud Computing Third Edition - Chapter 5 77
Nested virtualization
 System organization when a guest hypervisor runs inside a VM
which is itself running under a host hypervisor.
 Nested virtualization is useful for:
1. Experimenting with server setup or testing configurations.
2. Allows IaaS users to run their own hypervisor as a VM.
3. Can be also used for live migration of hypervisors together with their
guests VM for load balancing, for hypervisor-level protection, and for
supporting other security mechanisms.
4. Experiments with cloud interoperability alternatives.
 Nested virtualization is limited by the hardware support.
 Main concern – handling sensitive instructions.
 When the hardware supports multi-level nested virtualization, each
hypervisor handles all traps caused by sensitive instructions of
guest hypervisors running directly above of it.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 78

 Nested virtualization (Cont’d)

 Nested virtualization. KVM allows three VM to run concurrently. Two VMs

run hypervisors Xen and ESXi and the third runs Windows. A VM runs Linux
under Xen and another VM runs Windows under ESXi.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 79

Hardware support for nested virtualization

Nested virtualization with single-level hardware virtualization support. A trap is

handled by the L0 trap handler regardless of the hypervisor where a trap
occurs. The flow of control of nested traps:
(Left) Two-level, L0, L1, and L2 nested hypervisor(s);
(Right) Three-level, L0, L1, L2, and L3 nested hypervisor(s).

When hardware supports multi-level nested virtualization, each hypervisor

handles all traps caused by sensitive instructions of guest hypervisors running
directly above of it.
Dan Marinescu Cloud Computing Third Edition - Chapter 5 80
VMX support for virtualization
 VMX is an extension of the x86 Intel architecture for virtualization. It
adds ten new instructions permitting entering and exiting a virtual
execution mode.
 In this mode the guest OS views itself as running with full privileges
(running in protection ring 0) but the host OS remains full protected.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 81

 Virtual Machine Control Structure (VMCS)
 VMCS is used by the VMX extension to x86 architecture to track
host state and guest VMs as control is transferred between them.
 Three types of data are stored in VMCS:
 Guest state. Holds virtualized CPU registers (e.g., control registers or
segment registers) automatically loaded by the CPU when switching
from kernel mode to guest mode on VMEntry
 Host state. Data used by the CPU to restore register values when
switching back from guest mode to kernel mode on VMExit.
 Control data. Data used by the hypervisor to inject events such as
exceptions or interrupts into VMs and to specify which events should
cause a VMExit; it is also used by the CPU to specify the VMExit
reason.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 82

 Intel supports nested virtualization. KVM runs at level L0, Xen runs
at level L1 and KVM uses VMCS01 for the VM running Xen.
VMCS – Virtual machine control structure.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 83

Multiplexing multiple virtualization levels
 Intel VMX instructions can only be successfully executed in kernel
mode.
 A guest hypervisors at level Li operates in guest mode and
whenever it executes a VMX instruction to launch a level Li+1 guest,
the instruction is trapped and handled at level L0.
 Trapping execution exceptions enables the host hypervisor at level
L0 running in kernel mode to emulate VMX instruction executed by
guest hypervisors at level Li.
 This mechanism supports a critical idea for increasing efficiency of
nested virtualization, multiplexing multiple hypervisors.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 84

Dan Marinescu Cloud Computing Third Edition - Chapter 5 85
 Nested virtualization based on multiplexing

 As long as the host hypervisor at level L0 emulates faithfully the VMX

instruction set, a guest hypervisor at level L1 cannot distinguish if it is
running directly on the hardware or not. It follows that the guest
hypervisor at level L1 can lunch VMs using the standard mechanisms.
 The guest hypervisor can use the same technique to give another guest
hypervisor at level L2 the same illusion that it is running directly on the
hardware. The process can be extended, a hypervisor at level L i giving
the illusion that the one at level Li+1 is running directly on the hardware.
 The processor runs L1 and L2 maintained by L0 using VMCS0  1 and
VMCS0  2 environment specification, respectively. L1 creates VMCS0  2
within its own virtualized environment and the processor uses it to
emulate VMX for L1. Switching from one level to another is emulated.

Dan Marinescu Cloud Computing Third Edition - Chapter 5 86

Dan Marinescu Cloud Computing Third Edition - Chapter 5 87
Performance comparison of virtual machines
 Compare performance of Xen and OpenVZ with, a plain vanilla
Linux.
 Questions examined are:
 How the performance scales up with the load?
 What is the impact of a mix of applications?
 What are the implications of the load assignment on individual
servers?
 Main conclusions:
 The virtualization overhead of Xen is considerably higher than that of
OpenVZ, due primarily to L2-cache misses.
 Performance degradation when the workload increases is also
noticeable for Xen.
 Hosting multiple tiers of the same application on the same server is
not an optimal solution.

Dan Marinescu Cloud Computing Third Edition - Chapte 88

r5
 Linux OpenVZ Xen

Web Web Web

server server server

MySQL MySQL MySQL

server server server

(a)

Linux OpenVZ Xen

Web MySQL Web MySQL Web MySQL

server server server server server server

(b)

Linux OpenVZ Xen

Web MySQL Web MySQL Web MySQL
server server server server server server
Web MySQL Web MySQL Web MySQL
server server server server server server
Web MySQL Web MySQL Web MySQL
server server server server server server
Web MySQL Web MySQL Web MySQL
server server server server server server

(c)

The setup for the performance comparison of a native Linux system with OpenVZ, and
the Xen systems. The applications are a web server and a MySQL database server. (a)
The first experiment, the web and the DB, share a single system; (b) The second
experiment, the web and the DB, run on two different systems; (c) The third experiment,
the web and the DB, run on two different systems and each has four instances.

Dan Marinescu Cloud Computing Third Edition - Chapte 89

r5
The darker side of virtualization

 In a layered structure, a defense mechanism at some layer can be

disabled by malware running at a layer below it.
 It is feasible to insert a rogue VMM, a Virtual-Machine Based
Rootkit (VMBR) between the physical hardware and an operating
system.
 Rootkit - malware with a privileged access to a system.
 The VMBR can enable a separate malicious OS to run
surreptitiously and make this malicious OS invisible to the guest OS
and to the application running under it.
 Under the protection of the VMBR, the malicious OS could:
 observe the data, the events, or the state of the target system.
 run services, such as spam relays or distributed denial-of-service
attacks.
 interfere with the application.

Dan Marinescu Cloud Computing Third Edition - Chapte 90

r5
 Application

Application
Malicious Guest OS
OS
Operating
Malicious system (OS)
OS Virtual machine monitor

Virtual machine based rootkit Virtual machine based rootkit

Hardware Hardware

(a) (b)
Insertion of a Virtual-Machine Based Rootkit (VMBR) as the lowest layer of the
software stack running on the physical hardware below:
(a) An operating system;
(b) A legitimate hypervisor; TVMBR enables a malicious OS to run surreptitiously
and makes it invisible to genuine or guest OS and to application.

Dan Marinescu Cloud Computing Third Edition - Chapte 91

r5
8. Security risks posed by shared
images
 Image sharing poses security risks for IaaS.
 A study conducted during the period November 2010 - May 2011
analyzed AMIs available through the public catalog at Amazon
 5,303 Linux AMIs

 1,202 Windows AMIs.

 Many images analyzed allowed a user to undelete files, recover
credentials, private keys, or other types of sensitive information with
little effort and using standard tools.
 Critical software vulnerability revealed by the audit:
 98% of the Windows AMIs (249 out of 253)
 58% of Linux AMIs (2005 out of 3,432).
 The average number of vulnerabilities per AMI: 46 for Windows
AMIs and 11 for Linux AMIs.

Dan Marinescu Cloud Computing Third Edition - Chapte 92

r5
Security risks
 Three types of security risks were analyzed: (1) backdoors and
leftover credentials, (2) unsolicited connections, and (3) malware.
 To rent a Linux AMI a user must provide the public part of the her
ssh key and this key is stored in the authorized_keys in the home
directory. This opens a backdoor for a malicious creator of an AMI
who does not remove her own public key from the image and can
remotely login to any instance of this AMI.
 Another backdoor is opened when the ssh server allows password-
based authentication and the malicious creator of an AMI does not
remove her own password.
 This backdoor is even wider open as one can extract the password
hashes and then crack the passwords using a tool such as John the
Riper.

Dan Marinescu Cloud Computing Third Edition - Chapte 93

r5
More security risks
 Omission of the cloud-init script that should be invoked when the
image is booted. This script provided by Amazon regenerates the
host key an ssh server uses to identify itself; the public part of this
key is used to authenticate the server. When this key is shared
among several systems these systems become vulnerable to man-
in-the middle attacks.
 When this script does not run, an attacker can use the NMap tool to
match the ssh keys discovered in the AMI images with the keys
obtained with NMap. The study identified more than 2,100 instances
following this procedure.
 About 22% of the scanned Linux AMIs contained credentials
allowing an intruder to remotely login to the system. Some 100
passwords, 995 ssh keys, and 90 cases when both could be
retrieved were identified.

Dan Marinescu Cloud Computing Third Edition - Chapte 94

r5