Operating Systems: A Comprehensive Guide
Overview
An operating system (OS) is the software layer that manages computer hardware
and provides services for application software. It acts as a resource manager,
abstracts hardware complexity, and offers a stable execution environment. This
guide covers OS principles, process and thread management, concurrency, mem-
ory management, storage and filesystems, I/O, security, virtualization, kernel
architectures, performance, and hands-on labs.
Intended audience: undergraduate students, system programmers, and engineers
learning system-level concepts.
Table of Contents
1. History and Purpose of Operating Systems
2. OS Architecture: Kernel and User Space
3. Processes and Threads
4. CPU Scheduling
5. Concurrency: Synchronization and Deadlock
6. Memory Management and Virtual Memory
7. File Systems and Storage
8. I/O and Device Management
9. System Calls and Process Controls
10. Security and Access Control
11. Virtualization and Containers
12. Kernel Design: Monolithic vs Microkernel
13. Real-Time and Embedded OS
14. Performance, Monitoring, and Debugging
15. Hands-on Labs and Exercises
16. Glossary and Further Reading
1. History and Purpose of Operating Systems
Early computers ran single programs directly on hardware. As systems evolved,
OSes were introduced to multiplex CPU, memory, and I/O among multiple
programs. Key milestones: - Batch systems, time-sharing systems (1960s) -
UNIX (1970s): portable, multiuser OS with a powerful shell and POSIX APIs -
Personal OSes: DOS, Windows, Mac OS - Modern UNIX-like systems: Linux,
BSD families
Primary OS functions: - Process and thread management - Memory management
- File and storage management - I/O control and drivers - Security and access
control - Networking and communication - Resource isolation and virtualization
1
2. OS Architecture: Kernel and User Space
The kernel is the privileged core that interacts directly with hardware. User-
space processes run with limited privileges and use system calls to request kernel
services.
Modes of operation: - Kernel mode: full access to CPU instructions and memory
- User mode: restricted access for safety
Common kernel components: - Process scheduler - Memory manager - File
system manager - Network stack - Device drivers
Kernel interfaces: - System call interface (POSIX on UNIX-like systems) - /proc
and /sys pseudo-filesystems on Linux for introspection
3. Processes and Threads
Process: instance of a running program with its own address space, file descriptors,
and execution state.
Thread: lightweight execution unit within a process that shares the process
address space but has its own CPU registers and stack.
Types of threading models: - User-level threads: managed in user space (fast
context switch but blocking syscall blocks whole process) - Kernel-level threads:
scheduled by the kernel (better integration with blocking I/O) - Hybrid threading:
combination of both
Process creation and lifecycle (UNIX): fork(), execve(), wait(), exit() - fork()
creates a copy of the process; execve() replaces process image with a new pro-
gram. - POSIX threads (pthreads) provide APIs: pthread_create, pthread_join,
pthread_mutex.
Context switching: switching CPU from one process/thread to another involves
saving/restoring registers, updating kernel data structures, and switching page
tables if switching address spaces — relatively expensive compared to user-level
function calls.
4. CPU Scheduling
Scheduler goal: allocate CPU time fairly and efficiently while providing through-
put, low latency, and fairness.
Common scheduling algorithms: - First-Come First-Served (FCFS): simple, non-
preemptive, suffers convoy effect - Shortest Job First (SJF): optimal for average
wait time but requires knowledge of job lengths - Round Robin (RR): time-
slicing, preemptive, good for time-sharing systems - Priority Scheduling: tasks
prioritized; may cause starvation - Multilevel Queue and Multilevel Feedback
Queue (MLFQ): support different classes and dynamic priorities - Completely
2
Fair Scheduler (CFS): Linux’s default scheduler providing fairness by virtual
runtime
Real-time scheduling: - Rate Monotonic Scheduling (RMS) and Earliest Deadline
First (EDF) are used for real-time tasks with deadlines.
Metrics: - Throughput: jobs completed per time unit - Turnaround time:
completion time - arrival time - Waiting time: time spent in ready queue -
Response time: time from submission to first response
Example: simple Round Robin pseudocode
while true:
for each process in ready_queue:
run process for time_slice
if process not complete: append to ready_queue
5. Concurrency: Synchronization and Deadlock
Concurrency arises when multiple threads/processes access shared resources.
Synchronization primitives prevent race conditions.
Primitives: - Mutex (mutual exclusion): binary lock protecting critical sections -
Semaphore: integer-valued synchronization primitive; can be counting or binary -
Condition variable: used with mutexes to wait for conditions - Read-write locks:
allow multiple readers or one writer
Classic problems: producer-consumer, readers-writers, dining philosophers.
Example (pthread mutex in C):
pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_lock(&lock);
// critical section
pthread_mutex_unlock(&lock);
Deadlock: system state where processes wait indefinitely. Necessary conditions
(Coffman conditions): mutual exclusion, hold and wait, no preemption, circular
wait.
Deadlock handling strategies: - Prevention (remove one of Coffman’s conditions)
- Avoidance (Banker’s algorithm) - Detection and recovery (resource allocation
graph, preemption, killing processes)
Banker’s algorithm (brief): determine safe states by simulating allocation to
ensure system can meet maximum needs without deadlock.
3
6. Memory Management and Virtual Memory
Memory management abstracts physical memory into virtual addresses for
processes, enabling protection and convenient programming models.
Key concepts: - Virtual memory: process sees contiguous address space mapped
by page tables - Paging: fixed-size pages (commonly 4KB) map virtual pages
to physical frames - Segmentation: partition address space into variable-sized
segments (code, stack, heap) - Demand paging: load pages into memory on
access
Translation Lookaside Buffer (TLB): cache of recent virtual-to-physical transla-
tions.
Page replacement algorithms (when physical memory full): - Optimal (Belady’s):
replace page not used for longest future time (requires future knowledge) - FIFO:
replace oldest loaded page - LRU (Least Recently Used): replace page not used
for the longest past time - Clock (Second Chance): efficient approximation of
LRU using a circular buffer and reference bits
Thrashing: excessive page swapping causing severe performance loss. Reduce by
increasing memory, reducing degree of multiprogramming, or using working set
model.
Memory allocation strategies in kernel and user space: - malloc/free: user-space
allocator (ptmalloc, jemalloc, tcmalloc) - Slab allocator: kernel memory allocator
for frequently used small objects (object caching) - Buddy allocator: manages
memory in power-of-two sized blocks for efficient splitting/coalescing
Address space layout: code/text, data, heap, shared libraries, stack, memory-
mapped files.
7. File Systems and Storage
File system responsibilities: organize data on persistent storage, provide naming,
metadata, and access semantics.
Key components: - VFS (Virtual File System): abstraction layer supporting
multiple concrete filesystem implementations - Inode: filesystem metadata (own-
ership, permissions, timestamps, block pointers) - Directory structures and path
resolution
Types of file systems: - Ext family (ext2/3/4): commonly used on Linux - XFS,
Btrfs: advanced features (extent-based allocation, snapshots) - NTFS: Windows
filesystem with journaling and metadata - FAT32, exFAT: simple filesystems for
removable media
Journaling file systems write changes to a journal before committing to the
main filesystem to ensure consistency after crashes. Journaling modes: ordered,
writeback, full data journaling.
4
Block allocation strategies: contiguous, linked, indexed (inode), extents.
Storage devices: - HDD: rotational latency and seek time matter - SSD: no
seek; wear-leveling and TRIM commands relevant - NVMe: high-performance
interface for SSDs using PCIe
Mounting and permissions: mount points, fstab, mount options, file-level per-
missions (rwx), ACLs for finer control.
8. I/O and Device Management
I/O system handles communication between kernel and devices. Two main
models: - Programmed I/O: CPU polls device registers — CPU intensive -
Interrupt-driven I/O: device signals CPU on completion — efficient - DMA
(Direct Memory Access): device transfers data to/from memory without CPU
Device drivers implement device-specific control and integrate with kernel sub-
systems. Drivers operate at kernel level and must be carefully written to avoid
crashes.
Character vs block devices: character devices (e.g., serial ports) stream data;
block devices (e.g., disks) provide block-level access.
I/O scheduling: reorder requests to optimize throughput and reduce latency
(elevator algorithm, CFQ in Linux historically).
9. System Calls and Process Controls
System calls: gateway from user space to kernel services (e.g., open, read, write,
fork, execve, ioctl, mmap).
Example: open/read/write/close sequence in POSIX
Signals: asynchronous notifications (SIGINT, SIGTERM, SIGSEGV). Processes
can install signal handlers or use default actions. Signals can interrupt blocking
syscalls.
Interprocess communication (IPC): pipes, FIFOs, message queues, shared mem-
ory (shmget/shmat), sockets, signals.
Process groups, sessions, and job control: shell job control uses process groups
to manage foreground/background processes and terminal control.
10. Security and Access Control
Basic security mechanisms: - Discretionary Access Control (DAC): owners set
permissions (UNIX rwx) - Mandatory Access Control (MAC): system-enforced
policies (SELinux, AppArmor) - Capabilities: split privilege into fine-grained
capabilities rather than all-or-nothing root
Authentication and user identity: UIDs, GIDs, supplementary groups.
5
Isolation and least privilege: run services with restricted privileges, use chroot
or namespaces for containment.
Kernel hardening features: NX bit, ASLR (Address Space Layout Randomiza-
tion), stack canaries, seccomp filters.
Auditing and logging: syslog, audit frameworks (auditd), kernel logs (dmesg).
11. Virtualization and Containers
Virtualization enables multiple OS instances to share hardware.
Full virtualization (hypervisor): Guest OS runs unmodified under a hypervisor
(Type 1: bare-metal like Xen; Type 2: hosted like VirtualBox). Uses techniques:
trap-and-emulate, binary translation, hardware virtualization extensions (Intel
VT-x, AMD-V).
Para-virtualization: guest OS modified to cooperate with hypervisor (historical
approach for performance).
Containers: OS-level virtualization using kernel namespaces and cgroups: process
isolation via PID, mount, network namespaces; resource limits via cgroups.
Containers share the host kernel, are lightweight compared to VMs.
Examples: Docker (container runtime), Kubernetes (orchestration), LXC, rkt.
Live migration: moving running VMs with minimal downtime by pre-copying
memory and synchronizing dirty pages.
12. Kernel Design: Monolithic vs Microkernel
Monolithic kernels (Linux): most services (drivers, filesystems, network stack)
run in kernel space, offering performance and simplicity but larger trusted
computing base.
Microkernels (e.g., MINIX, QNX): minimal kernel performs IPC and scheduling;
services run in user space promoting modularity and reliability at the cost of
performance overhead due to IPC.
Hybrid kernels (Windows NT): blend microkernel ideas with monolithic perfor-
mance.
13. Real-Time and Embedded OS
Real-Time OS (RTOS): provide deterministic timing guarantees. Two classes:
- Hard real-time: missing a deadline can lead to catastrophic failure - Soft
real-time: occasional misses degrade quality but not catastrophic
RTOS features: priority inheritance, minimal latency, predictable scheduling
(RMS, EDF), small footprint.
Embedded OS choices: FreeRTOS, Zephyr, embedded Linux for larger devices.
6
14. Performance, Monitoring, and Debugging
Monitoring tools (Linux): top/htop, vmstat, iostat, sar, free, ps, netstat/ss,
iproute2, perf, dtrace (some systems), ftrace.
Debugging tools: strace (system call tracing), ltrace (library calls), gdb (debug-
ger), valgrind (memory debugging, memcheck), perf (profiling), eBPF tools for
dynamic tracing.
Performance tuning: identify bottleneck (CPU, memory, I/O, network), tune
kernel parameters (/proc/sys/vm for VM settings), use correct scheduler and
IO schedulers, tune filesystem options, network stack tuning (tcp buffer sizes),
employ caching and concurrency.
15. Hands-on Labs and Exercises
1. Process and threads
• Lab: write a C program that forks a child, communicates via pipe, and
synchronizes using waitpid. Use strace to inspect system calls.
2. Pthreads and synchronization
• Lab: implement producer-consumer using pthreads, mutexes, and condition
variables. Measure throughput with varying producer/consumer counts.
3. Implement a simple scheduler simulator
• Simulate RR, SJF, FCFS, and MLFQ schedulers on synthetic job arrival
patterns; compute average turnaround and waiting times.
4. Virtual memory simulation
• Implement a page replacement simulator for FIFO, LRU, and Clock al-
gorithms; feed realistic memory access traces and compare page fault
rates.
5. Filesystems and storage
• Mount a filesystem image (loopback) and explore inode structures using
debugfs (on Linux). Create and corrupt a journal and observe recovery.
6. Container basics
• Use Docker to run an Ubuntu container, inspect namespaces and cgroups
(lsns, cat /proc/[pid]/ns/*), and limit CPU and memory with Docker run
flags.
7. Tracing and profiling
• Use strace to trace open/read/write calls of a small program. Use perf to
identify hotspots in a computational workload.
8. Build a simple device driver (kernel module)
7
• For learning only: write a minimal Linux kernel module that registers a
character device and supports read/write. Load with insmod and remove
with rmmod.
Safety note: kernel programming can crash the system; use a VM.
16. Glossary and Further Reading
Glossary: process, thread, context switch, TLB, inode, DMA, VM, page fault,
slab allocator, system call, kernel module, namespace, cgroup, hypervisor.
Recommended books and resources: - “Operating System Concepts” — Silber-
schatz, Galvin, Gagne - “Modern Operating Systems” — Andrew S. Tanenbaum
- “Linux Kernel Development” — Robert Love - Linux kernel source and Docu-
mentation/ directory - [Link] for kernel development articles
Cheat sheet — useful commands (Linux): - ps aux | grep processname - top /
htop - strace -f -e trace=open,read,write ./myprog - ltrace ./myprog - vmstat 1
- iostat -x 1 - free -m - cat /proc/cpuinfo - lsns # list namespaces - systemctl
status sshd
Exercises recap and suggested schedule: - Week 1: processes, threads, basic
system calls, fork/exec - Week 2: synchronization, deadlocks, scheduler simulation
- Week 3: virtual memory, page replacement labs - Week 4: filesystems, device
drivers basics - Week 5: containers and virtualization labs