Operating Systems
1. Introduction to Operating Systems (OS)
An Operating System (OS) is the core authority of a computer. It acts as the brain, command center,
and resource manager for all programs and hardware.
Why is the OS important?
• It controls everything: CPU, memory, files, processes.
• It decides who gets what resource and when.
• For hackers, compromising the OS = full control (persistence, data access, privilege
escalation).
Key mindset:
If the OS is compromised, the entire system is compromised.
2. Core Components of an OS
Every OS consists of three fundamental components:
2.1 Kernel – The Heart of the OS
• Runs in Kernel Mode / Ring 0 (highest privilege).
• Manages the most important resources:
o CPU scheduling
o Physical memory
o Device drivers
o System calls
• Cannot be directly accessed by regular applications.
Hacking Relevance
• Target for rootkits (stealth malware running inside the kernel).
• A hacked kernel becomes invisible and unstoppable.
2.2 Shell – The Interface to the Kernel
• Acts as the bridge between the user/application and the kernel.
• Types:
o Command-line shell: Bash (Linux), PowerShell (Windows)
o Graphical shell: Desktop interface (Windows GUI, GNOME, KDE)
Hacking Relevance
• Goal of attackers: Obtain a shell (reverse shell / web shell).
• A shell = remote command execution and full operational access.
2.3 File System
Organizes data into files and directories.
• Windows → NTFS
• Linux → ext4
Hacking Relevance
• Determines how data is stored, accessed, and protected.
• File system weaknesses = privilege escalation, data extraction.
3. Processes, Threads & Memory Management
Understanding how programs run is essential for hacking.
3.1 Processes
A process is:
• A running program with its own memory space.
• Isolated from other processes.
Hacker View
• Inject malicious code into a trusted process (process hollowing, code injection).
• Abuse running processes to hide activity.
3.2 Threads
• Lightweight units of execution inside a process.
• One process can have many threads running simultaneously.
3.3 Virtual Memory
• Every process gets its own “fake large memory” illusion.
• OS handles mapping to physical RAM.
Why this matters?
• Attackers target specific memory regions of specific processes.
• Isolation is good, but flawed software can break it.
4. Memory Regions: Stack & Heap
These two memory regions are major exploitation targets.
4.1 Stack
Used for:
• Function calls
• Return addresses
• Local variables
Highly structured.
Major vulnerability:
➡️ Stack-Based Buffer Overflow
When too much data is written to a buffer, it spills over and overwrites return addresses, allowing
attackers to redirect execution flow.
Result: Remote Code Execution (RCE).
4.2 Heap
Used for:
• Dynamic memory allocation
• Data whose size is unknown beforehand
Less structured than stack.
Major attack:
➡️ Heap Spraying
Attackers fill the heap with malicious data to increase exploit success.
5. System Calls – Gateway to Privileged Actions
Applications can’t perform dangerous operations directly. They must ask the kernel using system
calls.
Examples:
• Write to a protected file
• Access hardware
• Open a network socket
Hacking relevance
• Exploiting system call handlers = kernel-level access.
• Malware often hides by bypassing system call monitoring.
6. Windows vs Linux Security Models
Understanding the differences helps attackers choose strategies.
6.1 Privilege Levels
Aspect Windows Linux
Highest Privilege Administrator / SYSTEM Root
Default User Standard User Standard User
Privilege Escalation UAC prompt sudo (password required)
System Files Often accessible Strict, root-owned
Interpretation
• Windows attackers target GUI users → phishing, macro malware, credential theft.
• Linux attackers target misconfigurations → writable files, weak permissions, vulnerable
services.
6.2 Security Philosophy
Feature Windows Approach Linux Approach
Design Goal Ease of use Multi-user security
Firewall On by default Off unless configured
Access Control ACLs (complex) rwx permissions (simple)
6.3 Attack Surface Differences
Attacking Windows
• Phishing and social engineering
• Malicious documents (macros)
• Weak credentials
• Active Directory exploitation
Attacking Linux
• Misconfigured services (SSH, Apache)
• Weak file permissions
• Writable scripts executed by privileges
• Cron jobs and SUID binaries
7. File System Permissions (Linux rwx Model)
When running ls -l, you see:
-rwxr-xr--
Breakdown:
Position Meaning
1 File type (- or d)
2–4 Owner permissions (rwx)
5–7 Group permissions (r-x)
8–10 Others permissions (r--)
Critical Security Insight
If “others” has write (w):
• Anyone can modify the file
• Possible privilege escalation
• Dangerous in system directories
Attackers actively search for files with:
--w
--wx
-rwxrwxrwx
8. Process Scheduling & CPU Management
The OS decides:
• Which process runs
• For how long
• Priority of tasks
Hacking Relevance
• Denial-of-Service (DoS) by exhausting CPU
• Starving critical processes
• Locking resources
9. I/O Management
Controls:
• Keyboard
• Mouse
• Disk operations
• Network devices
Attack Implications
• Keylogging
• Disk tampering
• Network packet manipulation
10. Summary – Why OS Knowledge is Critical for Hackers
✔ The kernel is the ultimate target
✔ Memory flaws → Remote Code Execution
✔ Permissions determine privilege escalation paths
✔ Windows & Linux require different attack strategies
✔ Misconfigurations are goldmines
✔ Processes, system calls, and memory regions are core attack surfaces
Key Takeaway for Students
To become a skilled ethical hacker, you must understand how the OS works at its core—from
memory to processes to permissions. The OS is always the final battlefield.