Security in Computing
Sixth Edition
Charles P. Pfleeger
Shari Lawrence Pfleeger
Lizzie Coles-Kemp
Chapter 5: Operating Systems
In This Chapter
• Operating system structure and design
• Object protection: virtualization, sharing
• Memory protection: registers, paging,
segmentation
• Design qualities: modularity, layering,
kernelization
• Trusted systems: TCB, reference monitor,
trusted path, object reuse, evaluation criteria
• Rootkits: power, design
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 2
Background: Operating System
Structure
•OS as Executive Controller: Manages resources and user
access for devices.
•Broad Application: Found in devices beyond computers,
like thermostats, pacemakers, cars, smartphones, and
network systems.
•Complexity Varies by Device: Control levels and
interaction depend on the device.
•Security Role: Critical for managing resource access and
determining user permissions.
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 3
Security in Operating Systems
FIGURE 5-1 Operating System Functions
illustrates how an operating system interacts with
users, provides services, and allocates resources
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 4
Operating Systems Functions
• Enforced sharing
• Interprocess communication and synchronization
• Protection of critical operating system data
• Guaranteed fair service
• Interface to hardware
• User authentication
• Memory protection
• File and I/O device access control
• Allocation and access control to general objects
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 5
Protected Objects in
Multiprogramming
• Memory
• Sharable I/O devices, such as disks
• Serially reusable I/O devices, such as
printers and tape drives
• Sharable programs and subprocedures
• Networks
• Sharable data
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 6
Layered Operating Systems (cont'd)
the diagram shows a layered
approach to security, with the
most critical and trusted functions
in the center and less trusted
layers protecting it from the
outside. This model ensures that
only trusted components have full
access to sensitive data.
FIGURE 5-3 Authentication Functions Spanning Layers in an Operating System
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 7
Operating System Tools for Security
• Audit log: Records user actions on objects to trace access and potential
security breaches.
• Virtualization: Creates virtual resources to limit user access to specific
system parts.
• Virtual machine: A simulated system providing isolated resources for
different user sets.
• Hypervisor: Software that manages virtual machines, allowing
secure, controlled access to resources.
• Sandbox: A restricted environment where programs can run without
affecting external resources.
• Honeypot: A decoy system designed to attract attackers for
monitoring and analysis.
• Separation and sharing: Techniques to isolate user processes and
resources through physical, temporal, logical, or cryptographic methods.
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 8
Separation in Operating System
• Physical separation: Different processes use
different physical objects
• Temporal separation: Processes having different
security requirements are executed at different times
• Logical separation: Users operate under the illusion
that no other processes exist, as when an operating
system constrains a program’s accesses so that the
program cannot access objects outside its permitted
domain
• Cryptographic separation: Processes conceal their
data and computations in such a way that they are
unintelligible to outside processes
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 9
Hardware Protection of Memory
• Fence: A boundary that confines users to a specific memory area, protecting the
operating system from user programs.
• Base/bounds registers: A pair of registers setting lower and upper limits for
memory access, confining each user’s address space to a specific area.
• Tagged architecture: Each memory word has access rights tags, enabling
controlled access for different data types and security levels.
• Virtual memory: Simulates more memory than physically available, allowing
programs to be divided and swapped between main memory and storage as
needed.
• Segmentation: Divides a program into logical segments (e.g., code, data) with
unique names and access rights, allowing controlled access to specific memory
sections.
• Paging: Divides memory into fixed-size pages, improving memory management
efficiency without logical grouping.
• Combined paging with segmentation: Combines segmentation’s logical structure
with paging’s efficiency, allowing segments to be split into pages for better
memory management and protection.
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 10
Security in the Design of Operating Systems
• Simplicity of design: A simple, modular design minimizes complexity, reducing
security vulnerabilities and easing maintenance.
• Layered design
• Conventional (nonhierarchical) :Layers interact directly without a strict
hierarchy, which can increase the risk of vulnerabilities spreading.
• Hierarchical: Layers are organized hierarchically, with the most sensitive
operations in inner layers, limiting the spread of issues.
• Kernelized design: The kernel is the core of the OS, performing essential low-
level functions like process management and interprocess communication.
• Reference monitor: A security component that enforces access controls, ensuring
that subjects only access objects according to policy. It must be tamperproof,
unbypassable, and analyzable.
• Correctness and completeness: Security functionality should be correctly
implemented in all areas of the OS, with comprehensive checks to meet all
security requirements.
• Secure design principles: Security must be integrated from the start of design and
development, rather than added as an afterthought.
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 11
Reference Monitor
[Link] (S): These are entities (e.g., users or
processes) that request access to certain resources or data
within the system. In the diagram, several subjects attempt
to interact with the system.
[Link] (O): These represent the resources or data that
subjects want to access. The diagram shows multiple
objects that subjects might attempt to reach.
[Link] Monitor: Positioned as a gatekeeper, the
reference monitor stands between subjects and objects. It
controls and mediates all access requests to ensure they
comply with security policies. Every access attempt by a
subject must pass through the reference monitor.
[Link] Control Enforcement: The reference monitor
checks each access request based on established policies,
allowing or denying access depending on whether the
subject is authorized for the specific object.
[Link] and Protection: The reference monitor is
shown as an unbreakable wall, ensuring subjects cannot
bypass it to directly access objects. This guarantees that all
access is controlled and monitored, maintaining system
security.
FIGURE 5-15 Reference Monitor:illustrates the role of a reference
monitor in controlling access between subjects (S) and objects (O)
within an operating system or trusted software environment.
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems
Secure Design Principles
• Least privilege :Users and programs are given the minimum level of access
necessary to perform their tasks, limiting potential security risks.
• Economy of mechanism: Security mechanisms should be as simple and
straightforward as possible, reducing the chance for errors or vulnerabilities.
• Open design: The security design should not rely on secrecy; the system should
remain secure even if its design is publicly known.
• Complete mediation: Every access to every resource must be checked for
authorization, ensuring that no access bypasses security checks.
• Permission-based: The system should default to denying access, requiring explicit
permission for any access to resources.
• Separation of privilege: Multiple conditions should be required for granting
permissions, preventing unauthorized access if a single control fails.
• Least common mechanism: Resources should be shared as little as possible
among users, reducing the likelihood of shared vulnerabilities.
• Ease of use: Security mechanisms should be user-friendly, ensuring users don’t
bypass them due to complexity.
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 13
Trusted Systems
• Definition: Trusted systems are designed to faithfully
perform security functions and enforce policies.
• Purpose: Counteract malicious software by ensuring the
system’s behavior meets security expectations.
• Key Characteristics:
• Functional correctness
• Integrity enforcement
• Limited privilege
• Confidence level
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 14
Key Terms in Trusted Systems
• Trust vs. Security: Trust implies confidence based on
evidence; security is a binary state.
• Trusted Computing Base (TCB): Core components
required to enforce security policies.
• Trusted Path: A secure channel between the user and
OS, ensuring unforgeable communication.
• Trusted Process and Software: Critical code and
processes that reliably enforce security policies.
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 15
Trusted Computing Base (TCB) Components and
Functions
• Components:
• Hardware (processors, memory, I/O)
• Security-related files (access control, authentication data)
• Protected memory for integrity
• Functions:
• Process Activation: Manages secure process switches.
• Domain Switching: Controls access across different domains.
• Memory Protection: Ensures secure memory access.
• I/O Operations: Monitors data flow across domains.
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 16
Design and Structure of TCB
•Separation: TCB code is protected and operates
independently from non-TCB elements.
•Benefits:
•Enhanced security by isolating TCB code
•Flexibility to modify non-TCB code without impacting
security
•Implementation Considerations: TCB forms a security
kernel that mediates all access to protected resources.
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 17
Trusted Path and Object Reuse
•Trusted Path: Ensures user-to-OS communication is
secure and tamper-proof (e.g., CTRL+ALT+DELETE for
login).
•Object Reuse and Data Sanitization:
•Cleans memory and storage before reassignment to
prevent data leakage.
•Prevents attacks by malicious users accessing residual
data from previous processes.
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 18
Audit Requirements and Security Logging
• Audit Logs :security-relevant changes like
software installation or configuration
modifications.
• Tamper-Proof: Audit logs are protected from
unauthorized access and alteration.
• Importance: Essential for accountability and
identifying security breaches.
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 19
Trusted Systems Evaluation – Orange Book
and Common Criteria
•Orange Book: Early U.S. Department of Defense standard for
secure computing, categorizing systems by security levels (C1 to
A1).
•Common Criteria:
•International standard for security evaluation, with flexibility in
features and assurance levels.
•Assurance levels from EAL1 (basic) to EAL7 (high), adaptable to
evolving technology needs.
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 20
Rootkits
• Root (in Unix): Identity of the most privileged
user, capable of full system control.
• Rootkit: Malicious software that grants
unauthorized root-level access to an attacker.
• characteristics:
• Evade detection
• Operate unchecked
• Exploit permissions
• Usurp control
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 21
Common Rootkit Techniques
•Stealth Operations: Modifies system processes to hide files
and activities.
•Interception: Alters system API calls to skip over malicious
files (e.g., hiding mal_code.exe).
•Early Execution: Loads during boot to avoid detection by
security software.
•Kernel Integration: Embeds within the OS kernel to gain
unrestricted access.
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 22
Examples of Rootkit Attacks
• Mobile Rootkits: Eavesdrops, tracks
location, or drains battery without user
knowledge.
• Sony XCP: Installed on PCs without consent
to prevent copying of Sony CDs;
inadvertently opened security vulnerabilities.
• TDSS Family: Evolves from TDL-1 to TDL-4,
avoiding detection through encryption,
custom file systems, and MBR infection.
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 23
Rootkit Countermeasures
• Rootkit Revealers: Specialized tools that detect hidden
files by comparing actual vs. reported directory contents.
• Secure Boot and Kernel Signing: Ensures only verified
code runs during OS startup.
• Audit and Monitoring: Regular system checks and logs
to identify anomalies in system behavior.
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems 24
Example: Rootkit Filtering File
FIGURE 5-22 Rootkit Filtering File Description
Result: illustrates how a rootkit intercepts and
filters function calls to hide its presence in the
operating system.
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems
Summary
• Survey of operating systems for important
security concept development
• Operating systems as place of detailed
analysis of access control
• Operating system design strategies
• Strength of the operating system due to tight
integration with hardware, simple design, and
security focus
• Advantage of self-contained operating system
on a distinct platform
© 2024 by Pearson Education, Inc. Chapter 5: Operating Systems