ECCS-4221: Operating Systems
Credit :4-3-3-0
Instructor : Lt .Measho Berhe
Fall: 2025
ECCS-4221: Operating Systems
Chapters 6: Security and Protection
• Introduction to Security and Protection
• The Security Problem
• Program Threats
• System and Network Threats
• Cryptography as a Security Tool
• User Authentication
• Implementing Security Defenses
• Goals of Protection
Learning Outline
• Principles of Protection
• Domains of Protection
• Access Matrix Model
• Implementation of Access Matrix
• Access Control Mechanisms
• Revocation of Access Rights
• Capability-Based Protection
• Language-Based Protection
3
• After completing this chapter, students will be able to:
• Explain the basic security problem in computer
systems.
• Identify program, system, and network security threats.
• Describe cryptography and its role in system security.
• Explain user authentication methods.
• Discuss techniques for implementing security
Learning
defenses.
Objectives
• State the goals and principles of protection.
• Explain protection domains and the access matrix
model.
• Describe access control mechanisms and access
revocation.
• Compare ACL-based and capability-based protection
• Explain language-based protection mechanisms
4
Security
The Security Problem Security Violation Categories
• Security : is the measure of confidence that
the integrity of a system and its data will be 1. Breach of confidentiality
preserved, ensuring protection against
• unauthorized access, • Unauthorized reading of data
• modification, 2. Breach of integrity
• destruction, or
• disruption. • Unauthorized modification of data
• Secure system : if resources used and 3. Breach of availability
accessed as intended under all circumstances
• Unachievable • Unauthorized destruction of data
• Intruders (crackers) attempt to breach 4. Theft of service
security • Unauthorized use of resources
• Threat is potential security violation 5. Denial of service (DOS)
• Attack is attempt to breach security • Prevention of legitimate use
• Attack can be accidental or malicious • CIA in security stands for Confidentiality, Integrity, and
• Easier to protect against accidental than Availability, which are the three primary goals of information
malicious misuse security.
5
Cont.
Security Violation Methods Security Measure Levels
• Masquerading (breach authentication) • Impossible to have absolute security; security aims to make attacks costly
,difficult and time-consuming so attackers are deterred.
• Pretending to be an authorized user to escalate
privileges • Security must occur at four levels to be effective:
• Physical
• Replay attack • Data centers, servers, connected terminals
• when an attacker captures a valid message and • Application
retransmits it later to gain unauthorized access • Benign or malicious apps can cause security problems
without modifying the message. • Operating System
• Protection mechanisms, debugging
• Man-in-the-middle attack
• Network
• when an attacker secretly intercepts and • Intercepted communications, interruption, DOS
possibly modifies communication between two
legitimate parties. • Security is as weak as the weakest link in the chain
• Humans a risk too via phishing and social-engineering attacks
• Session hijacking
• Intercept an already-established session to
bypass authentication
• Privilege escalation
• Common attack type with access beyond what a
user or resource is supposed to have
6
Program Threats
• Many variations, many names
• Malware - Software designed to exploit, disrupt, damage, or gain
unauthorized access to computer systems.
• Trojan Horse – A program that appears legitimate but secretly performs
unauthorized or malicious actions.
• Spyware Software that is often installed with legitimate programs and secretly collects
user information or monitors activity.
• Trap Door – A hidden mechanism (e.g., a specific user ID or password) that bypasses
normal security controls
• It may be intentionally embedded in software, including a compiler.
• Ransomware – Malware that encrypts data or locks systems and demands
payment to restore access.
• Viruses
• Code fragment embedded in legitimate program
• Self-replicating, designed to infect other programs or systems
• Very specific to CPU architecture, operating system, applications
• Usually borne via email or as a macro
• Visual Basic Macro to reformat hard drive
7
System and Network Threats
• General Issues • Port Scanning
• Many systems are open by default rather than • Automated probing of ports on one or more IP
addresses
secure
• Identifies:
• Reducing attack surface improves security but
• Active services and protocols
increases admin complexity
• OS and software versions
• Network threats are harder to detect and prevent
• Common tools: nmap, nessus
• Protection mechanisms are generally weaker over
• Often launched from zombie systems to hide origin
networks
• Shared secrets are harder to maintain at scale • Denial of Service (DoS)
• Overloads a target system to prevent normal operation
• No physical boundaries once connected to the
Internet • Distributed DoS (DDoS)
• Identifying attacker location is difficult • Attacks launched from multiple systems
simultaneously
• IP address may be the only visible information
• Exploits limits such as:
• Worms • TCP connection handling (e.g., SYN flood)
• Standalone, self-propagating programs • Can be:
• Spread automatically using network • Accidental (poorly written code)
vulnerabilities • Intentional (extortion, disruption)
8
System and Network Threats (Cont.)
9
Cryptography as a Security Tool
• Broadest security tool available
• Internal to a given computer, source and destination of messages can be known and
protected
• OS creates, manages, protects process IDs, communication ports
• Source and destination of messages on network cannot be trusted without cryptography
• Local network – IP address?
• Consider unauthorized host added
• WAN / Internet – how to establish authenticity
• Not via IP address
• Cryptography: Means to constrain potential senders (sources) and / or
receivers (destinations) of messages
• Based on secrets (keys)
• Enables
• Confirmation of source
• Receipt only by certain destination
• Trust relationship between sender and receiver
10
Encryption
• Constrains the set of possible receivers of a message
• Encryption algorithm consists of
• Set K of keys
• Set M of Messages
• Set C of ciphertexts (encrypted messages)
• A function E : K → (M→C). That is, for each k K, Ek is a function for generating ciphertexts
from messages
• Both E and Ek for any k should be efficiently computable functions
• A function D : K → (C → M). That is, for each k K, Dk is a function for generating messages
from ciphertexts
• Both D and Dk for any k should be efficiently computable functions
• An encryption algorithm must provide this essential property: Given a
ciphertext c C, a computer can compute m such that Ek(m) = c only if it
possesses k
• Thus, a computer holding k can decrypt ciphertexts to the plaintexts used to produce
them, but a computer not holding k cannot decrypt ciphertexts
• Since ciphertexts are generally exposed (for example, sent on the network), it is
important that it be infeasible to derive k from the ciphertexts
11
Encryption(Cont.)
Symmetric Encryption
• Same key used to encrypt and decrypt Secure Communication over Insecure Medium
• Therefore k must be kept secret
• DES was most commonly used symmetric block-
encryption algorithm (created by US Govt)
• Encrypts a block of data at a time
• Keys too short so now considered insecure
• Triple-DES considered more secure
• Algorithm used 3 times using 2 or 3 keys
• For example
• 2001 NIST adopted new block cipher - Advanced
Encryption Standard (AES)
• Keys of 128, 192, or 256 bits, works on 128 bit
blocks
• RC4 is most common symmetric stream cipher,
but known to have vulnerabilities
• Encrypts/decrypts a stream of bytes (i.e.,
wireless transmission)
• Key is a input to pseudo-random-bit generator
• Generates an infinite keystream
12
Encryption(Cont.)
Asymmetric Encryption
• Public-key encryption based on each user having two keys:
• public key – published key used to encrypt data
• private key – key known only to individual user used to decrypt data
• Must be an encryption scheme that can be made public without making it easy
to figure out the decryption scheme
• Most common is RSA block cipher
• Efficient algorithm for testing whether or not a number is prime
• No efficient algorithm is know for finding the prime factors of a number
• Formally, it is computationally infeasible to derive kd,N from ke,N, and so ke need
not be kept secret and can be widely disseminated
• ke is the public key
• kd is the private key
• N is the product of two large, randomly chosen prime numbers p and q (for example, p and q
are 512 bits each)
• Encryption algorithm is Eke,N(m) = mke mod N, where ke satisfies kekd mod (p−1)(q −1) = 1
• The decryption algorithm is then Dkd,N(c) = ckd mod N
13
Asymmetric Encryption Example(Cont. )
• For example. make p = 7and q = 13
• We then calculate N = 7∗13 = 91 and (p−1)(q−1) = 72
• We next select ke relatively prime to 72 and< 72,
yielding 5 (gcd(ke ,72)=1)
• Finally, we calculate kd such that kekd mod 72 = 1,
yielding 29
• We how have our keys
• Public key, ke,N = 5, 91
• Private key, kd,N = 29, 91
• Encrypting the message 69 with the public key results
in the cyphertext 62
• Cyphertext can be decoded with the private key
• Public key can be distributed in cleartext to anyone
who wants to communicate with holder of public key
14
Authentication
• Constraining set of potential senders of a message
• Complementary to encryption
• Also can prove message unmodified
• Algorithm components
• A set K of keys
• A set M of messages
• A set A of authenticators
• A function S : K → (M→ A)
• That is, for each k K, Sk is a function for generating authenticators from messages
• Both S and Sk for any k should be efficiently computable functions
• A function V : K → (M × A→ {true, false}). That is, for each k K, Vk is a function for verifying authenticators on
messages
• Both V and Vk for any k should be efficiently computable functions
• For a message m, a computer can generate an authenticator a A such that Vk(m, a) = true only if it possesses k
• Thus, computer holding k can generate authenticators on messages so that any other computer possessing k can verify
them
• Computer not holding k cannot generate authenticators on messages that can be verified using Vk
• Since authenticators are generally exposed (for example, they are sent on the network with the messages themselves), it
must not be feasible to derive k from the authenticators
• Practically, if Vk(m,a) = true then we know m has not been modified and that send of message has k
• If we share k with only one entity, know where the message originated
15
Cont.
Authentication – Hash Functions
• Basis of authentication
• Creates small, fixed-size block of data message digest (hash value)
from m
• Hash Function H must be collision resistant on m
• Must be infeasible to find an m’ ≠ m such that H(m) = H(m’)
• If H(m) = H(m’), then m = m’
• The message has not been modified
• Common message-digest functions include MD5, which produces a
128-bit hash, and SHA-1, which outputs a 160-bit hash
• Not useful as authenticators
• For example H(m) can be sent with a message
• But if H is known someone could modify m to m’ and recompute H(m’) and modification not detected
• So must authenticate H(m)
16
Cont.
Authentication - MAC
• Symmetric encryption used in message-authentication code (MAC)
authentication algorithm
• Cryptographic checksum generated from message using secret key
• Can securely authenticate short values
• If used to authenticate H(m) for an H that is collision resistant, then
obtain a way to securely authenticate long message by hashing them
first
• Note that k is needed to compute both Sk and Vk, so anyone able to
compute one can compute the other
17
Cont.
Authentication – Digital Signature
• Based on asymmetric keys and digital signature algorithm
• Authenticators produced are digital signatures
• Very useful – anyone can verify authenticity of a message
• In a digital-signature algorithm, computationally infeasible to derive ks from kv
• V is a one-way function
• Thus, kv is the public key and ks is the private key
• Consider the RSA digital-signature algorithm
• Similar to the RSA encryption algorithm, but the key use is reversed
• Digital signature of message Sks (m) = H(m)ks mod N
• The key ks again is a pair (d, N), where N is the product of two large, randomly chosen prime numbers p
and q
• Verification algorithm is Vkv(m, a) (akv mod N = H(m))
• Where kv satisfies kvks mod (p − 1)(q − 1) = 1
• Why authentication if a subset of encryption?
• Fewer computations (except for RSA digital signatures)
• Authenticator usually shorter than message
• Sometimes want authentication but not confidentiality
• Signed patches et al
• Can be basis for non-repudiation
18
Key Distribution
• Delivery of symmetric key is huge challenge
• Sometimes done out-of-band
• Asymmetric keys can proliferate – stored on key ring
• Even asymmetric key distribution needs care – man-in-the-
middle attack
19
Digital Certificates
• Proof of who or what owns a public key
• Public key digitally signed a trusted party
• Trusted party receives proof of identification from entity and certifies
that public key belongs to entity
• Certificate authority are trusted party – their public keys included
with web browser distributions
• They vouch for other authorities via digitally signing their keys, and so on
20
User Authentication
• Crucial to identify user correctly, as protection systems depend on user ID
• User identity most often established through passwords, can be
considered a special case of either keys or capabilities
• Passwords must be kept secret
• Frequent change of passwords
• History to avoid repeats
• Use of “non-guessable” passwords
• Log all invalid access attempts (but not the passwords themselves)
• Unauthorized transfer
• Passwords may also either be encrypted or allowed to be used only once
• Does encrypting passwords solve the exposure problem?
• Might solve sniffing
• Consider shoulder surfing
• Consider Trojan horse keystroke logger
• How are passwords stored at authenticating site?
21
Passwords
• Store hashed passwords (not encrypted)
• Hash function should be easy to compute but hard to invert
• Only the hash is stored; passwords are never decrypted
• Add a salt to prevent identical passwords from producing the same hash
• Password files should be protected
• Example: Unix stores password hashes in /etc/shadow (root-only access)
• One-time passwords
• Use a function based on a seed to compute a password, both user and computer
• Hardware device / calculator / key fob to generate the password
• Changes very frequently
• Biometrics
• Some physical attribute (fingerprint, hand scan)
• Multi-factor authentication
• Need two or more factors for authentication
• i.e., USB “dongle”, biometric measure, and password 22
Implementing Security Defenses
• Defense in depth is most common security theory – multiple layers of security
• Security policy describes what is being secured
• Vulnerability assessment compares real state of system / network compared to security policy
• Intrusion detection endeavors to detect attempted or successful intrusions
• Signature-based detection spots known bad patterns
• Anomaly detection spots differences from normal behavior
• Can detect zero-day attacks
• False-positives and false-negatives a problem
• Virus protection
• Searching all programs or programs at execution for known virus patterns
• Or run in sandbox so can’t damage system
• Auditing, accounting, and logging of all or specific system or network activities
• Practice safe computing – avoid sources of infection, download from only “good” sites, etc
23
Firewalling to Protect Systems and Networks
• A network firewall is placed between trusted and untrusted hosts
• The firewall limits network access between these two security domains
• Can be tunneled or spoofed
• Tunneling allows disallowed protocol to travel within allowed protocol (i.e., telnet inside of HTTP)
• Firewall rules typically based on host name or IP address which can be spoofed
• Personal firewall is software layer on given host
• Can monitor / limit traffic to and from the host
• Application proxy firewall understands application protocol and can control them (i.e., SMTP)
• System-call firewall monitors all important system calls and apply rules to them (i.e., this program can
execute that system call)
Network Security Through Domain Separation Via Firewall
24
Protection
Goals of Protection Principles of Protection
• Protection : is the set of mechanisms that • Guiding principle – principle of least privilege
control the access of processes and • Programs, users and systems should be
users to the resources defined by a given just enough privileges to perform their
computer system tasks
• In one protection model, computer • Properly set permissions can limit damage
if entity has a bug, gets abused
consists of a collection of objects(
hardware or software) • Can be static (during life of system, during
life of process)
• Each object has a unique name and can be • Or dynamic (changed by process as needed)
accessed through a well-defined set of – domain switching, privilege escalation
operations • Compartmentalization a derivative
• Protection problem - ensure that each concept regarding access to data
object is accessed correctly and only by • Process of protecting each individual
those processes that are allowed to do so system component through the use of
specific permissions and access
restrictions
25
Principles of Protection (Cont.)
• Must consider “grain” aspect
• Rough-grained privilege management easier, simpler, but least privilege now done
in large chunks
• For example, traditional Unix processes either have abilities of the associated
user, or of root
• Fine-grained management more complex, more overhead, but more protective
• File ACL lists, RBAC
• Domain can be user, process, procedure
• Audit trail – recording all protection-orientated activities, important to
understanding what happened, why, and catching things that shouldn’t
• No single principle is a panacea for security vulnerabilities – need
defense in depth
26
Protection Rings
Protection Rings Protection Rings (MULTICS)
• Components ordered by amount of • Let Di and Dj be any two domain
privilege and protected from each other
• For example, the kernel is in one ring and
rings
user applications in another • If j < I Di Dj
• This privilege separation requires hardware
support
• Gates used to transfer between levels, for
example the syscall Intel instruction
• Also traps and interrupts
• Hypervisors introduced the need for yet
another ring
• ARMv7 processors added TrustZone(TZ) ring
to protect crypto functions with access via
new Secure Monitor Call (SMC) instruction
• Protecting NFC secure element and
crypto keys from even the kernel
27
Domain of Protection
• Rings of protection separate functions into domains and order them
hierarchically
• Computer can be treated as processes and objects
• Hardware objects: such as CPU, RAM, Disk drives, Network interface cards, Printers,
and I/O devices
• software objects: such as files, programs, and semaphores
• Process for example should only have access to objects it currently requires
to complete its task – the need-to-know principle
• Implementation can be via process operating in a protection domain
• Specifies resources process may access
• Each domain specifies set of objects and types of operations on them
• Ability to execute an operation on an object is an access right
• <object-name, rights-set>
• Domains may share access rights
• Associations can be static or dynamic
• If dynamic, processes can domain switch
28
Domain Structure
• Access-right = <object-name, rights-set>
where rights-set is a subset of all valid operations that can be
performed on the object
• Domain = set of access-rights
29
Access Matrix
• View protection as a matrix (access matrix)
• Rows represent domains
• Columns represent objects
• Access(i, j) is the set of operations that a process executing in
Domaini can invoke on Objectj
30
Use of Access Matrix
• If a process in Domain Di tries to do “op” on object Oj, then “op” must be in the access matrix
• User who creates object can define access column for that object
• Can be expanded to dynamic protection
• Operations to add, delete access rights
• Special access rights:
• owner of Oi
• copy op from Oi to Oj (denoted by “*”)
• control – Di can modify Dj access rights
• transfer – switch from domain Di to Dj
• Copy and Owner applicable to an object
• Control applicable to domain object
• Access matrix design separates mechanism from policy
• Mechanism
• Operating system provides access-matrix + rules
• If ensures that the matrix is only manipulated by authorized agents and that rules are strictly enforced
• Policy
• User dictates policy
• Who can access what object and in what mode
• But doesn’t solve the general confinement problem
31
Access Matrix of with Domains as
Objects Access Matrix with Copy Rights
32
Cont.
Access Matrix With Owner Rights Modified Access Matrix of Figure
33
Revocation of Access Rights
• Revocation of access rights is the process of removing previously granted permissions from a
user, process, or domain.
• Why Revocation Is Needed Revocation in Access Matrix Implementations
• User leaves the system • Access List – Delete access rights from access list
• Role or responsibility changes • Simple – search access list and remove entry
• Immediate, general or selective, total or partial, permanent or
• Temporary access expires temporary
• Compromised account • Capability List – Scheme required to locate capability in the
system before capability can be revoked
• Security policy update • Reacquisition – periodic delete, with require and denial if revoked
• Various options to remove the • Back-pointers – set of pointers from each object to all capabilities of
that object (Multics)
access right of a domain to an • Indirection – capability points to global table entry which points to
object object – delete entry from global table, not selective (CAL)
• Keys – unique bits associated with capability, generated when
• Immediate vs. delayed capability created
• Selective vs. general • Master key associated with object, key matches master key for
access
• Partial vs. total • Revocation – create new master key
• Policy decision of who can create and modify keys – object owner
• Temporary vs. permanent or others?
34
Access Control Models and Access Matrix Implementation
Mandatory Access Control (MAC) Role-based Access Control
• is a strict access control model where the operating system • assigns permissions to roles, and users gain access by being
enforces access policies using assigned those roles
• security labels and
• clearances, • so the system manages roles instead of individual user permissions.
• and users cannot change permissions at their discretion. • Protection can be applied to non-file resources
• Traditionally, operating systems relied on Discretionary • Oracle Solaris 10 provides role-based access control (RBAC) to
Access Control (DAC) (e.g., UNIX file permissions and implement least privilege
Windows ACLs). • Privilege is right to execute system call or use an option within a system call
• Can be assigned to processes
• DAC is weaker because protection depends on user or
• Users assigned roles granting access to privileges and programs
administrator actions.
• Enable role via password to gain its privileges
• MAC provides stronger security: even the root/administrator • Similar to access matrix
cannot bypass policies.
• Resources become accessible only to properly authorized
subjects.
• At its core, MAC assigns labels to subjects and objects.
• When a subject requests access, the system checks policy
rules to decide whether the action is permitted.
• Modern systems often support both DAC and MAC, with MAC
as an optional, high-security configuration (e.g., SELinux,
Trusted Solaris, TrustedBSD/macOS).
35
Cont.
Language-Based Protection Capability-Based Systems
• supplements OS protection by • grant access through capabilities—unforgeable tokens that
specify which object can be accessed and what operations are
enforcing safety properties at allowed.
compile-time and run-time • Early systems: Hydra and CAP
• Specification of protection in a • Modern OS support: Linux and Android (based on POSIX.1e
capabilities)
programming language allows the
• Breaks root privileges into fine-grained units
high-level description of policies for
• Each privilege is represented by a bitmap bit
the allocation and use of resources
• Fine-grained control achieved by setting or masking bits
• Language implementation can provide • Three capability sets:
software for protection enforcement • Permitted – maximum privileges allowed
when automatic hardware-supported • Effective – currently active privileges
checking is unavailable • Inheritable – passed to child processes
• Capabilities apply per process or per thread
• Interpret protection specifications to
• Processes start with full privileges and voluntarily drop them
generate calls on whatever protection
• Direct implementation of the Principle of Least Privilege
system is provided by the hardware
and the operating system • More secure than all-powerful root, but less flexible when adding
new privileges
36