Operating System Security

Operating System Security

 Operating System Security refers to the set of measures
and controls implemented within an operating system to
protect the system and its data from unauthorized access,
misuse, or damage.

It ensures the confidentiality, integrity, and availability of

resources by managing:
User authentication and authorization
Access controls
System updates and patches
Malware protection
Secure configuration of system components
10/08/2025 2
Operating System

 each layer of code needs

measures in place to
provide appropriate
security services

 each layer is vulnerable

to attack from below if
the lower layers are not
secured appropriately

Security Layers
 Measures
• the 2010 Australian Defense Signals Directorate
(DSD) list the “Top 35 Mitigation Strategies”
• over 70% of the targeted cyber intrusions
investigated by DSD in 2009 could have been
prevented
• the top four measures for prevention are:
– patch operating systems and applications
using auto- update
– patch third-party applications
– restrict admin privileges to users who need them
– white-list approved applications
 Operating System Security
• possible for a system to be compromised during the
installation process before it can install the latest
patches
• building and deploying a system should be a planned
process designed to counter this threat
• process must:
– assess risks and plan the system deployment
– secure the underlying operating system and then
the key applications
– ensure any critical content is secured
– ensure appropriate network protection
mechanisms are used
– ensure appropriate processes are used to
maintain security
 System Security Planning

the first step in deploying a

new system is planning
plan needs to identify
appropriate
personnel and
training to install and planning should
manage the system include a wide
security assessment
of the organization

planning process needs to

determine security
requirements for the aim is to maximize
system, applications, data, security while
and users minimizing costs
System Security Planning Process
any additional security
the purpose of the system, measures required on the
the type of information who will administer the
system, and how they will system, including the use of
stored, the applications and
manage the system (via host firewalls, anti-virus or
services provided, and their
local or remote access) other malware protection
security requirements
mechanisms, and logging

what access the system has

the categories of users of to information stored on
the system, the privileges
other hosts, such as file or
they have, and the types of
database servers, and how
information they can access
this is managed

how access to the

how the users are information stored on the
authenticated
system is managed
 Operating Systems Hardening
• first critical step in securing a system is to
secure the base operating system
• basic steps
• install and patch the operating system
• harden and configure the operating system to
adequately address the identified security needs of the
system
• install and configure additional security controls, such
as anti-virus, host-based firewalls, and intrusion
detection system (IDS)
• test the security of the basic operating system to
ensure that the steps taken adequately address its
security needs
 Initial Setup and Patching

should stage
system and validate all
security begins patches on the
with the test systems
installation of initial before
overall deploying them
the operating installation
boot
should
system install the process in production
minimum must also
necessary for be secured
the desired
system

the integrity critical that

full the system
ideally new installation and source
systems of any be kept up to
and date, with all
should be hardening additional
device critical
constructed process security
on a should occur driver code
must be related
protected before the patches
network system is carefully
validated installed
deployed to
its intended
location
 •Remove
Unnecessary
Services, • when performing the initial
Applications, installation the supplied
Protocols defaults should not be
used
– default
configuration is set to
• if fewer software packages maximize ease of use and
are available to run the risk functionality rather than
is reduced security
• system planning process – if additional
should identify what is packages are needed
actually required for a later they can be installed
given system when they are required
 • system planning process should
consider:
•Configure
– categories of users on
Users, Groups, the system
and – privileges they have
Authentication
– types of information they
can access
– how and where they
are defined and
authenticated
• not all users with access to a • default accounts included as part
system will have the same of the system installation should be
access to all data and secured
resources on that system
– those that are not
• elevated privileges should required should be either
be restricted to only those removed or disabled
users that require them, and
then only when they are – policies that apply to
needed to perform a task authentication credentials
configured
10/08/2025 11
 • Install
• Configure
• Additional
• Resource
• Security
• Controls
• Controls

• once the users and

groups are defined, • further security
appropriate possible by installing
permissions can be set and configuring
on data and resources additional security
• many of the security tools:
hardening guides – anti-virus software
provide lists of – host-based firewalls
recommended changes –
to the default access IDS or IPS software
configuration – application white-
listing
 • checklists are included in
• Test security hardening guides
• there are programs
the specifically designed to:
System – review a system to
ensure that a system
Security meets the basic security
requirements
– scan for known
• final step in the process of vulnerabilities and poor
configuration practices
initially securing the base
operating system is • should be done following
security testing the initial hardening of the
• goal: system
– ensure the previous • repeated periodically as
security configuration steps part of the security
are correctly implemented
–
maintenance process
identify any
possible
vulnerabilities

10/08/2025 13
 Application Configuration
• may include:
– creating and specifying appropriate data storage
areas for application
– making appropriate changes to the application or service
default configuration details
• some applications or services may include:
– default data
– scripts
– user accounts
• of particular concern with remotely accessed services such
as Web and file transfer services
– risk from this form of attack is reduced by ensuring that
most of the files can only be read, but not written, by the server
 Encryption Technology

is a key
enabling
technology
that may be if secure network cryptographic
used to services are file systems are
secure data must be provided using if secure network
another use of
TLS or IPsec encryption
both in configured services are
transit and and suitable public and provided using
private keys must SSH, appropriate
when stored appropriate server and client
cryptographic be generated for keys must be
keys created, each of them created
signed, and
secured
 Security Maintenance
• process of maintaining security is
continuous
• security maintenance includes:
– monitoring and analyzing logging
information
– performing regular backups
– recovering from security
compromises
– regularly testing system security
– using appropriate software
maintenance processes to patch and
update all critical software, and to
Logging in OS Security
Logging is the process by which an operating system
records system events, user activities, and security-
related actions in log files. These logs are essential for
monitoring, auditing, troubleshooting, and incident
response.
Why Logging Matters in OS Security
 Detect suspicious activity (e.g., failed login attempts)
 Track user actions for accountability
 Audit changes to critical files or settings
 Investigate incidents after a breach
 Ensure compliance with legal and industry standards

10/08/2025 17
 automated can only inform you
analysis is about bad things
preferred that have already
happened

generates significant in the event of a system

volumes of information and breach or failure, system
it is important that administrators can more
sufficient space is quickly identify what
allocated for them happened

Logging
range of data acquired should key is to ensure you
be determined during the capture the correct
system planning stage data and then
information appropriately monitor
can be and analyze this data
generated
by the
system,
network and
applications
 Data Backup and Archive

performing regular needs and policy relating

backups of data is to backup and archive
a critical control backup archive should be determined
that assists with during the system
• the process • the process of
maintaining the of making retaining copies of data
planning stage
integrity of the copies of data over extended periods • kept online or offline
system and user at regular of time in order to meet • stored locally or transported to
intervals legal and operational a remote site
data requirements to access • trade-offs include ease
• may be legal or past data of implementation and cost
operational versus greater security and
requirements for the robustness against different
retention of data threats
 Linux/Unix Security
• patch management
– keeping security patches up to date is a widely
recognized and critical control for maintaining security
– application and service configuration
– most commonly implemented using separate text files
for each application and service
– generally located either in the /etc directory or in the installation
tree for a specific application
– individual user configurations that can override the system
defaults are located in hidden “dot” files in each user’s home directory
– most important changes needed to improve system
security are to disable services and applications that are not
required
 Linux/Unix Security
• users, groups, and permissions
– access is specified as granting read, write,
and execute permissions to each of owner, group,
and others for each resource
– guides recommend changing the
access permissions for critical directories
and files
– local exploit
• software vulnerability that can be exploited by an
attacker to gain elevated privileges
– remote exploit
• software vulnerability in a network server that could be
triggered by a remote attacker
 Linux/Unix Security

remote access controls logging and log rotation

• several host firewall programs • should not assume that the
may be used default setting is
• most systems provide an necessarily
administrative utility to select appropriate
which services will be permitted
to access the system
 Linux/Unix Security
• chroot jail
– is a security technique in which the root
directory for a running process is changed to a
specified directory, effectively isolating the
process from the rest of the file system.
– restricts the server’s view of the file system
to just a specified portion
– uses chroot system call to confine a
process by mapping the root of the
filesystem to some other directory
– file directories outside the chroot jail aren’t
visible or reachable
– main disadvantage is added complexity
 Windows Security

patch management users administration

• “Windows Update” and and access controls
“Windows Server • systems implement
Update Service” assist discretionary access
with regular controls resources
maintenance and
should be used • Vista and later systems
include mandatory integrity
• third party applications controls
also provide automatic
update support • objects are labeled as
being of low, medium, high,
or system integrity level
• system ensures the
subject’s integrity is equal
or higher than the object’s
level
• implements a form of the
Biba Integrity model
 Windows Security
Users Administration and Access Controls

Windows systems also

define privileges combination of share and
NTFS permissions may be
• system wide and granted to user
accounts used to provide additional
security and granularity
when accessing files on a
shared resource

User Account Control (UAC) Low Privilege Service

• provided in Vista and later systems Accounts
• assists with ensuring users with • used for long-lived service
administrative rights only use processes such as file, print, and
them when required, otherwise DNS services
accesses the system as a
normal user
 Windows Security
application and service
configuration

• much of the configuration information

is centralized in the Registry
• forms a database of keys and values that
may be queried and
interpreted by applications
• registry keys can be directly modified
using the “Registry Editor”
• more useful for making bulk changes
 Windows Security
– other security controls
– essential that anti-virus, anti-spyware, personal firewall,
and other malware and attack detection and handling software
packages are installed and configured
– current generation Windows systems include basic
firewall and malware countermeasure capabilities
– important to ensure the set of products in use are compatible

– Windows systems also support a range of cryptographic

functions:
– encrypting files and directories using the Encrypting File
System (EFS)
– full-disk encryption with AES using BitLocker

– “Microsoft Baseline Security Analyzer”

– free, easy to use tool that checks for compliance with
Microsoft’s security recommendations
 Virtualization
• Virtualization is the process of creating a virtual
version of something, such as an operating
system, server, storage device, or network
resources. It enables multiple virtual
environments (or virtual machines) to run on a
single physical system.
• In the context of OS security, virtualization
provides an additional layer of isolation and
resource control.
• provides support for multiple distinct operating
systems and associated applications on one
physical system
10/08/2025 28
 Types of Virtualization
1. Full Virtualization
 Uses a hypervisor (software layer) to create virtual machines (VMs)
that operate as if they have their own hardware.
 Examples: VMware, Hyper-V, KVM.
2. Paravirtualization
 The guest OS is aware of the virtualization and communicates
directly with the hypervisor to improve performance.
 Example: Xen.
3. Containerization
 A form of virtualization at the operating system level, where
applications are isolated in containers but share the same
underlying OS kernel.
 Examples: Docker, Kubernetes.

10/08/2025 29
 Virtualization Alternatives

application virtualization

full virtualization
allows
applications
written for one
environment to virtual machine monitor (VMM)
execute on some multiple full
other operating operating system
system instances execute
in parallel coordinates access between
hypervisor each of the guests and the actual
physical hardware resources
 Native vs. Hosted Virtualization
Security Layer
Virtualization comes in two main types: Native (or
Bare-Metal) and Hosted. Both types create virtual
machines (VMs), but they differ in how they
interact with the underlying hardware and their
security models.

10/08/2025 31
Native Virtualization Security Layers
Native virtualization runs directly on the physical hardware. The hypervisor
itself is installed as the operating system, meaning there’s no underlying host
OS between the hypervisor and the hardware.
Examples: VMware ESXi, Microsoft Hyper-V, Xen.

How it Works:
The hypervisor controls and manages the hardware resources directly, and
virtual machines are created and run on top of the hypervisor.
Security Layer:
•Hypervisor Security: The security of the hypervisor is critical, as it has direct
control over hardware resources. If the hypervisor is compromised, all VMs on
the host could be at risk.
•Isolation: VMs are well-isolated from each other because the hypervisor
controls the direct interaction with hardware and enforces security boundaries.
•No Host OS Dependency: There’s no intermediary OS running on the host,
which reduces potential attack vectors.
•Patch Management: Since the hypervisor is directly managing the hardware, it
requires regular security updates to prevent vulnerabilities.

10/08/2025 32
Native Virtualization Security Layers

User Apps User Apps User Apps

...
Guest O/S 1 Guest O/S 2 Guest O/S n
Kernel Kernel Kernel

Hypervisor/ VMM
BIOS / SMM

Physical Hardware

Figure 12.2 Native Virtualization Security Layers

Hosted Virtualization Security
Layers
Hosted virtualization involves installing a hypervisor on top of an existing
operating system (OS), which is known as the host OS. The hypervisor runs
as an application within the host OS.
Examples: VMware Workstation, Oracle VirtualBox, Parallels Desktop.
How it Works:
•The host OS provides the necessary resources (CPU, memory, storage), and
the hypervisor creates virtual machines on top of it.
Security Layer:
•Host OS Security: Since the hypervisor is an application within the host OS,
the security of the host OS is paramount. Any vulnerability in the host OS could
potentially affect the hypervisor and the VMs running on it.
•VM Isolation: The hypervisor ensures VM isolation, but there is an additional
layer (the host OS) between the virtual machines and hardware, which could
be exploited.
•Host OS Dependency: The security of the host OS directly impacts the VMs,
as they share the host OS’s resources.

10/08/2025 34
Hosted Virtualization Security Layers

User Apps User Apps

...
Guest O/S Guest O/S
Other 1 Kernel n Kernel
User
Apps Hypervisor/ VMM

Host Operating System Kernel

BIOS / SMM

Physical Hardware

Figure 12.3 Hosted Virtualization Security Layers

 Virtualization Security
• Issues
security concerns include:
– guest OS isolation
• ensuring that programs executing within a guest
OS may only access and use the
resources allocated to it
– guest OS monitoring by the hypervisor
• which has privileged access to the programs and
data in each guest OS
– virtualized environment security
• particularly image and snapshot management
which attackers may attempt to view
or modify
Securing Virtualization Systems
• carefully plan
the security of the
virtualized system
• secure all
organizations elements of a full
virtualization
using solution and
maintain their
virtualization security
• ensure that the
should: hypervisor is
properly secured
• restrict and
protect
administrator
access to the
virtualization
solution
 Hypervisor Security
• should be
– secured using a process similar to securing an operating
system
– installed in an isolated environment
– configured so that it is updated automatically
– monitored for any signs of compromise
– accessed only by authorized administration
• may support both local and remote administration so must be
configured appropriately
• remote administration access should be considered and
secured in the design of any network firewall and IDS
capability in use
• ideally administration traffic should use a separate network
with very limited access provided from outside the
organization
Virtualization
Infrastructure access to VM

Security image and

snapshots
must be
carefully
controlled

access must
be limited to
just the
appropriate
guest

systems manage
access to hardware
resources
 Summary
• system security planning • Linux/Unix security
• operating systems hardening – patch management
– initial setup and patching – application configuration
– remove unnecessary services – users, groups, permissions
– configure users and groups – remote access
– test system security – security testing
• application security • windows security
– application configuration – patch management
– encryption technology – users administration
– security maintenance and access controls
– data backup – application
– virtualization security and service
configuration
• virtualization alternatives
– security testing