Chapter 21
Secure Operating System
Llorenç Romá and Bernard Tellenbach
21.1 Introduction
The operating system (OS) is the backbone of every modern computer system,
managing the system’s resources and executing applications. Its security is critical
as a vulnerability in the OS or any applications running on it can expose the entire
system to risk. Different types of OS can be considered security-wise: (1) security-
focused OS and (2) security-evaluated OS. A security-focused OS aims to provide
a higher level of security by protecting the rest of the system from modules that an
attacker might exploit. In contrast, a security-evaluated OS is certified by an external
security-auditing organization. Hardening measures will vary as different use cases
have different requirements for a secure OS. In addition, different technologies are
used to complement the security provided by the OS. The trend for secure OSes
is, among others, the use in container-focused OSes and intelligent vehicles where
digital features are increasing, as well as in mobile phones.
21.2 Analysis
21.2.1 Definition
Every modern computer system runs a core piece of software executed on top of the
hardware. This software is the operating system (OS). It is responsible for allocating
the primary resources of the system (e.g., CPU, memory, communication ports) and
L. Romá () · B. Tellenbach
Cyber-Defence Campus, Thun, Switzerland
e-mail: [Link]@[Link]; [Link]@[Link]
© The Author(s) 2023 115
V. Mulder et al. (eds.), Trends in Data Protection and Encryption Technologies,
[Link]
116 L. Romá and B. Tellenbach
supervising the execution of all the applications within the system. Given the crucial
role of the OS, its security (or the lack of security) might have a significant impact
on the whole system: a vulnerability in the OS, or any applications running in it,
exposes a danger to all the other applications running in the system as well as to
all the data stored in it. This situation becomes highly problematic when the system
stores important (confidential) data or runs critical applications in high-risk facilities
(e.g., satellite communications, power plants, banking systems, aircraft systems, and
SCADA systems). Therefore, it is essential to improve OS security to ensure data
integrity, confidentiality and availability.
When discussing secure operating systems, we generally refer to (1) security-
focused OS and (2) security-evaluated OS. In any case, such operating systems are
designed to provide a higher level of security.
(1) Security-Focused OS
A security-focused operating system should guarantee the secure or trusted execu-
tion of components that might not be secure (programs). That is, the OS should
protect the rest of the system from modules that an attacker might exploit to get
control of the system, for instance, using sandboxing, compartmentalization or by
isolating cryptography functions and key management. QubeOS is one such OS,
which is especially valuable in industries where sensitive data has to be securely
segregated. Other examples include Tails OS and ReactOS.
In addition, to provide an extra level of security at different layers, OSes may
leverage other software and hardware technologies and mechanisms, described in
more detail in other chapters, such as Secure Boot, Trusted Platform Modules
(TPM), Hardware Security Modules (HSM), disk encryption, network protection
and other security-related features such as access control lists (ACLs), event
auditing.
One example of such hardware-based technology is HSM (Chap. 16). An HSM
can improve the security of an operating system by providing secure storage for
cryptographic keys and other sensitive data, such as passwords and certificates. This
makes it much more difficult for attackers to access the keys and other sensitive data,
even if they have successfully compromised the operating system or other software
on the computer. In addition, an HSM can also perform cryptographic operations,
such as encryption, decryption, and signing. By offloading these operations to the
HSM, the operating system can reduce its exposure to security threats, as the keys
and sensitive data are not accessible to the operating system or other software.
Similarly to an HSM, a Trusted Execution Environment, or TEE (Chap. 18) is
a secure area of a computing device, typically implemented on the chip itself, that
provides a secure environment for executing sensitive operations. The TEE is also
used to provide secure storage for cryptographic keys and other sensitive data, such
as passwords and certificates, which would protect such assets in a scenario where
the OS is compromised. For example, iOS uses a dedicated, isolated and hardware-
backed subsystem called secure enclave to isolate important cryptographic tasks.
And on Android smartphones, it depends on the manufacturer of the smartphone
21 Secure Operating System 117
whether and which type of TEE is present (e.g., Google Pixel smartphones contain
the Titan M chip for this purpose).
An example of a software-based solution that improves the security of an
operating system is SELinux [1]: a security feature built into the Linux operating
system that provides enhanced security through the use of mandatory access control
(MAC) policies. SELinux defines access control policies that restrict processes
and users’ actions on files, processes, and network resources. The policies are
implemented in software and are enforced by the Linux kernel
(2) Security-Evaluated OS
A security-evaluated OS is an OS that has achieved certification from an external
security-auditing organization. However, they still need to implement more security
mechanisms to make certain system areas more secure (e.g., cryptographic modules,
fine-grained access control) according to the criteria. Some of the most popular
evaluation criteria are Common Criteria [2], FIPS 140-2 [3], and ITSEC [4].
Examples of such OSs are SUSE Linux or some Red Hat Linux Enterprise versions,
Windows 10 Enterprise, etc.
Even though a baseline exists for achieving a minimum level of security, the
ultimate set of requirements to make a secure OS depends on the specific use
case. For instance, a mobile OS has different requirements than a container-focused
OS. Therefore, different measures can be taken to harden the underneath operating
system for each specific use case.
21.2.2 Trends
One envisioned trend of secure OSes is their use in container-focused OSes.
Over the last five years, many enterprises have moved their primary business
activities and deployed their applications in container environments. However, those
environments present particular risks since multiple applications/services run on
containers on the underlying OS, sharing the same set of resources. Therefore, if
an attacker manages to compromise the host OS, the rest of the system could be
affected. For instance, they disrupt the applications running on the top or steal
critical business information. On the other hand, if an attacker compromises an
application running inside a container, he/she could try to escape the container and
gain access to the host OS and/or pivot to other containers, achieving the same
results as in the previous example. With that in mind, it seems reasonable that a
container-focused OS might also be security-focused, including features such as
those mentioned in Sect. 21.2.1. That is why recently, the first standards on container
security are emerging [5]. Examples of well-known container-focused OSes are
FlatCar Container Linux [6] or Bottlerocket [7]. However, those are not considered
secure OSes since some of the features mentioned in previous sections are not
implemented. Another example of such an OS that focuses on security is ARCA
OS, from CySec [8], a Swiss startup launched in 2018 in Lausanne, EPFL.
118 L. Romá and B. Tellenbach
Another trend comes with the increasing development of intelligent vehicles.
Modern vehicle industries (i.e., automotive, aeronautic) deploy more and more
digital features; therefore, the attack surface widens significantly. A compromised
component should not be able to endanger the rest of the system. For example,
a car’s Bluetooth vulnerability should not allow an attacker to control the brakes.
Several efforts are being carried out to improve the development of vehicle-secure
OSes. For instance, Automotive Grade Linux [8] or Red-Hat In-Vehicle OS [9] and
the tendency should move to increase the safety of future intelligent vehicles.
Finally, in the last years, a trend toward more hardware security components,
especially for separation of cryptographic functions and implementation of critical
operations has emerged. Examples of technologies that are part of this trend have
been mentioned above, namely HSM, TEEs and SELinux. These components are
being used widely in critical infrastructures but also in smartphones to improve the
security provided by the operating system.
21.3 Consequences for Switzerland
The use of secure operating systems is beneficial not only for high-risk systems but
also for individuals who want to protect their assets.
In more critical environments, such as governments and military systems that
typically have higher security restrictions, security-evaluated OSes are a convenient
option used in other countries, ensuring those systems fulfill a set of security
requirements. In addition, these OSes ensure a minimum level of trustworthiness
and security by limiting access to specific resources and isolating components.
For Switzerland, a movement toward more secure operating systems is required
to improve the security of sectors such as the banking and the military sectors.
Individuals and businesses are also the targets of cyber attackers. While individ-
uals often opt for functionality over security, it is crucial to raise the importance of
using security-focused OSes with extra security features to reduce the attack surface
and protect their assets.
21.3.1 Implementation Possibilities: Make or Buy
Large amounts of knowledge, human resources, and time is required to create an
OS from scratch. An OS is a piece of software responsible for controlling a device’s
hardware and providing an interface whereby an operator can use it. However, most
OSes are much more sophisticated and perform many tasks: manage multitasking,
memory management, multiple processor cores, networking support, and drivers
for all standard hardware. For instance, the Linux kernel (i.e., one part of an OS)
consists of several millions of lines of code. Therefore, if the development of an OS
were already a complex task, adding security concerns on top of that would require
21 Secure Operating System 119
Table 21.1 Implementation possibilities for different sectors
even more resources. For instance, the vulnerability management system is typically
the most critical and time-consuming part of an OS conception. In addition, one of
the most significant shortcomings of OSes built from scratch is the limited support
for existing software, which could limit the functionalities of a given system.
On the other hand, given the availability of existing solutions buying appears to
be a preferable option, especially regarding the maturity of the existing solutions
compared to an OS developed from zero and given the complexity of such a task.
When building an OS from scratch, many bugs might be introduced, and the time to
reach a certain level of stability and maturity might require several years. Operating
systems in the market have been developed for decades, and security has been
considered a significant concern to all of them. In addition, one could implement
additional security features on top of an existing OS to fill the needs (e.g., some
specific cryptographic functions or authentication mechanisms) (Table 21.1).
21.4 Conclusion
Although the use of secure operating systems is not a definitive solution to protect
against all the dangers of current cyberspace, it is clear that it can reduce the
impact of individual vulnerable applications or modules being exploited on the
whole system. Moreover, improving the security of the operating system is only
one measure that can be adopted to reduce the attack surface: the combination
of several other technologies, such as the ones discussed in other chapters, might
120 L. Romá and B. Tellenbach
increase the protection against cyber attacks and limit the ability of an attacker
to exploit our systems. In addition, the human factor is still a significant factor
concerning the overall security of a system. A secure OS will reduce the attack
surface. Nevertheless, training teams of users is essential for security.
References
1. Stephen Smalley Walsh and Greg. Selinux: Nsa’s open source security enhanced linux, 2003.
2. Common Criteria: New CC Portal. [Link] August 2022.
3. National Institute of Standards and Technology. Security Requirements for Cryptographic
Modules. Technical Report Federal Information Processing Standard (FIPS) 140-2, U.S.
Department of Commerce, December 2002.
4. C. Jahl. The information technology security evaluation criteria. pages 306 – 312. IEEE
Computer Society, January 1991.
5. Murugiah Souppaya, John Morello, and Karen Scarfone. Application Container Security Guide.
Technical Report NIST Special Publication (SP) 800-190, National Institute of Standards and
Technology, September 2017.
6. Flatcar Container Linux. [Link] August 2022. Flatcar.
7. Bottlerocket OS. [Link] August 2022. original-date:
2019-04-03T23:28:55Z.
8. ARCA | Confidential Computing | Container Workload. [Link]
9. What is automotive grade linux? [Link] August 2022. Automotive
Grade Linux.
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0
International License ([Link] which permits use, sharing,
adaptation, distribution and reproduction in any medium or format, as long as you give appropriate
credit to the original author(s) and the source, provide a link to the Creative Commons license and
indicate if changes were made.
The images or other third party material in this chapter are included in the chapter’s Creative
Commons license, unless indicated otherwise in a credit line to the material. If material is not
included in the chapter’s Creative Commons license and your intended use is not permitted by
statutory regulation or exceeds the permitted use, you will need to obtain permission directly from
the copyright holder.