PART IV Cloud Programming Paradigms

Chapter 13 Controller-Based Management Software

Introduction
• Previous chapters describe paradigms used to create cloud-native software systems, including the MapReduce
and microservices paradigms.

• This chapter focuses on software that automates the management of resources. The chapter explains controller-
based designs and concepts of declarative specification and automated state transition.

Traditional Distributed Application Management

• An understanding of controller-based designs begins with understanding how traditional distributed systems are
managed.

• Consider an application that maintains information about employees. One way to deploy such an application
requires each department to run an instance of the application that stores data about the employees in the
department. Authorized users run client applications that access the instance.

• Using multiple instances means the system is general because it allows an authorized user in any department to
access information about an employee in other departments, and the system is efficient because most accesses
go to the local copy.

• Because a traditional distributed system runs instances on many physical computers, managing such a system
usually employs a monitoring tool that alerts a human operator when a server application stops responding.

Periodic Monitoring
• To test an instance, many automated monitoring tools send a special management message to which the
application responds. If no response arrives, the tool might retry once and then alert the operator about the
problem.

• To avoid flooding an instance with a continual stream of monitoring requests, most tools check periodically.

• A monitoring tool runs in the background and never terminates. The code is arranged to repeat a set of steps
indefinitely.

Managing Cloud-Native Applications

• Most cloud-native applications use container replication to scale out services.

• Unfortunately, a cloud-native application is much more complex than a traditional distributed system. To
understand the complexity, consider a few differences.

• Difference in instances. A traditional distributed application consists of a fixed set of static instances. By
contrast, cloud-native applications deploy instances dynamically as needed, creating additional instances to scale
out an application. During times of low demand, only a few instances remain active, and during times of high
demand, many instances remain active. Thus, unlike a tool that monitors a traditional distributed system, a tool
that monitors a cloud-native application must avoid sending erroneous alerts about instances that have been shut
down to decrease the scale.

• Difference in instance locations. In a traditional distributed system, the location of each instance is known in
advance and never changes. For a cloud-native application, however, orchestration software uses the current
load on each server and other factors to choose where to deploy an instance.

• The structure of applications. Unlike traditional distributed systems in which each application is constructed
as a monolithic program, cloud-native applications consist of disaggregated programs that run as multiple
 microservices. Thus, instead of monitoring multiple copies of a single application, software built to manage a
cloud-native application must monitor multiple instances of multiple microservices.

• Application persistence. A traditional application instance executes as a server process that starts when a
computer boots and remains running until the computer shuts down. In contrast, cloud-native software runs in
containers. Typically, a container is designed to service one request and then terminate; a new container is
created for each request, possibly at a new location. Thus, monitoring a cloud-native service requires handling
extremely rapid changes in the set of active instances.

• Figure 13.2 summarizes differences between a traditional distributed application and a cloud-native application
that make managing a cloud-native application complex.

The number of instances does not change The number of instances changes dynamically an application
scales out

The location of each instance is known in advance The locations of instances are chosen dynamically to balance
Load

Each instance consists of a monolithic application An application is disaggregated into multiple microservices

An instance runs as a process An instance runs as a container that that persists indefinitely
exits after handling one request

Figure 13.2 A comparison of traditional and cloud-native applications.

• Monitoring a cloud-native application involves monitoring multiple microservices that are each composed
of a varying number of instances at changing locations, with each instance only persisting for a short time.

Control Loop Concept

• The term control loop is used in automation systems to refer to a non-terminating conceptual cycle that adjusts a
system to reach a specified state.

• A control loop implements a declarative, intent-based interface in which the user merely specifies the intended
result and the control loop implements the changes needed to achieve the result.

Figure 13.3 The conceptual cycle of a control loop for a thermostat that regulates temperature.
Control Loop Delay, Hysteresis, And Instability
• The delay step in a control loop is optional, but can be important for two reasons. First, taking measurements too
rapidly can waste resources and lead to unexpected behavior.

• Hysteresis to refer to the lag between the time a change is initiated and the time it takes effect. Hysteresis is
important because it can cause unexpected results.

• If a control loop takes measurements before a change has time to take effect, the results can be unexpected;
adding a delay to the loop may be necessary to prevent oscillations and guarantee that the system
convergences on the desired state.

The Kubernetes Controller Paradigm And Control Loop

• A programmer who writes traditional control loop code for an IoT device must plan the loop carefully to
prevent unintended behavior that can result from taking measurements too quickly.

• Cloud orchestration systems, including Kubernetes, employ a variant of a control loop that eliminates periodic
measurements.

• Recall from Chapter 10 that Kubernetes uses a set of controllers to automate various management tasks, such
as scale out. We can now understand how a controller can manage container deployments and microservices:
the software employs an intent-based control loop.

• The term controller pattern to characterize the design. Conceptually, each Kubernetes controller:

o Runs a control loop indefinitely

o Compares the actual and desired states of the system
o Makes adjustments to achieve the desired state

• Algorithm 13.2 captures the essence of a Kubernetes controller by showing how it follows a declarative, intent-
based paradigm to manage a microservice.

-------------------------------------------------------------------------------------------------------------------------------------------------------------
Purpose: Act as a controller that manages a microservice consisting of multiple pods of containers

Given: A specification of a desired state of the service

Method: Continually obtain information about the actual state of the service. Compare the actual state to
the desired state, and make adjustments to move the service toward the desired state
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Algorithm 13.2 A Kubernetes controller for a microservice.

An Event-Driven Implementation Of A Control Loop

• Kubernetes can obtain accurate information about the actual state of the system without repeatedly polling to
take measurements by using an event-driven control loop.

• Instead of arranging for the controller to check status periodically, an event-driven control loop configures
components to inform the controller whenever a change occurs.

• Typically, event-driven systems use message passing. When a change occurs, the controller receives a message
about the change.

• For example, when the specification file changes, a file system component sends the controller a message.
Similarly, the orchestration component sends the controller a message when an instance exits.

• In essence, using an event-driven approach reverses the responsibility for obtaining information about the state
 of the system. In a traditional control loop, the controller actively polls the system to obtain the information; in
an event-driven control loop, the controller waits passively to be informed when a change occurs.

• Figure 13.4 shows the steps an event-driven controller follows to react to incoming messages.
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Read and parse the specification file
Use the specification to create pods for the system loop forever {
wait to receive a message
if the specification changed, read and parse the file
If the status of a pod has changed, record the change
if the change leaves the service outside the desired state, adjust the system to move toward the desired state
}
-------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure 13.4 An event-driven implementation of a Kubernetes controller in which the controller waits
passively to be informed when a change occurs.

• Note that using the event-driven paradigm means that a controller does not perform a delay step as part of the
control loop. From a programmer’s point of view, the approach means that there is no need to choose a delay,
and there is never a problem with a delay that is too small or too large.

• The controller paradigm used by Kubernetes follows a declarative, intent-based approach in which a user
specifies the desired state of a service and a controller continually adjusts the system to move toward the desired
state. Using an event-driven implementation for a controller avoids needless polling.

Components Of A Kubernetes Controller

• Conceptually, Kubernetes runs multiple, independent controllers. In practice, Kubernetes implements all
controllers with a single background daemon process, the kube-controller-manager.

• Kubernetes divides each controller into three pieces:

o Informer or SharedInformer (watcher) - looks for changes in the state of Kubernetes objects and sends
an event to the Workqueue whenever a change occurs.

▪ To avoid polling a list to find new items, Kubernetes includes a Listwatcher component that
generates a notification about the creation of a new instance or a change in a current instance.

▪ An Informer component keeps a local cache of the state of resources. Many microservices use
a set of controllers to handle multiple aspects of the service. In such cases, SharedInformer
provides an alternative that allows cached information about the state of the service to be shared
among a set of controllers.

o Workqueue and Dispatcher – contains a list of changes that have occurred in the state of the service.
New items are added when a change occurs, and a Dispatcher removes items as they are processed.
When an item appears on the Workqueue, it may mean that the current state of the service no longer
adheres to the desired state.

o Workers – When it extracts an item from the Workqueue, the Dispatcher checks the specification. If the
system no longer conforms to the desired state, the Dispatcher invokes a worker to make adjustments to
move toward the desired state (e.g., by creating a new instance to replace one that has completed). As
with most cloud facilities, it is possible to scale a controller by creating multiple workers.

• A programmer must create code that specifies the steps that should be taken to align the state of a computation
with the desired state. The programmer writes a function, Reconcile, that Kubernetes calls. When it runs,
Reconcile will have access to the specification, and can adjust the state of the system accordingly.
 • Although the description above implies that the controller operates continuously, Kubernetes does provide a way
for a controller to run periodically. A value known as ResyncPeriod specifies when a controller should
revalidate all items in the cache. In essence, ResyncPeriod specifies the time at which the controller compares
the current state to the desired state and takes action accordingly. Of course, a designer must be careful not to
set ResyncPeriod too low or the controller will incur a high computational load needlessly.

Custom Resources And Custom Controllers

• Kubernetes defines a set of core resources and controllers that cover most of the common tasks associated with
deploying and managing a cluster that includes multiple pods.

• In addition to an extensive set of built-in controllers and pre-defined resources, Kubernetes allows a user to
define custom resources and custom controllers.

• Like built-in controllers, a custom controller employs the event-driven approach. To create a custom
controller, a software engineer must write code for the basic components:

o a Workqueue
o Worker(s) to process events
o and one of the following:
▪ SharedInformer, if the controller maintains information about multiple pods
▪ an Informer/watcher, if the controller has small scope.

• Although creating such components may seem complex, the kube-controller-manager handles many of the
details.

Kubernetes Custom Resource Definition (CRD)

• Kubernetes provides a facility, known as the Custom Resource Definition (CRD), that helps a software
engineer create a custom resource. The facility allows one to define new objects and then integrate them with a
Kubernetes cluster. Once it has been defined, a new object can be used exactly like a native Kubernetes object.

• Various approaches have been used that allow an application to access multiple underlying facilities. For
example, Figure 13.5 illustrates a proxy service.

Figure 13.5 Illustration of a proxy service that allows applications to access multiple types of
databases.

• A proxy service fits between applications and underlying databases. Instead of accessing a database directly, an
application invokes the proxy service.

• Each database technology defines a specific set of commands that must be used to communicate with the database
(i.e., a database-specific API). The proxy accepts a generic set of database requests from applications. To fulfill
a request, the proxy translates the request into database-specific commands (e.g., for Redis or MongoDB), and
issues the commands to the underlying database.

• A proxy service offers a single, generic interface that applications use, and only the proxy needs to understand
the details of underlying databases. The key point is that an application can switch from one database to another
without being modified.
 • The CRD facility in Kubernetes offers the same benefits as a proxy without requiring a user to create and manage
a separate service.

• To use CRD, a software engineer creates a Custom Resource Definition that accepts generic database commands
just as a proxy does along with code that translates generic requests into database-specific commands.

• A CRD can function like a proxy service, but instead of running and managing a separate service, Kubernetes
can manage instances of the CRD.

Service Mesh Management Tools

• Management functions include service discovery, health checking, secure communication among a set of
services, key-value object storage, and support for deploying an application across multiple data centers.

• Most service mesh tools have been designed to work with Kubernetes. Examples include HashiCorp Consul,
Istio, and Linkerd from the Cloud Native Computing Foundation (CNCF) project.

• One of the main arguments for mesh management tools arises from the need for security. Kubernetes provides a
way to deploy services, and a mesh management tool ensures that communication among services remains
secure. In addition, a mesh management tool may offer additional control functions for Kubernetes clusters.

Reactive Or Dynamic Planning

• The term reactive (dynamic) planning refers to the rapid planning required to accommodate new conditions.

• The idea of reactive/dynamic planning is straightforward: adapt quickly to the constant stream of changes in both
the environment and demand by planning a new desired state and moving the service to the desired state. The
controller paradigm and control loops can be used to implement reactive/dynamic planning.

• Interestingly, constant change may mean that a particular service never reaches a steady state. Instead, the
management system adapts to continual changes by constantly revising the desired state of the service. Provided
controllers are designed to make useful decisions and accommodate any hysteresis introduced by queues, the
service can continue to change without becoming unstable.

A Goal: The Operator Pattern

• Most controllers handle routine, repetitive tasks. The term operator pattern describes an envisioned control
system that can handle the remaining management tasks that normally require a human operator, such as
identifying anomalies, diagnosing problems, understanding how and when to apply and deploy software updates,
and how to delete resources associated with a service when the service is shut down. Software that achieves the
envisioned goal will require AIops.

Summary
• Managing a cloud-native application is inherently more complex than managing a traditional distributed system.
In a cloud-native application, the number and location of instances changes, a single application may be
disaggregated into multiple microservices, and instead of persisting, an instance exits after handling one request.

• A control loop is a non-terminating, intent-based computation that continually measures the state of a system,
compares the actual state to the desired state, and makes adjustments to move the system toward a desired state.

• Kubernetes provides a set of built-in controllers that each run a control loop for one particular aspect of a
cluster. A Kubernetes controller consists of three components that implement an event-driven approach: an
Informer/watcher (or SharedInformer) that sends events when the state of the system changes, a Workqueue that
holds a list of events, and a Worker that handles events.

• Kubernetes offers users the ability to define custom resources and custom controllers. A Custom Resource
Definition (CRD) can be used to create a generic interface for applications and allow each instance to use a
specific underlying technology.