Three Pillars Of Observability

Observability forms a crucial aspect in modern software systems, especially when it comes to distributed
and complex architectures. It entails the ability to understand, measure, and gain insights on what’s
happening within the system by analyzing data from various sources. Observability enables the
developers, operational teams, and other stakeholders to troubleshoot and understand the behavior of
their applications and infrastructure.

In other words, observability allows you to answer simple questions such as:

 What’s happening in the system at any particular time?

 How the system is performing in terms of response times and resource utilization?
 Are there any errors/anomalies happening within the system?
 How are different components interacting with each other at any particular time in the system?

Note that the concept of observability is vital in dynamic and highly distributed systems such as
microservices architecture, cloud-based applications such as Azure and AWS, as well as in containerized
environments like Docker and Kubernetes. These kinds of systems generate huge amounts of data that
calls for effective tools and practices to collect, store and analyze such kind of data to gain useful
insights.

In this post, you will cover the different pillars of observability and how you can leverage each of them
to monitor, analyze and troubleshoot systems for improved reliability, fast incidence response, better
performance and maintain overall higher quality software.

Importance of Observability for System Health and Monitoring

Observability is a critical aspect in ensuring the health and proper functioning of complex systems
whether software applications, IT infrastructure or integrated systems. It plays a crucial role in system
health and troubleshooting for various reasons:

 Issue detection and diagnosis: Observability provides real-time insights on system behavior and
performance. This means that observability tools and metrics quickly detect any errors,
anomalies and deviation from system normal functioning, leading to prompt diagnosis of
problems thereby minimizing downtime.
 Root cause analysis: It is crucial to identify the accurate root cause of the system failure or
performance issue. With observability tools, engineers can pinpoint the real cause of the
problem/issue, as they provide detailed information about the system’s internal components,
dependencies and integrations.
 Reducing MTTR (Mean Time to Repair): Observability enables faster resolution of issues by
reducing the time to identify, dragonize and fix problems. Since the observability tools ensures
clear understanding of what happening within the system, IT teams can therefore respond to
issues faster, hence reduced MTTR, hence increasing overall system reliability.
  Proactive monitoring: Using observability tools, organizations can proactively monitor their
systems and identify potential issues before they escalate into bigger problems. Predictive
analysis drawn from the observability data assists to prevent service outages and system
disruptions enhancing overall system reliability.
 Continuous improvement: Observability data helps organizations to analyze long term data
trends and patterns, hence supporting the continuous improvement initiatives. By monitoring
Key Performance Indicators (KPI) over time, teams can identify opportunities for possible
improvements and optimizations.

Reasons of using Observability Approach over Traditional Monitoring

Approaches
The observability approach offers several benefits compared to the traditional monitoring. Traditional
monitoring approach involves using predefined metrics and monitoring tools to keep track of the
system’s performance and health. It heavily relies on the monitoring Key Performance Indicators (KPIs)
and predefined thresholds to alert administrators when certain metrics exceeds predefined limits.

Despite, the traditional monitoring approach been the standard for many years and serves its purpose
still, It faces major challenges when it comes to complex and dynamic systems. As a result, the
organizations are switching to observability approach as a more comprehensive and adaptable way to
monitor and understand modern software systems.

The observability approach offers several compelling reasons to choose it over the traditional
monitoring approaches as explained below:

 Visibility into complex systems: Observability provides deep insights into complex, distributed
systems that involve microservices, containers, serverless architectures and other modern
systems. It allows engineers to better understand the system’s internal states and interactions
that traditional monitoring cannot reveal.
 Adaptability to change: Traditional monitoring relies on predefined metrics that are not suitable
for the system that is evolving and new components added. Observability is designed to handle
constant changes and supports adding new components without necessarily modifying the
monitoring infrastructure.
 Support for unknown unknowns: In traditional monitoring, you can only monitor what you
know and need to measure. However, normal systems don’t work that way as unprecented
issues may occur. Observability solves the issue as it allows the engineers to explore vast data
and uncover the patterns they were not explicitly looking for.
 Distributed tracing: One of the core pillars of observability is tracing. Distributed tracing
provides end to end visibility to into the flow of requests through multiple microservices. This
makes pinpointing performance bottlenecks easier and enhances the overall understanding of
interactions between different services, which is not possible in traditional monitoring.
 Holistic understanding of system health: Observability tools and practices focuses on the entire
system’s health, not just on individual components or metrics unlike traditional monitoring.
 Therefore, it helps the engineers to identify how different parts of the system interact with each
other and how they collectively contribute to the overall system performance.

Overall, observability is becoming increasing popular in modern software development and operations
due to its ability to provide a deeper understanding of complex systems and enable more effective
monitoring, debugging and problem solving processes.

Three Pillars of Observability

The three pillars of observability include metrics, logs, and traces. These pillars serve as the foundation
for building a comprehensive observability strategy.

Metrics
Metrics refer to quantitative measurements that provide insights into system’s performance, behavior,
and health. These metrics are typically collected and recorded over time to assist the developers,
operators, and other stakeholders understand the system’s current state, identify trends and detect
anomalies.

The roles of metrics in observability are crucial and they include:

 Performance monitoring: Metrics help to track the performance of various components of a

system such as CPU usage, memory consumption, network throughput, request latency, among
the others. By monitoring these metrics, teams can identify performance bottlenecks and
optimize resource allocation.
 Issue detection and troubleshooting: Occasionally unexpected behavior and errors occurs in a
system, metrics plays a crucial role in identifying the root cause. Deviations and anomalies in
metrics indicate potential issues that require investigation and resolution.
 Capacity planning and scalability: Metrics are used to forecast resource requirements and plan
for capacity upgrades to enable the system handle the increasing workloads. This ensures that
the system can scale effectively as the demand grows.
 Service Level Agreement (SLA) compliance: Metrics play a significant role in establishing and
measuring the performance of a system to ensure the compliance with the SLAs. They provide
quantitative evidence of the system’s performance against agreed-upon targets.
 Alerting and monitoring: Metrics are fundamental to setting up monitoring and alerting
systems. When specific metrics cross predefined thresholds or exhibits abnormal behavior,
alerts are triggered enabling the teams to respond promptly to potential issues.

Various types of metrics are used in observability, some of the commonly used include:

 Counters: These are metrics that continuously increase over time and represent a count of
specific events or occurrences. They track a number of times an event happens, such as the total
number of requests, errors, or messages processed.
 Gauges: They are metrics that represent a single value at a particular point in time. Unlike
counters, gauges can go up and down, therefore reflecting instantaneous measurements of a
 specific state. For example, gauges can monitor metrics such as current CPU usage, memory
utilization, or the number of active connections.
 Histograms: They observe the distribution of values over time. Histograms group data into
configurable ranges and track the frequency of data points falling into each range. They are
useful in analyzing the spread of values, identifying outliers and calculating percentiles.
Histograms commonly measures request latencies, response sizes and other continuous
variables.
 Timers: They are similar to histograms only that they specifically measure durations or latencies.
Timers are useful for monitoring the time taken to complete certain operations or tasks.

In observability, collecting metrics data plays fundamental aspect of monitoring and understanding the
behavior of a system or application. There exist a number of methods for collecting metrics data and the
choice of method is dependent on several factors such as the nature of the system, the scale of
infrastructure, and the monitoring tools in use. Some of the common methods include:

 Instrumentation libraries: They collect metrics directly from the application code. Developers
can add code snippets or use pre-built libraries that automatically gather and report relevant
metrics to the monitoring system.
 API endpoints: Some systems expose their API endpoints specifically for metrics collection.
Monitoring tools can make requests to these endpoints to fetch the current metric values.
 Agents and proxies: Dedicated agents and proxies are deployed alongside applications to collect
metrics. These agents extract data directly from the application memory, runtime, or network
interfaces. They then send the collected data to the central monitoring system.
 Exporters and integrations: Most systems and services provide exporters or integrations that
allow seamless integrations with monitoring platforms. The exporters then translate native
metrics from these systems into a format that monitoring solution can understand.
 Infrastructure monitoring: Monitoring agents or daemons are installed on the servers to collect
and report data to the central monitoring system. Monitoring tools such as Nagios or Zabbix use
this approach.

Next you will look at some of the scenarios of how metrics are useful in identifying and troubleshooting
various system issues:

 High CPU usage: If the CPU usage metrics shows a sustained high percentage, it indicates that
the system is under heavy load and there might be experiencing performance issues. The
possible reasons are inefficient code, resource contention, or unexpected surge in traffic.
 Increased error rate: A sudden increase in the error rate metric can signal issues with the
application stability or functionality. The team should investigate the error logs corresponding to
the spike in errors to identify the root cause and address the underlying issue.
 Memory leak: A gradual increase in memory utilization over time may suggest a memory leak in
the application. The team may need to monitor the memory usage trend and compare it to the
normal behavior.
  Network bottlenecks: If the network traffic metric indicates a surge in data transfer, it could
lead to network congestion and potential bottlenecks. The concerned stakeholders should
analyze the network throughput and identify any specific connections or services causing high
traffic.
 Load balancer performance: If the load balancer request count is not evenly distributed across
the backend servers, it could signal an issue with the load balancing algorithm or the backend
server health. The team should analyze the load balancing metrics and verify if all backend
servers are healthy and handling requests appropriately.

In summary, by monitoring a wide range of relevant metrics and analyzing their trends and patterns,
teams can gain valuable insights into system behavior and identify the issues proactively. This leads to
effective maintenance of system health and reliability.

Logs
Logs are a record of events, activities and messages generated by various components within the
software system. These events may include error messages, warning messages, informational messages
and other relevant data useful in monitoring and analyzing the system’s health and performance.

The primary role of logs in observability is to provide visibility into the system’s internal workings. Below
are some of the key roles of the logs in observability:

 Debugging and troubleshooting: Logs plays a crucial role when it comes to identifying and
diagnosing issues within the software system. When an error occurs or there is an unexpected
behavior, developers and system administrators may refer to logs to understand the sequence
events leading up to the issue hence pinpointing the root cause.
 Performance monitoring and optimization: Logs provide useful insights into the system’s
performance. This is possible through analyzing the performance related logs whereby teams
can identify the bottlenecks, inefficiencies, and areas of optimization.
 Auditing and compliance: In most cases, software systems are required to comply with the
specific regulations and standards. Logs serves as an audit trail, capturing various activities and
actions taken within the system, hence ensuring systems are compliant.
 Security and intrusion detection: Logs are significant when it comes to detecting and
investigating security incidences. Unusual and suspicious activities are identified by analyzing
the logs hence aiding in the detection of potential security breaches.
 Capacity planning and resource management: Observing patterns and trends in logs can assist
predict resource utilization and help in capacity planning hence ensuring that the system has
enough resources to handle workloads efficiently.

Logs are categorized in different type depending on the nature of the information they capture and in
which context they are used. The main types of logs in observability and how they can be used to
troubleshoot problems:

 Application logs: These are logs that are generated by a software application itself. They gather
information about the application’s internal state, actions performed, and event triggered
 during its execution. These logs include error messages, debug information, warning, user
interactions among others. They play a vital role to developers and operators as they assist
them to understand the application’s behavior, diagnose issues, and track down bugs.
 Server logs: These logs are generated by the underlying server infrastructure hosting the
application or service. The logs record server-related activities such as incoming requests, server
errors, resource usage, as well as server status. Server logs are vital for monitoring server health,
identifying performance bottlenecks, and optimizing resource utilization.
 Access logs: They capture information about incoming requests t a web server or API, including
details such as IP address of the client, the requested resource, the timestamp, the HTTP
method used, and the response status code. Access logs are useful for understanding traffic
patterns, identifying potential security threats, and analyzing the usage of the specific endpoints
or resources.
 Audit logs: These logs keep track of critical system action and configuration changes. Audit logs
provide an audit trail of administrative activities, system changes, and user actions that might
have implications for compliance and regulatory requirements.

To effectively leverage logs for observability, organizations should opt for centralized logging solutions
that aggregate logs from different sources, making them easily searchable and analyzable. Proper log
management and analysis enable teams to gain comprehensive view or their software system and
efficiently diagnose and resolve issues to ensure its reliability and performance.

In observability, collecting logs is crucial in gaining insights into behavior and performance of a software
system. Various methods and tools are employed to collect logs data effectively. Below are some of the
common logs collection methods:

 Logging libraries and frameworks: Most programming languages have logging libraries and
frameworks that developers use to instrument their applications and log relevant events. These
libraries allow developers to define log levels, log formats, and destination.
 Syslog: It refers to standard protocol that sends log messages to a central logging server. It is
widely supported by various operating systems and networking devices. Applications and
services can be configured to send their logs a syslog server hence facilitating centralized log
management.
 Log aggregators: These tools collect logs from various sources and centralize them in as single
location. Log aggregators handle large volume of logs, provide search capabilities, and other
visualization features for analysis.
 Container logging: For applications running in containers such as Docker, container logging
drivers or plugins capture logs generated within the containers and forward them to the host
system or a centralized log collector.
 Cloud-based logging services: Cloud providers such as Azure, Google Cloud and AWS provide
logging services that simplify log data collection in cloud environments.
It is better to note that logs data collection methods can be combined for more comprehensive
observability.

Traces
A trace is a sequence of events that occur as a request or transaction flows through a distributed
system. Each event within a trace represents a distinct operation or action that is part of the overall
request’s journey across different services or components. Traces are typically represented as a Directed
Acyclic Graph (DAG) or a tree-like structure, with each node representing an event and the edges
representing the causal relationship between events.

Traces are an essential component of distributed tracing, a technique that monitors and troubleshoots
complex systems, especially those built using microservices architecture.

Traces play a vital role in achieving observability in the following ways:

 Request flow visualization: Traces allow the developers to visualize the entire path of a request
or transaction through system. This helps in understanding the flow of operations and
identifying any bottlenecks or performance issues that may arise.
 Latency analysis: By recording the timestamps of each event in a trace, it is possible to measure
the time taken for each operation. This helps to pinpoint where the delays occur and identify
the root causes, thereby enabling the teams to optimize the system’s performance.
 Distributed context propagation: Traces include a unique identifier that ties together all the
events related to a single request or transaction. This identifier allows for distributed context
propagation, hence enabling tracking and correlation of logs, metrics and other telemetry data
related to the same request across different services.
 Performance optimization: Traces provides insights into the interactions between services.
Armed with this information, developers can optimize service-to-service communication ,
identify unnecessary dependencies, and make informed decisions ti improve overall system
performance.

Traces play a significant role in troubleshooting problems in complex distributed systems. Below are
some of the scenarios where traces can be used to identify and resolve issues:

 Error identification: Traces reveal error patterns across the services. If a specific request is
constantly throwing up errors, you can track the trace to find the initial source of the error. This
is useful in identifying the error root cause, whether it’s a misconfiguration, a bug or an issue
with third party dependencies.
 Service dependency issues: When a certain service is experiencing problems, traces can direct
you to other dependent services and the way they interact. If the issue lies with the
downstream service, the trace can highlight the point of failure and guide you to the problem’s
guide.
 Dropped requests: If you suspect that requests are getting lost somewhere in your system,
traces can help you trace the request’s path and identify where it goes missing. This information
is crucial for understanding network or communication issues.
  Concurrency and parallelism issues: Traces can help to reveal potential concurrency or
parallelism problems in your application. By inspecting the trace data, you can determine if
multiple requests are interfering with each other, leading to deadlocks , race conditions, or
contention problems.
 Excessive retry attempts: Some systems may implement automatic retries for failed requests.
Traces can highlight excessive retries attempts, indicating that a service might be experiencing
temporary issues. Investing the trace can help you determine whether the retries are due to
legitimate transient failures or indicate deeper problems that need to be addressed.

In summary traces provide detailed insights into behavior of a distributed system, making them
indispensable for troubleshooting problems. By analyzing traces, developers can identify performance
bottlenecks, trace errors, verify system behavior and resolve issues more efficiently.

Popular tools used in Observability

. Every organization is striving to improve their software systems. This is through gaining insights and
understanding the internal workings of their systems. To achieve this organizations are leveraging
observability tools in their complex systems. Several tools for observability exist in the market today as
heighted below:

 Solarwinds Observability: It is an integrated, full-stack observability solution built to connect

data from web applications, their services, cloud and hybrid infrastructure including Kubernetes,
AWS, and Azure, databases, as well as end user experience. They deliver a holistic business
insights, operational intelligence and smart automation to help solve complex business
problems. The tool simplifies business processes, optimize DevOps teams performance and
increase the reliability of the business critical systems.
 Prometheus: It is an open source monitoring and alerting toolkit that specializes in collecting
time-series data and storing it in a time-series databases. Prometheus provide a powerful
querying language and integrates well with Grafana for visualization.
 Grafana: It is an open source platform used in data visualization and monitoring. Grafana can
connect to various data sources including Prometheus and provides flexible and interactive
dashboards to display metrics and logs.
 Datadog: It’s a cloud based observability platform that offers monitoring, logging, and tracing
capabilities. It supports various integrations and provide a unified view of your infrastructure
and applications.
 Sysdig: It’s a cloud-native intelligence platform that provides monitoring, security and
troubleshooting for containerized environments and microservices.

The landscape of observability tools is constantly evolving, and new tools and features are added. It’s
essential to stay up-to-date with the latest developments in the field to choose the most suitable tool
for your specific needs. SolarWinds Observability integrates most of the tools needed for observability
and incorporates all the three pillars. Furthermore it has been built for DevOps, why not try their free
trial today?
Wrapping up
In conclusion, observability is an indispensable aspect of modern system design and maintenance. It
empowers teams to detect, diagnose, and respond to issues effectively, leading to a more reliable,
performant, and secure systems. As technology continues to evolve, the importance of observability will
only grow, shaping the way we build and manage complex systems in the future.