Modern Distributed Systems Architecture — Principal Engineer Reference

Modern Distributed
Systems Architecture
A Principal Engineer's Complete Reference

From Browser Request to Microservice Response

Covering: CDN · Load Balancing · API Gateway · Service Mesh · Kubernetes

Pod Networking · Secrets Management · mTLS · Authentication · Observability

Page 1 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 1: The Anatomy of a Web Request — End-

to-End Overview
Before we dive into the internals of every layer, it is essential to build a mental model of the
complete journey a request takes—from the moment a user presses Enter in a browser to the time
a rendered response is displayed on screen. This chapter gives you the 30,000-foot view that the
remainder of this document fills in with extreme depth.

1.1 The Big Picture

A modern web application request traverses multiple distinct logical and physical layers. Each layer
adds value—caching, security enforcement, routing, protocol translation, load distribution, and
business logic execution. Understanding all of these layers, and crucially how they interact, is what
separates a senior engineer from a principal engineer.

1.1.1 The Twelve Layers of a Modern Request

1. DNS Resolution — translating the human-readable hostname to an IP address.
2. TLS Handshake — establishing an encrypted channel between client and first hop.
3. CDN Edge Node — serving cached assets, absorbing DDoS, and performing edge logic.
4. Origin Shield / CDN Middle Tier — second-tier caching before traffic hits the origin.
5. Load Balancer (L4 / L7) — distributing connections across upstream groups.
6. API Gateway — authenticating, rate-limiting, transforming, and routing requests.
7. Service Mesh Ingress / Sidecar Proxy — enforcing mTLS, observability, and policy.
8. Kubernetes Service — virtual IP and kube-proxy / eBPF rules to reach a pod.
9. Target Microservice Pod — executing business logic.
10. Inter-service Communication — gRPC, REST, or message queues to downstream services.
11. Data Layer — databases, caches, object stores.
12. Response Path — the reverse journey, including streaming and compression.

1.2 Why Each Layer Exists

Each layer was introduced to solve a specific class of problem. Understanding the motivation for
each layer prevents over-engineering and helps you choose the right tool for the right job.
Term / Component Description / Detail
DNS Decouples human-readable names from IP addresses; enables
failover, geo-routing, and blue/green traffic splits at the DNS level.
TLS at Edge Offloads cryptographic computation from application servers;
centralises certificate management; enables HTTP/2 and HTTP/3
multiplexing.

Page 2 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Term / Component Description / Detail

CDN Reduces latency by serving content from the nearest Point of Presence
(PoP); absorbs bandwidth attacks; lowers origin compute costs.
Load Balancer Eliminates single points of failure; enables horizontal scaling; provides
health-checking and graceful drain during deploys.
API Gateway Centralises cross-cutting concerns: authn/authz, rate limiting,
request/response transformation, API versioning, and observability.
Service Mesh Moves network reliability concerns (retries, timeouts, circuit breaking,
mTLS) out of application code and into infrastructure.
Kubernetes Declarative container orchestration: scheduling, bin-packing, self-
healing, rolling updates, secrets injection, and service discovery.

1.3 Latency Budget

As a principal engineer, you must reason about latency budgets. Below is a representative
breakdown for a p99 request in a well-architected system:
Term / Component Description / Detail
DNS lookup 0–50ms (cached after first request; TTL-controlled)
TLS 1.3 handshake (CDN 10–30ms (0ms with session resumption)
edge)
CDN edge processing 1–5ms
WAN transit to origin 10–150ms (depends on geography)
Load balancer 0.1–1ms
API Gateway processing 2–10ms (auth token validation, rate limit check)
Service mesh sidecar 0.2–1ms
(ingress)
Application processing 10–200ms (business logic, DB queries)
Inter-service calls (x2) 5–50ms each
Service mesh sidecar 0.2–1ms
(egress)
Response path (reverse) ~same as forward path

Total budget for a well-designed e-commerce checkout: 100–500ms. Everything over 500ms
requires investigation. Everything over 2s is a user experience failure.

Page 3 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 2: DNS Resolution — The First Step

Every request begins with DNS. DNS is far more than a phonebook; in modern architectures it is
the first tier of traffic management, failover, and geo-routing.

2.1 The DNS Resolution Chain

When a user types [Link] in their browser, the following chain of queries begins:
13. Browser DNS cache — checked first. TTL-controlled entries survive tab refreshes.
14. Operating System DNS cache — the OS resolver checks its own cache.
15. OS stub resolver — reads /etc/[Link] (Linux) or system preferences (macOS/Windows)
to find the recursive resolver.
16. Recursive resolver (ISP or [Link]) — performs the full resolution on behalf of the client if the
answer is not cached.
17. Root name servers — the resolver queries one of the 13 root server clusters to find the TLD
server.
18. TLD name servers — .com TLD servers return the authoritative name server for
[Link].
19. Authoritative name server — returns the final A/AAAA record (or CNAME chain).

2.2 DNS Record Types Used in Production

Term / Component Description / Detail
A Maps a hostname to an IPv4 address. Multiple A records = round-robin
DNS load balancing.
AAAA Maps a hostname to an IPv6 address.
CNAME Canonical name alias. CDNs use CNAMEs extensively:
[Link] → [Link].
ALIAS / ANAME A non-standard record that behaves like CNAME at the zone apex (@
record). Used by Route 53, Cloudflare.
NS Delegates a zone to authoritative name servers.
MX Mail exchanger records.
TXT Arbitrary text: SPF, DKIM, DMARC, domain ownership verification,
Let's Encrypt ACME challenges.
SRV Service location records: host, port, weight, priority. Used by
Kubernetes for headless service DNS.
PTR Reverse DNS (IP → hostname). Required by some security tools and
email deliverability.

Page 4 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Term / Component Description / Detail

CAA Certificate Authority Authorization — restricts which CAs can issue
certificates for a domain.

2.3 DNS TTL Strategy

TTL (Time To Live) is the number of seconds resolvers are permitted to cache a record. This
directly impacts:
• Failover speed — lower TTL = faster failover, but more DNS queries (cost and latency).
• Cache pollution — stale records in recursive resolvers can cause partial outages.
• Deployment agility — DNS-based traffic shifts require TTL to be low before the shift.

Best practices for TTLs:

Term / Component Description / Detail
Normal production records 300–3600 seconds (5–60 minutes)
Before a planned migration Lower TTL to 60s at least 2x the current TTL before switching
CDN CNAME records 300s (the CDN handles the sub-record TTL internally)
Health-checked failover 30–60s to minimise failover blast radius
records

2.4 DNS-Based Traffic Management

2.4.1 Weighted Routing
Route 53, Cloudflare Load Balancing, and other DNS providers support returning different records
with different probabilities. Example use case: send 10% of traffic to a canary endpoint during a
blue/green deployment.

2.4.2 Latency-Based Routing

The DNS resolver is queried from multiple PoPs; the provider returns the record pointing to the
region with the lowest measured latency to the client's resolver. AWS Route 53 uses EIP-to-region
latency measurements for this.

2.4.3 Geo-Routing
Records are returned based on the geographic location of the DNS resolver. Used for compliance
(data sovereignty), language-specific endpoints, and performance. Note: geo-routing is based on
resolver IP, not client IP—VPN users and corporate proxies can subvert it.

Page 5 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

2.4.4 Health-Checked Failover

DNS providers continuously health-check endpoints. When an endpoint fails its health check, its
DNS record is automatically withdrawn. Clients see the failover record instead. This is a coarse but
effective disaster recovery mechanism.

2.5 DNS in Kubernetes

Kubernetes has its own internal DNS system, typically implemented by CoreDNS. Every Service
gets a DNS entry of the form:
<service-name>.<namespace>.[Link]
For a headless service (clusterIP: None), DNS returns individual pod IPs (A records) instead of the
service VIP. This is used by StatefulSets for stable pod identity.

2.5.1 CoreDNS Architecture

CoreDNS runs as a Deployment in the kube-system namespace. The kubelet configures each pod's
/etc/[Link] to point to the CoreDNS ClusterIP. CoreDNS reads the Kubernetes API to build its
zone file dynamically.
CoreDNS plugins (in order of execution):
• errors — logs errors to stderr.
• health — exposes /health HTTP endpoint.
• ready — exposes /ready HTTP endpoint for readiness probes.
• kubernetes — serves Kubernetes cluster DNS records.
• prometheus — exposes metrics at :9153/metrics.
• forward — forwards non-cluster queries to upstream resolvers.
• cache — caches DNS responses; default TTL 30s.
• loop — detects forwarding loops.
• reload — watches Corefile for changes.
• loadbalance — round-robins A record responses.

2.5.2 DNS Search Domains

Kubernetes configures pods with search domains. When a pod queries 'payments', the resolver
tries:
[Link]
[Link]
[Link]
payments (absolute)

Page 6 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

This means short names work within a namespace, but cross-namespace calls must use the full
FQDN to avoid ambiguity and unnecessary DNS lookups. This is a common source of latency and
bugs in microservice architectures.

Page 7 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 3: TLS — Encrypting the Channel

Transport Layer Security is the foundation of trust on the internet. Understanding TLS in depth is
mandatory for a principal engineer because misconfigurations can lead to data breaches, downtime
during certificate expiry, or performance regressions from suboptimal cipher negotiation.

3.1 TLS 1.3 Handshake — Step by Step

TLS 1.3 (RFC 8446) reduced the handshake from 2 round-trips to 1, and eliminated broken cipher
suites from 1.2.
20. Client Hello — client sends: TLS version support, a random nonce, a list of supported cipher
suites, key_share extension (Diffie-Hellman public key), and SNI (Server Name Indication).
21. Server Hello — server responds with: chosen cipher suite, its own DH public key, and
begins sending encrypted extensions immediately.
22. Both sides compute the session keys — using ECDH key exchange, both sides
independently derive the same symmetric key without ever transmitting it.
23. Server sends Certificate, CertificateVerify, and Finished — the client verifies the server's
certificate chain against trusted CAs.
24. Client sends Finished — handshake complete. All subsequent data is encrypted with the
negotiated symmetric key.

Principal Engineer Note: With TLS 1.3, you can send application data in the first round-trip after
resumption using 0-RTT (Zero Round-Trip Time Resumption). However, 0-RTT data is replay-
vulnerable and should NEVER be used for non-idempotent operations like payments or state
changes.

3.2 TLS Termination Strategies

3.2.1 Edge Termination
The CDN or load balancer terminates TLS. Traffic between the CDN/LB and your origin may be
plain HTTP (for internal traffic on a trusted network) or re-encrypted with a separate TLS session.
This is the most common pattern.

3.2.2 Passthrough
The L4 load balancer forwards TLS bytes without decryption. TLS is terminated at the application
server. Limits L7 routing capabilities but useful when end-to-end encryption is required for
compliance (e.g., PCI-DSS, HIPAA).

3.2.3 Re-encryption

Page 8 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

TLS is terminated at the edge, inspected/transformed, then a new TLS session is established to the
origin. Provides full L7 visibility plus end-to-end encryption. This is the model used by service
meshes with mTLS.

3.3 Certificate Lifecycle Management

Certificate expiry is one of the most common causes of production outages. A principal engineer
should ensure cert management is fully automated.

3.3.1 Certificate Types

Term / Component Description / Detail
Domain Validated (DV) Proves domain control only. Issued in seconds by ACME CAs (Let's
Encrypt). Used for most web services.
Organization Validated (OV) CA verifies the legal entity. Appears in the certificate Subject. Used by
enterprises.
Extended Validation (EV) Highest manual verification. Browsers no longer show the green bar,
so EV adds little user value today.
Wildcard *.[Link] covers all single-level subdomains. Cannot cover sub-
subdomains (*.[Link] requires a separate cert).
Subject Alternative Name A single cert covering multiple FQDNs. Preferred over wildcard for
(SAN) security (limits blast radius of key compromise).
Internal / Private CA Signed by your own CA. Used for internal service-to-service mTLS.
Must be distributed to all clients that need to trust it.

3.3.2 ACME Protocol and Let's Encrypt

The ACME protocol (RFC 8555) automates the issuance and renewal of DV certificates. The CA
proves domain control via one of three challenge types:
• HTTP-01 — CA fetches a specific token from [Link]
challenge/<token>. Requires the domain to resolve to a server you control on port 80.
• DNS-01 — CA checks for a TXT record at _acme-challenge.<domain>. Works for wildcard
certs and internal services not accessible from the internet.
• TLS-ALPN-01 — CA connects to port 443 using a special ALPN protocol. Requires control of
the TLS stack.

3.3.3 Cert-Manager in Kubernetes

cert-manager is the de facto standard for certificate management in Kubernetes. It watches
Certificate custom resources and automatically provisions, renews, and stores certificates as
Kubernetes Secrets.
apiVersion: [Link]/v1
kind: Certificate

Page 9 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

metadata:
name: api-tls-cert
namespace: production
spec:
secretName: api-tls-secret
duration: 2160h # 90 days
renewBefore: 360h # Renew 15 days before expiry
dnsNames:
- [Link]
- [Link]
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer

Page 10 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 4: Content Delivery Networks (CDN)

A CDN is a globally distributed network of servers (Points of Presence, or PoPs) that cache and
serve content from locations geographically close to end users. CDNs are not just caches—they are
a comprehensive edge computing, security, and traffic management platform.

4.1 How a CDN Request Works

When a user requests [Link]
25. The domain resolves (via Anycast DNS) to the IP address of the nearest CDN PoP.
26. The browser establishes a TLS connection to the PoP. The PoP holds the TLS certificate
and terminates the connection locally—this alone saves 50–150ms of WAN round-trip latency.
27. The CDN edge node checks its cache for the requested object using the cache key (typically
URL + selected headers).
28. Cache HIT — the response is returned directly from edge memory or SSD. Time to first byte
(TTFB) is sub-10ms.
29. Cache MISS — the edge node forwards the request to the next tier (origin shield or origin).
The edge waits for the response, caches it per the cache-control headers, and returns it to the
client. Subsequent requests for the same object will hit the cache.

4.2 CDN Caching Semantics

4.2.1 Cache-Control Headers
The origin server controls CDN caching behaviour through HTTP response headers:
Term / Component Description / Detail
Cache-Control: public, max- Cache for 24 hours at both CDN and browser.
age=86400
Cache-Control: s- Cache at CDN for 1 hour; do not cache in browser.
maxage=3600, max-age=0
Cache-Control: no-store Do not cache anywhere. Used for sensitive personalised data.
Cache-Control: no-cache Must revalidate with origin on every request (but can use conditional
GET).
Surrogate-Control: max- CDN-specific header (Varnish, Fastly). Takes precedence over Cache-
age=7200 Control for CDN layer; stripped before sending to client.
Vary: Accept-Encoding, CDN must maintain separate cache entries per unique header
Accept-Language combination.

4.2.2 Cache Keys

Page 11 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

The cache key determines when two requests are considered identical. The default key is the full
URL. CDNs allow you to customise the cache key:
• Strip query parameters — cache /products?ref=email the same as /products.
• Include headers — cache separate copies for mobile vs desktop User-Agent.
• Include cookies — dangerous; rarely appropriate for public content; can cause cache
poisoning.

4.2.3 Cache Invalidation

The two hardest problems in computer science are cache invalidation and naming things. CDN
cache invalidation strategies:
• URL purge — invalidate a specific URL. Fast but requires knowing every cached URL variant.
• Tag-based purge (Surrogate-Key / Cache-Tag) — origin tags responses with logical tags
(e.g., 'product-123'). A single purge call invalidates all responses with that tag. Supported by
Fastly, Cloudflare, Varnish.
• Prefix purge — invalidate all URLs under a path prefix. Blunt but useful for deployments.
• Full cache flush — nuclear option. Avoid in production; causes thundering herd at origin.

4.3 CDN Security Features

4.3.1 DDoS Protection
CDNs absorb volumetric attacks at the edge, far from your origin. Mechanisms include:
• Anycast absorption — attack traffic is distributed across all PoPs, diluting the impact.
• Bandwidth scrubbing — traffic is passed through scrubbing centres that filter malicious
packets.
• Rate limiting at edge — IP-level rate limiting before traffic reaches the origin.
• Challenge pages — CAPTCHA or JavaScript challenge pages for suspicious IPs.

4.3.2 Web Application Firewall (WAF)

A WAF deployed at the CDN edge inspects HTTP requests and blocks known attack patterns
before they reach your application. Key capabilities:
• OWASP Top 10 rule sets — SQL injection, XSS, command injection, path traversal.
• Custom rules — block requests from specific countries, ASNs, or matching custom patterns.
• Bot management — distinguish legitimate bots (Googlebot) from scrapers and credential
stuffing bots.
• Rate limiting rules — enforce per-IP, per-path, or per-user rate limits.

4.3.3 TLS Configuration at Edge

CDNs centralise TLS configuration. Principal engineer concerns:

Page 12 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

• Minimum TLS version — enforce TLS 1.2 minimum; prefer TLS 1.3 only for modern clients.
• Cipher suite selection — disable weak ciphers (RC4, DES, 3DES). Prefer ECDHE for forward
secrecy.
• HSTS (HTTP Strict Transport Security) — instruct browsers to always use HTTPS. Include
subdomains and preload for maximum coverage.
• Certificate pinning — avoid in CDN scenarios; use CAA records instead to restrict issuers.

4.4 Edge Computing and Edge Workers

Modern CDNs support running code at the edge via V8-based runtimes (Cloudflare Workers) or
Wasm modules (Fastly Compute). Use cases:
• A/B testing and feature flags — vary content without an origin round-trip.
• Authentication at edge — validate JWTs before requests reach the origin, blocking
unauthorised traffic at the cheapest possible point.
• Request/response transformation — add security headers, rewrite URLs, inject
personalisation tokens.
• Geographic compliance — block requests from sanctioned countries or redirect to region-
specific origins.
• Bot detection — run ML inference at edge to classify traffic.

Page 13 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 5: Load Balancers — L4 and L7

A load balancer is the workhorse of horizontal scaling. It distributes incoming requests across a
pool of backend servers, performs health checking, and enables zero-downtime deployments.
Understanding the difference between L4 and L7 load balancing is fundamental.

5.1 L4 vs L7 Load Balancing

Term / Component Description / Detail
Layer L4 (Transport)
OSI Layer 4 (TCP/UDP)
Visibility IP, port, TCP state
Routing basis Source IP, port
Latency overhead < 0.1ms
SSL termination Pass-through only
Session stickiness IP-hash
WebSocket support Yes
Typical use High-throughput TCP, UDP

5.2 Load Balancing Algorithms

5.2.1 Round Robin
Requests are distributed sequentially across backend pool members. Simple and effective when all
backends are homogeneous. Does not account for request duration or backend capacity.

5.2.2 Least Connections

Each new request is sent to the backend with the fewest active connections. Better than round-
robin for workloads with variable request durations (e.g., long-polling, file uploads).

5.2.3 Weighted Round Robin / Weighted Least Connections

Backends are assigned weights proportional to their capacity. Newer or smaller instances receive
fewer requests. Critical for canary deployments.

5.2.4 IP Hash / Consistent Hash

The client's IP (or a specified header/cookie value) is hashed to a consistent backend. Provides
session stickiness without requiring shared session state. Consistent hashing (as used in Nginx,
Envoy, HAProxy) minimises disruption when backends are added or removed.

Page 14 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

5.2.5 Least Request (Power of Two Choices)

Used by Envoy and modern service meshes. Two backends are selected at random; the request
goes to whichever has fewer active requests. Provides near-optimal load distribution with O(1)
selection complexity.

5.2.6 Random
Each request goes to a randomly selected backend. Surprisingly effective for large pools (law of
large numbers). Used internally by some service meshes for simplicity.

5.3 Health Checking

Health checking is the mechanism by which a load balancer removes unhealthy backends from the
rotation. Three types:

5.3.1 TCP Health Check

The LB attempts a TCP connection to the backend port. If the connection succeeds, the backend is
considered healthy. Cheap but does not verify application-level health.

5.3.2 HTTP Health Check

The LB sends an HTTP GET request to a configured path (e.g., /healthz or /ready) and expects a
2xx response within a timeout. This verifies the application is running and its dependencies are
accessible.

5.3.3 gRPC Health Check

Uses the gRPC Health Checking Protocol ([Link]/Check). Returns NOT_SERVING,
SERVING, or UNKNOWN. Supported natively by Envoy and Kubernetes liveness/readiness
probes.

Best Practice: Separate /healthz (liveness — is the process running?) from /readyz (readiness — can
this pod serve traffic?). The LB should only route to pods that return 200 on /readyz. During startup
or graceful shutdown, /readyz should return 503 even if the process is alive.

5.4 Connection Draining and Graceful Shutdown

When a backend is removed from the pool (during a deployment or scale-down), connection
draining ensures in-flight requests complete before the backend is terminated:
30. The LB stops sending new connections to the draining backend.
31. Existing connections are allowed to complete, up to a configured timeout (typically 30–60
seconds).
32. After the timeout, remaining connections are forcibly terminated.
33. The backend is removed from the pool.

Page 15 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

In Kubernetes, this maps to the terminationGracePeriodSeconds field on a Pod spec and the
preStop lifecycle hook.

5.5 AWS ALB / NLB Deep Dive

5.5.1 Application Load Balancer (ALB)
AWS ALB is an L7 load balancer. Key features:
• Content-based routing — route based on path, host header, HTTP method, query string,
source IP.
• Target groups — backends can be EC2 instances, IP addresses, Lambda functions, or ECS
tasks.
• Listener rules — ordered rule evaluation with default catch-all rule.
• WebSocket and HTTP/2 support.
• Sticky sessions via load balancer-generated cookie (AWSALB) or application cookie.
• WAF integration — attach AWS WAF web ACLs directly.
• Access logs — S3 delivery of per-request logs including client IP, backend IP, latency, SSL
details.

5.5.2 Network Load Balancer (NLB)

AWS NLB operates at L4. It preserves the client source IP (unlike ALB which uses X-Forwarded-
For). Key features:
• Millions of requests per second — ultra-low latency, designed for extreme throughput.
• Static IP addresses — one static Elastic IP per AZ. Allows whitelisting of IPs in firewalls.
• TLS termination — NLB can terminate TLS while preserving the source IP.
• Zonal DNS — each AZ has its own DNS name, allowing AZ-affinity for inter-service calls.

Page 16 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 6: API Gateway — The Front Door

The API Gateway is the single entry point for all client requests to your microservices backend. It
implements cross-cutting concerns that would otherwise need to be duplicated in every service:
authentication, rate limiting, request routing, protocol translation, and observability.

6.1 Core Responsibilities of an API Gateway

Term / Component Description / Detail
Authentication & Validate JWTs, API keys, OAuth tokens, OIDC sessions. Enforce role-
Authorization based access control (RBAC).
Rate Limiting & Throttling Protect backends from traffic spikes. Per-client, per-API, per-IP rate
limits using token bucket or leaky bucket algorithms.
Request Routing Map external API paths to internal service endpoints. Support
versioning (/v1/, /v2/).
Protocol Translation Translate REST to gRPC, GraphQL to REST, HTTP/1.1 to HTTP/2.
Request Transformation Add/modify/remove headers; transform request/response bodies;
enrich with context from auth.
SSL/TLS Termination Terminate TLS; optionally re-encrypt to backend.
Observability Emit metrics, traces, and structured logs for every request.
Circuit Breaking Fail fast when backend is unhealthy rather than queuing requests
indefinitely.
Caching Cache idempotent responses at the gateway layer to reduce backend
load.
API Composition Fan out a single client request to multiple backend services and merge
responses.

6.2 Authentication at the API Gateway

6.2.1 JWT (JSON Web Token) Validation
JWT is the most common mechanism for stateless authentication in microservices. A JWT consists
of three base64url-encoded parts separated by dots:
[Link]
Header: { alg: 'RS256', typ: 'JWT', kid: 'key-id-2024' }
Payload: { sub: 'user123', iss: '[Link] aud: '[Link]', exp: 1735689600,
iat: 1735686000, roles: ['admin'] }
Signature: RSA-SHA256(base64url(header) + '.' + base64url(payload), privateKey)

Page 17 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

The API gateway validates JWTs by:

34. Extracting the JWT from the Authorization: Bearer <token> header.
35. Decoding the header to extract the key ID (kid).
36. Fetching the corresponding public key from the JWKS endpoint
([Link] This is cached.
37. Verifying the signature using the public key.
38. Checking standard claims: exp (not expired), iss (correct issuer), aud (correct audience), nbf
(not before).
39. Extracting claims (user ID, roles, scopes) and forwarding them to the backend as HTTP
headers.

Security Note: Never trust JWTs forwarded from one internal service to another without re-validation.
Use a dedicated service account token (SPIFFE/SPIRE) for service-to-service calls, not user JWTs.

6.2.2 API Key Authentication

API keys are long, random, opaque strings used to identify API consumers (other services, third-
party integrations). The gateway validates the key against a key store (Redis, database), enforces
per-key rate limits, and injects the consumer identity into request headers.

6.2.3 OAuth 2.0 and OIDC

The OAuth 2.0 framework and OpenID Connect (OIDC) are used for delegated authorization and
federated identity. The API gateway acts as a resource server:
• Token introspection — calls the authorization server's /introspect endpoint to validate opaque
tokens. Synchronous; adds latency.
• JWT verification — validates signed JWTs locally. Stateless; fast; requires public key
distribution.
• OIDC session — for browser-based clients, the gateway manages the OIDC code flow and
sets a session cookie.

6.3 Rate Limiting Algorithms

6.3.1 Token Bucket
A bucket with capacity N is refilled at rate R tokens per second. Each request consumes one token.
When the bucket is empty, requests are rejected (429 Too Many Requests). Allows burst traffic up
to the bucket capacity. This is the most common algorithm.

6.3.2 Leaky Bucket

Requests enter a queue (the bucket) and are processed at a constant rate. Excess requests
overflow and are rejected. Provides a smooth output rate regardless of input burst. Better for
protecting backends with limited concurrency.

Page 18 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

6.3.3 Fixed Window Counter

Count requests within a fixed time window (e.g., per minute). Reset the counter at window
boundaries. Simple but susceptible to burst attacks at window boundaries (double the limit in a
short period spanning two windows).

6.3.4 Sliding Window Log / Sliding Window Counter

Maintains a log of request timestamps or uses a weighted combination of the current and previous
window counters. More accurate than fixed window; eliminates the boundary burst problem.

6.4 Popular API Gateway Solutions

Term / Component Description / Detail
Kong Nginx-based, Lua plugins, extensive plugin ecosystem. Can run on
Kubernetes with Helm. Supports DB-mode and DB-less (declarative)
configuration.
AWS API Gateway Fully managed. REST API, HTTP API (cheaper, less features), and
WebSocket modes. Native integration with Lambda, ALB, NLB, ECS,
AppSync.
Envoy + Contour / Emissary Envoy proxy as the data plane; Contour or Emissary as control plane.
Deeply integrated with Kubernetes via CRDs. Foundation of many
service meshes.
Traefik Go-based, cloud-native. Auto-discovers Kubernetes services via
labels/annotations. Excellent for developer experience.
NGINX Plus The OG. Mature, battle-tested. Rich configuration language. Suitable
when you need maximum control over routing logic.
Apigee (Google Cloud) Enterprise-grade API management. Analytics, developer portal,
monetisation, advanced security policies.
Azure API Management Azure-native. Policy-based configuration using XML. Deep Azure AD
integration.

Page 19 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 7: Service Mesh — The Infrastructure

Nervous System
A service mesh is a dedicated infrastructure layer for handling service-to-service communication. It
externalises cross-cutting network concerns from application code into a sidecar proxy (data plane)
controlled by a centralised control plane.

7.1 Why Service Meshes Exist

Without a service mesh, every microservice must implement its own:
• mTLS for secure service-to-service communication
• Retry logic with exponential backoff
• Circuit breaking and bulkhead patterns
• Distributed tracing header propagation
• Metrics collection
• Traffic shifting for canary deployments
This logic is hard to implement consistently across 50+ services written in Go, Java, Python, and
[Link]. A service mesh solves this by moving it to the infrastructure layer, making it language-
agnostic and centrally configurable.

7.2 Service Mesh Architecture

7.2.1 Data Plane
The data plane consists of sidecar proxies deployed alongside each pod. The most common
sidecar is Envoy Proxy (written in C++). Envoy intercepts all inbound and outbound traffic from the
application container by manipulating iptables rules in the pod's network namespace.
Envoy capabilities as a sidecar:
• L3/L4 proxying — handles raw TCP, UDP, gRPC.
• L7 routing — routes based on URL path, method, headers.
• Load balancing — least request, round robin, consistent hash.
• Circuit breaking — eject unhealthy endpoints from the load balancing pool.
• Retries — retry on specific conditions (connection failure, 5xx, reset).
• Timeouts — enforce per-route and per-cluster timeouts.
• mTLS — terminate and initiate TLS with client certificates; verify SPIFFE IDs.
• Distributed tracing — generate and propagate trace spans (Zipkin, Jaeger, OpenTelemetry).
• Metrics — emit per-request metrics: latency, request count, error rate, upstream success rate.

7.2.2 Control Plane

Page 20 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

The control plane configures the data plane. It does not handle traffic directly. In Istio, the control
plane is a single binary called istiod that implements:
• Pilot — converts Istio custom resources (VirtualService, DestinationRule, etc.) into Envoy xDS
configuration and pushes it to sidecars.
• Citadel (now part of istiod) — issues and rotates SPIFFE/X.509 certificates for each workload
identity. Acts as the mesh PKI.
• Galley (deprecated) — configuration validation and distribution.

7.3 Istio Traffic Management

7.3.1 VirtualService
A VirtualService defines routing rules for traffic destined for a service. It is evaluated by the sidecar
proxy of the calling service.
apiVersion: [Link]/v1beta1
kind: VirtualService
metadata:
name: payments-vs
spec:
hosts:
- [Link]
http:
- match:
- headers:
x-canary:
exact: 'true'
route:
- destination:
host: [Link]
subset: v2
weight: 100
- route:
- destination:
host: [Link]
subset: v1
weight: 90
- destination:
host: [Link]
subset: v2
weight: 10

7.3.2 DestinationRule

Page 21 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

A DestinationRule defines policies applied to traffic after routing has been determined. It defines
subsets (used by VirtualService) and configures connection pool and circuit breaker settings.
apiVersion: [Link]/v1beta1
kind: DestinationRule
metadata:
name: payments-dr
spec:
host: [Link]
trafficPolicy:
connectionPool:
tcp:
maxConnections: 100
http:
h2UpgradePolicy: UPGRADE
http2MaxRequests: 1000
httpMaxPendingRequests: 100
outlierDetection:
consecutive5xxErrors: 5
interval: 30s
baseEjectionTime: 60s
maxEjectionPercent: 50
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2

7.4 Mutual TLS (mTLS) in the Service Mesh

mTLS extends regular TLS by requiring both parties (client and server) to present certificates. This
enables cryptographic identity verification—any service calling the payments service must prove it
holds a valid certificate issued by the mesh CA.

7.4.1 SPIFFE and SPIRE

SPIFFE (Secure Production Identity Framework for Everyone) is a set of open standards for
workload identity. A SPIFFE ID takes the form:
spiffe://[Link]/ns/production/sa/order-service
This encodes the trust domain, namespace, and service account. The ID is embedded in the X.509
Subject Alternative Name (SAN) field of the workload certificate. When mTLS is established, each
side extracts the SPIFFE ID from the peer's certificate and enforces authorization policy.

Page 22 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

7.4.2 PeerAuthentication and AuthorizationPolicy

# Enforce mTLS cluster-wide
apiVersion: [Link]/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: istio-system
spec:
mtls:
mode: STRICT

# Only allow order-service to call payments

apiVersion: [Link]/v1beta1
kind: AuthorizationPolicy
metadata:
name: payments-authz
namespace: production
spec:
selector:
matchLabels:
app: payments
rules:
- from:
- source:
principals:
- [Link]/ns/production/sa/order-service
to:
- operation:
methods: ["POST"]
paths: ["/api/v1/charge"]

Page 23 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 8: Kubernetes — Container Orchestration

in Depth
Kubernetes (K8s) is the de facto standard for container orchestration. It provides the declarative
API and control loops needed to run, scale, and update containerised workloads reliably. This
chapter goes deep into every component a principal engineer must understand.

8.1 Kubernetes Architecture

8.1.1 Control Plane Components
Term / Component Description / Detail
kube-apiserver The front door to the Kubernetes API. All components communicate
through it. Validates and processes REST requests, writes to etcd.
Horizontally scalable.
etcd A distributed, consistent key-value store. The sole source of truth for all
cluster state. Uses the Raft consensus algorithm. Must be backed up
regularly. 3 or 5 members for HA.
kube-scheduler Watches for unscheduled pods and assigns them to nodes based on
resource requests, affinity rules, taints/tolerations, and topology spread
constraints.
kube-controller-manager Runs all core controllers: Deployment, ReplicaSet, StatefulSet,
DaemonSet, Job, CronJob, Endpoints, Namespace, ServiceAccount,
Node lifecycle.
cloud-controller-manager Integrates with cloud provider APIs: provisions LoadBalancers,
PersistentVolumes, updates node topology labels.

8.1.2 Node Components

Term / Component Description / Detail
kubelet The primary node agent. Watches the API server for pods scheduled to
its node. Manages container lifecycle via the CRI (Container Runtime
Interface). Reports node and pod status back to API server.
kube-proxy Implements Kubernetes Service networking using iptables rules (or
IPVS, or eBPF via Cilium). Maintains NAT rules that map Service
ClusterIPs to pod IPs.
Container Runtime OCI-compliant runtime that actually runs containers. containerd and
CRI-O are the dominant runtimes. Docker is no longer supported as a
direct runtime.
CNI Plugin Container Network Interface plugin. Assigns IP addresses to pods and
configures the network namespace. Examples: Flannel, Calico, Cilium,
Weave.

Page 24 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

8.2 The Pod Lifecycle

A pod is the smallest deployable unit in Kubernetes. Understanding the full pod lifecycle is critical
for debugging deployment failures, startup slowness, and graceful shutdown issues.

8.2.1 Pod Phases

Term / Component Description / Detail
Pending Pod has been accepted by the API server. Waiting for: image pull,
node assignment, PVC binding, Init Container completion.
Running At least one container is running. Not all containers may be Ready.
Succeeded All containers have terminated with exit code 0. Used for Jobs.
Failed At least one container terminated with non-zero exit code.
Unknown Node cannot report pod status (network partition between kubelet and
API server).

8.2.2 Pod Creation Flow (Detailed)

40. kubectl apply / client calls POST /api/v1/namespaces/production/pods
41. kube-apiserver authenticates and authorizes the request (RBAC), runs admission controllers
(PodSecurity, OPA/Gatekeeper, MutatingWebhook).
42. API server validates the pod spec against the schema and writes the pod object to etcd with
[Link]=Pending.
43. kube-scheduler watches for pods with [Link] empty. Filters nodes (resource fit,
affinity, taints), scores remaining nodes, assigns the pod to the highest-scoring node by
writing [Link].
44. The kubelet on the assigned node watches the API server (via a list-watch) and sees the
new pod assignment.
45. Kubelet calls the CRI (containerd) to pull images and create containers. It first runs Init
Containers sequentially.
46. After Init Containers complete, kubelet starts the main containers.
47. Kubelet executes postStart lifecycle hooks if defined.
48. Kubelet begins running startupProbe (if configured). Until startupProbe succeeds,
liveness/readiness probes are not run.
49. After startupProbe succeeds, kubelet begins running readinessProbe. When
readinessProbe passes, kubelet sets the pod's Ready condition to True.
50. The Endpoints controller watches for Ready pods and adds the pod IP to the Endpoints
object for matching Services.
51. kube-proxy (or Cilium) updates iptables/eBPF rules to include the new pod IP in Service
load balancing.
52. The pod is now receiving live traffic.

Page 25 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

8.3 Workload Resources

8.3.1 Deployment
A Deployment manages a ReplicaSet and provides declarative rolling updates. Key fields:
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0 # Never have fewer than desired replicas serving traffic
maxSurge: 1 # Allow 1 extra pod during update
selector:
matchLabels:
app: payments
template:
metadata:
labels:
app: payments
version: v2.3.1
spec:
containers:
- name: payments
image: [Link]/payments:v2.3.1
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: 1000m
memory: 1Gi

8.3.2 StatefulSet
StatefulSets are for workloads that require stable network identity, stable storage, and ordered
deployment/scaling. Use cases: databases (Postgres, MySQL), distributed systems (ZooKeeper,
Kafka, Elasticsearch).
• Pods get predictable DNS names: [Link]-
[Link]
• PersistentVolumeClaims are created per pod and survive pod deletion.
• Pods are created and deleted in order (0, 1, 2... and reverse for deletion).
• Parallel launch available with podManagementPolicy: Parallel.

Page 26 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

8.3.3 DaemonSet
DaemonSets ensure one pod runs on every node (or a subset matching a nodeSelector). Used for:
log collectors (Fluentd, Filebeat), monitoring agents (Prometheus node exporter, Datadog agent),
CNI plugins, kube-proxy itself.

8.3.4 HorizontalPodAutoscaler (HPA)

HPA automatically scales the number of pod replicas based on metrics:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: payments
minReplicas: 3
maxReplicas: 50
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Pods
pods:
metric:
name: http_requests_per_second
target:
type: AverageValue
averageValue: 1000

Page 27 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 9: Kubernetes Networking — Pod-to-Pod

Communication
Kubernetes networking is one of the most complex and misunderstood topics in the ecosystem.
This chapter explains, from first principles, how network packets travel between pods, services, and
external clients.

9.1 The Kubernetes Networking Model

Kubernetes mandates a flat network model with these requirements:
• Every pod gets its own unique IP address.
• Pods can communicate with any other pod on any node without NAT.
• Nodes can communicate with pods without NAT.
• The IP a pod sees for itself is the same IP other pods use to reach it.
This flat model simplifies application development (no port mapping needed) but places the burden
on the CNI plugin to implement it, often using overlay networks or BGP routing.

9.2 Network Namespaces and veth Pairs

Each pod runs in its own Linux network namespace. The CNI plugin creates a virtual ethernet pair
(veth pair) to connect the pod namespace to the node's root namespace:
53. One end of the veth pair (eth0) is placed inside the pod's network namespace.
54. The other end (vethXXXXXX) remains in the node's root namespace.
55. The CNI plugin attaches the host-side veth to a bridge (cbr0 for kubenet, cni0 for Flannel) or
configures routing rules directly (Calico, Cilium).
56. The CNI assigns an IP from the pod CIDR to the pod's eth0 interface.

9.3 Same-Node Pod Communication

When Pod A ([Link]) communicates with Pod B ([Link]) on the same node:
57. The packet leaves Pod A's eth0 with src=[Link], dst=[Link].
58. The packet traverses the veth pair to the node's root namespace.
59. The kernel's routing table matches the dst to the local bridge (cbr0).
60. The bridge forwards the packet to the veth connected to Pod B's namespace.
61. The packet arrives at Pod B's eth0.
This is a pure L2 bridge operation within the node. No routing, no NAT, no tunnelling.

Page 28 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

9.4 Cross-Node Pod Communication

When Pod A ([Link] on Node 1) communicates with Pod C ([Link] on Node 2):

9.4.1 Overlay Networks (VXLAN / Flannel)

62. Packet leaves Pod A with src=[Link], dst=[Link].
63. The node's routing table routes the packet to the flannel.1 VXLAN tunnel interface.
64. Flannel encapsulates the original IP packet in a UDP frame with the outer src=Node1_IP,
dst=Node2_IP, port=8472 (VXLAN).
65. The encapsulated packet traverses the physical network to Node 2.
66. Node 2's VXLAN interface decapsulates the packet.
67. The inner packet (dst=[Link]) is routed to Pod C via the bridge.
Overhead: ~50 bytes per packet header. VXLAN encapsulation and decapsulation consume CPU.
Modern kernels offload this to the NIC hardware.

9.4.2 Native Routing (Calico with BGP)

Calico can use BGP to advertise pod CIDRs between nodes, eliminating the overlay entirely:
68. Each node runs a BIRD BGP daemon.
69. Nodes peer with each other (or a route reflector) and advertise: 'I own [Link]/24'.
70. Packets destined for another node's CIDR are forwarded via the normal IP routing
infrastructure.
This is more efficient than overlay networks but requires the underlying network to support BGP
routing (possible in AWS VPC, GKE, bare metal—not always possible in managed environments).

9.4.3 eBPF Networking (Cilium)

Cilium replaces iptables with eBPF programs attached to network interface hooks. eBPF programs
run in the kernel without context switches:
• Faster than iptables — O(1) lookup via eBPF maps vs O(n) iptables rule chains.
• Richer policies — L7 aware; can enforce HTTP method, URL, gRPC method.
• Native load balancing — replaces kube-proxy entirely with eBPF-based service load
balancing.
• Transparent encryption — WireGuard-based pod-to-pod encryption without sidecars.
• Hubble — built-in network observability; captures and displays per-flow metrics.

9.5 Kubernetes Services

9.5.1 ClusterIP

Page 29 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

The default service type. A virtual IP (ClusterIP) is allocated from the service CIDR (e.g.,
[Link]/12). kube-proxy programs iptables DNAT rules so packets destined for the ClusterIP are
forwarded to one of the endpoint pod IPs.
apiVersion: v1
kind: Service
metadata:
name: payments
namespace: production
spec:
selector:
app: payments
ports:
- port: 80 # Service port
targetPort: 8080 # Container port
protocol: TCP

9.5.2 NodePort
Exposes the service on a static port (30000–32767) on every node. External traffic can reach the
service by hitting any node IP on the NodePort. Rarely used directly in production; typically used as
the backing mechanism for LoadBalancer services.

9.5.3 LoadBalancer
Provisions a cloud provider load balancer (AWS NLB, GCP TCP LB) and configures it to forward
traffic to the NodePort. The cloud-controller-manager handles provisioning. Supported by AWS
EKS, GKE, AKS, and other managed Kubernetes services.

9.5.4 Headless Service

Setting clusterIP: None creates a headless service. DNS returns A records for all Ready pod IPs
instead of a single virtual IP. Used by StatefulSets for stable pod identity and by service meshes
that implement their own load balancing.

9.5.5 ExternalName
Maps a service to an external DNS name. DNS returns a CNAME to the external name. Used to
abstract external dependencies (e.g., a managed RDS instance) behind a Kubernetes service
name, making migration transparent.

9.6 kube-proxy Deep Dive

Page 30 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

9.6.1 iptables Mode

kube-proxy in iptables mode creates:
• A KUBE-SERVICES chain in the NAT table: checks if dst matches any service ClusterIP.
• Per-service KUBE-SVC-XXXXX chains: selects an endpoint using statistic rules (probability-
based round-robin).
• Per-endpoint KUBE-SEP-XXXXX chains: applies DNAT to rewrite dst to the pod IP.
The return packet (from pod to client) is tracked by conntrack; the kernel automatically reverses the
DNAT for established connections.

9.6.2 IPVS Mode

IPVS (IP Virtual Server) is a L4 load balancer built into the Linux kernel. kube-proxy in IPVS mode
uses the kernel's IPVS module instead of iptables rules:
• O(1) lookup — uses hash tables instead of linear iptables chain traversal.
• More load balancing algorithms — round-robin, least connections, destination hash, source
hash.
• Better performance at scale — iptables becomes slow with thousands of services.

Page 31 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 10: Inter-Service Communication Patterns

How services communicate with each other is one of the most consequential architectural decisions
in a microservices system. The choice of communication pattern affects consistency, latency,
resilience, and operational complexity.

10.1 Synchronous Communication

10.1.1 HTTP/REST
REST over HTTP/1.1 or HTTP/2 is the most common inter-service communication protocol. Key
design decisions:
• HTTP/2 — enables multiplexing (multiple requests over one TCP connection), header
compression (HPACK), and server push. Eliminates head-of-line blocking at the HTTP layer.
• Keep-alive connections — reuse TCP connections. Critical for performance; eliminates TCP
and TLS handshake overhead on every request.
• Timeouts — always set read, write, and idle timeouts. The default of 'no timeout' will cause
thread/goroutine exhaustion in downstream failures.
• Idempotency keys — include a client-generated unique ID in requests that create or modify
state. Allows safe retries.

10.1.2 gRPC
gRPC is a high-performance RPC framework built on HTTP/2 and Protocol Buffers. It is the
preferred protocol for synchronous inter-service communication in performance-critical systems.
Advantages over REST:
• Efficient serialisation — Protobuf binary encoding is 3–10x smaller than JSON.
• Strongly typed contracts — .proto files are the API contract; code generation eliminates
marshalling bugs.
• Bidirectional streaming — client streaming, server streaming, and full-duplex streaming over a
single connection.
• Built-in deadline propagation — deadlines (not timeouts!) propagate across service
boundaries; a downstream service automatically cancels work when the upstream deadline
expires.
• Interceptors — middleware pattern for logging, tracing, authentication, and retry logic.

gRPC service definition example:

syntax = "proto3";
package payments.v1;

service PaymentService {
rpc Charge(ChargeRequest) returns (ChargeResponse);

Page 32 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

rpc StreamTransactions(StreamRequest) returns (stream Transaction);

}

message ChargeRequest {
string idempotency_key = 1;
string user_id = 2;
int64 amount_cents = 3;
string currency = 4;
string payment_method_id = 5;
}

10.2 Asynchronous Communication

10.2.1 Message Queues
Message queues decouple producers from consumers. The producer sends a message and returns
immediately; the consumer processes it independently. Key properties:
Term / Component Description / Detail
At-most-once delivery Message is delivered once or not at all. Suitable for non-critical metrics
and logs. Risk of message loss.
At-least-once delivery Message is delivered one or more times. Requires consumers to be
idempotent. Most common guarantee.
Exactly-once delivery Message is processed exactly once. Requires distributed transactions
or idempotency at the consumer. Expensive to implement correctly.

10.2.2 Apache Kafka

Kafka is a distributed log, not a traditional queue. Messages (events) are appended to partitioned,
replicated logs called topics. Consumers track their position (offset) independently.
• Topics and partitions — a topic is divided into N partitions. Each partition is an ordered,
immutable log. Partitions enable parallelism: N consumers in a consumer group can read in
parallel.
• Consumer groups — a named group of consumers that jointly consume a topic. Kafka assigns
partitions to consumers within a group; each partition is consumed by exactly one consumer in
the group.
• Retention — messages are retained for a configurable duration (e.g., 7 days) regardless of
consumption. This allows replay and consumer lag recovery.
• Replication factor — each partition has one leader and N-1 followers. Producers write to the
leader; followers replicate. If the leader fails, a follower is elected leader.
• Producer acknowledgement — acks=all requires all in-sync replicas to acknowledge before
the producer receives confirmation. This is the strongest durability guarantee.

Page 33 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

10.2.3 The Saga Pattern

In a microservices architecture, a business transaction that spans multiple services cannot use
ACID distributed transactions (they require tight coupling and don't work across independent
databases). The Saga pattern implements distributed transactions as a sequence of local
transactions with compensating transactions for rollback.
Example — Order placement saga:
71. Order Service creates order (status=PENDING) → emits OrderCreated event.
72. Inventory Service reserves items → emits InventoryReserved event.
73. Payment Service charges customer → emits PaymentCharged event.
74. Order Service marks order as CONFIRMED → emits OrderConfirmed event.
75. Notification Service sends confirmation email.
If step 3 fails, compensating transactions run in reverse: Inventory Service releases reservation,
Order Service marks order as FAILED.

10.3 Resilience Patterns

10.3.1 Circuit Breaker
The circuit breaker pattern prevents a failing service from being overwhelmed with requests. Three
states:
• Closed (normal) — requests pass through. Failure count is tracked.
• Open (failing) — requests are rejected immediately with a fallback response. No calls to the
failing service.
• Half-Open — after a timeout, a test request is allowed through. If it succeeds, the circuit
closes; if it fails, the circuit remains open.

10.3.2 Retry with Exponential Backoff and Jitter

Retrying immediately after a failure often worsens the situation (thundering herd). Exponential
backoff with jitter is the standard approach:
delay = min(cap, base * 2^attempt) + random_jitter
Example: base=100ms, cap=30s, jitter=±50%
Attempt 1: ~100ms
Attempt 2: ~200ms
Attempt 3: ~400ms
Attempt N: ~30s (capped)
Only retry on idempotent operations or with idempotency keys. Never retry without understanding
the downstream contract.

10.3.3 Bulkhead Pattern

Page 34 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Isolate resources for different consumers to prevent a slow consumer from exhausting shared
resources. Example: a service that calls both a fast API and a slow database uses separate thread
pools / connection pools for each. If the database becomes slow and saturates its pool, the fast API
calls are unaffected.

Page 35 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 11: Secrets Management

Secrets management is the practice of securely storing, distributing, rotating, and auditing sensitive
configuration values: database passwords, API keys, TLS certificates, encryption keys, and service
account credentials. This is a critical principal engineer concern; secrets mismanagement is one of
the most common causes of data breaches.

11.1 What is a Secret?

Secrets are values that, if exposed, would allow an adversary to authenticate as your application,
access your databases, decrypt your data, or impersonate your services. Common types:
• Database credentials (username/password, connection strings)
• External API keys (Stripe, Twilio, SendGrid, AWS access keys)
• TLS certificates and private keys
• Encryption keys and key encryption keys (KEK)
• OAuth client secrets
• Internal service-to-service shared secrets
• SSH private keys for CI/CD pipelines

11.2 Kubernetes Secrets

11.2.1 How Kubernetes Secrets Work
Kubernetes Secrets are API objects that store base64-encoded (not encrypted!) key-value pairs.
They are stored in etcd. Critical facts that every principal engineer must know:

WARNING: Kubernetes Secrets are base64-encoded, NOT encrypted at rest by default. Anyone with
read access to etcd has access to all secrets. You MUST enable encryption at rest for etcd and
restrict RBAC for Secret access.

Creating a secret:
kubectl create secret generic db-creds \
--from-literal=username=app_user \
--from-literal=password=super_secret_password

Secrets are delivered to pods via:

• Environment variables — mounted as env vars. Simple but visible in process listings and
crash dumps.
• Volume mounts — mounted as files in a tmpfs volume (in-memory). Better: secrets are not in
env, files can be updated on rotation without pod restart (if application watches the file).

Page 36 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

spec:
containers:
- name: app
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-creds
key: password
volumeMounts:
- name: db-secret
mountPath: /etc/secrets/db
readOnly: true
volumes:
- name: db-secret
secret:
secretName: db-creds

11.2.2 Encryption at Rest

Kubernetes supports encrypting secrets in etcd using the EncryptionConfiguration API:
apiVersion: [Link]/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <base64-encoded-32-byte-key>
- identity: {} # Fallback for decrypting old secrets
For production, use a KMS provider (AWS KMS, GCP Cloud KMS, HashiCorp Vault) instead of
aescbc. This avoids storing the encryption key on disk with the data.

11.3 HashiCorp Vault

Vault is the industry-standard secrets management platform. It provides dynamic secrets,
encryption as a service, audit logging, and fine-grained access control.

11.3.1 Vault Architecture

Page 37 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Term / Component Description / Detail

Storage Backend Where Vault persists encrypted data. Options: Consul, DynamoDB, S3,
Integrated Raft. The data is always encrypted with the master key
before writing.
Seal/Unseal Vault starts in a sealed state where data cannot be accessed.
Unsealing requires a threshold of Shamir's Secret Sharing key
fragments (or auto-unseal via KMS). This protects against theft of the
storage backend.
Auth Methods How clients authenticate to Vault: Kubernetes auth, AWS IAM auth,
GCP auth, LDAP, userpass, AppRole, OIDC.
Secret Engines Plugins that generate or store secrets. KV (key-value), Database
(dynamic credentials), PKI (cert authority), AWS, GCP, SSH, Transit
(encryption as a service).
Policies HCL-based access control policies. Define read/write/list capabilities on
paths.
Leases and Renewal Dynamic secrets have a TTL (lease). Clients must renew leases or the
secret is revoked automatically.

11.3.2 Kubernetes Auth Method

Services running in Kubernetes authenticate to Vault using the pod's service account JWT token:
76. The pod presents its service account token (mounted at
/var/run/secrets/[Link]/serviceaccount/token).
77. Vault calls the Kubernetes TokenReview API to validate the token.
78. Vault returns a Vault token with the policies mapped to the Kubernetes role.
79. The pod uses the Vault token to read secrets.
vault write auth/kubernetes/role/payments \
bound_service_account_names=payments-sa \
bound_service_account_namespaces=production \
policies=payments-policy \
ttl=1h

11.3.3 Dynamic Database Credentials

Vault's Database secrets engine generates unique, short-lived database credentials per application
instance:
80. Application authenticates to Vault and requests a database credential.
81. Vault connects to the database using a privileged service account and executes a CREATE
ROLE statement with a unique username and random password.
82. Vault returns the credentials to the application with a TTL (e.g., 1 hour).
83. The application uses the credentials. When the lease expires, Vault revokes the credentials
by dropping the database role.

Page 38 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

This means: no shared passwords, passwords rotate automatically, compromised credentials

expire quickly, and audit logs show exactly which application instance accessed the database.

11.4 External Secrets Operator

The External Secrets Operator (ESO) is a Kubernetes operator that synchronises secrets from
external stores (Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault) into
Kubernetes Secrets.
apiVersion: [Link]/v1beta1
kind: ExternalSecret
metadata:
name: db-credentials
namespace: production
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: db-credentials
creationPolicy: Owner
data:
- secretKey: password
remoteRef:
key: production/payments/database
property: password

Page 39 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 12: Pod Authentication and Workload

Identity
Authentication in a microservices system operates at multiple levels: user-to-service, service-to-
service, and pod-to-infrastructure (cloud APIs, secrets stores, databases). This chapter covers
every authentication mechanism a principal engineer must understand.

12.1 Kubernetes Service Accounts

A ServiceAccount is a Kubernetes object that provides an identity to pods running within a
namespace. By default, every pod gets the 'default' service account, which should be avoided—
each application should have a dedicated service account with minimal permissions.
apiVersion: v1
kind: ServiceAccount
metadata:
name: payments-sa
namespace: production
annotations:
# For AWS IRSA (IAM Roles for Service Accounts)
[Link]/role-arn: arn:aws:iam::123456789:role/payments-role

12.1.1 Service Account Tokens

Modern Kubernetes (1.21+) uses bound service account tokens (TokenRequest API). These tokens
are:
• Audience-bound — valid only for specific audiences (e.g., 'vault', '[Link]').
• Time-bound — automatically expire (default: 1 hour for projected volumes).
• Pod-bound — the token becomes invalid when the pod is deleted.
spec:
serviceAccountName: payments-sa
volumes:
- name: token
projected:
sources:
- serviceAccountToken:
audience: vault
expirationSeconds: 3600
path: vault-token

12.2 AWS IAM Roles for Service Accounts (IRSA)

Page 40 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

IRSA enables pods running in EKS to assume AWS IAM roles without AWS access keys in the
environment. This is the recommended mechanism for granting pods access to AWS services (S3,
SQS, DynamoDB, Secrets Manager).
How IRSA works:
84. EKS exposes an OIDC provider endpoint
([Link]
85. An IAM role is created with a trust policy that trusts the EKS OIDC provider and the specific
service account.
86. The pod's service account is annotated with the IAM role ARN.
87. The AWS SDK's credential provider chain calls the pod's projected service account token
and exchanges it for temporary AWS credentials via STS AssumeRoleWithWebIdentity.
88. The pod receives time-limited AWS credentials (access key, secret key, session token)
without any long-term credentials on disk.
# IAM Trust Policy
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456789:oidc-provider/[Link]-east-
[Link]/id/CLUSTER"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"[Link].../id/CLUSTER:sub": "system:serviceaccount:production:payments-sa"
}
}
}]
}

12.3 Workload Identity on GKE

GKE's Workload Identity is the equivalent of IRSA for Google Cloud. A Kubernetes ServiceAccount
is bound to a Google Cloud Service Account (GSA). Pods use the GSA's identity to call Google
APIs.
The binding is configured via an IAM policy annotation on the Kubernetes ServiceAccount and an
IAM policy binding on the GCP service account. The GKE metadata server (running as a
DaemonSet) intercepts calls to the GCE metadata API and returns tokens for the bound GSA.

12.4 SPIFFE/SPIRE for Workload Identity

Page 41 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

SPIFFE provides a universal workload identity standard that works across multiple clouds, on-
premise, and edge environments. SPIRE is the reference implementation of the SPIFFE
specifications.

12.4.1 SPIRE Architecture

• SPIRE Server — acts as the Certificate Authority. Stores the node attestation database. Runs
outside the cluster it serves.
• SPIRE Agent — runs as a DaemonSet on each Kubernetes node. Attests the node identity to
the SPIRE Server using the underlying platform (AWS EC2 attestor, GCP attestor,
Kubernetes attestor).
• Workload API — SPIRE Agent exposes a Unix domain socket. Workloads (pods) call the
Workload API to receive their SVID (SPIFFE Verifiable Identity Document) — an X.509
certificate with the SPIFFE ID in the SAN field.
The SPIFFE ID format: spiffe://trust-domain/path
spiffe://[Link]/ns/production/sa/payments-sa

12.5 OAuth 2.0 Token Exchange for Service-to-Service Auth

In some architectures, services authenticate to each other by exchanging tokens. A service obtains
a client credentials grant (OAuth 2.0 RFC 6749, Section 4.4) from the authorization server using its
client ID and secret:
POST /oauth/token
Content-Type: application/x-www-form-urlencoded

grant_type=client_credentials
&client_id=payments-service
&client_secret=<secret>
&scope=inventory:read orders:write
The authorization server returns a short-lived access token. The calling service presents this token
in the Authorization header of downstream calls. The downstream service validates the token at the
API gateway or in its own middleware.

12.6 mTLS as Authentication

In a service mesh with mTLS enforced (see Chapter 7), the X.509 certificate presented during the
TLS handshake is the authentication mechanism. The SPIFFE ID in the certificate's SAN field
identifies the caller. No additional token exchange is needed—identity is cryptographically proven at
the transport layer.
This is the strongest form of service-to-service authentication because:
• Certificates are issued by the mesh CA (SPIRE or Istiod) and rotate automatically (typically
every 24 hours).
• A compromised certificate is valid for at most 24 hours.

Page 42 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

• The private key never leaves the pod (generated locally, only the CSR is sent to the CA).
• Istio's AuthorizationPolicy can enforce which SPIFFE identities are allowed to call which
services and operations.

Page 43 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 13: Deployment Strategies

How you deploy new versions of your services is as important as how you build them. The wrong
deployment strategy can cause prolonged outages, failed rollbacks, and loss of customer trust. This
chapter covers every deployment pattern used in production Kubernetes environments.

13.1 Rolling Update (Default)

Kubernetes deployments use rolling updates by default. Old pods are replaced incrementally with
new pods, maintaining availability throughout.
Key parameters:
• maxUnavailable: 0 — never reduce capacity below the desired count. Essential for production.
• maxSurge: 1 — allow one extra pod to be running simultaneously during the update. Requires
1.33x normal capacity headroom.
Rolling updates are appropriate for backward-compatible changes. For breaking API changes or
database migrations, more sophisticated strategies are required.

13.2 Blue/Green Deployment

Two identical environments (blue=current, green=new) run simultaneously. Traffic is switched
atomically from blue to green. Blue is kept running for rollback.
Implementation options:
• DNS-based — change the DNS record to point to the green load balancer. Rollback is fast but
limited by DNS TTL.
• Load balancer target group swap — ALB listener rule updated to point to the green target
group. Instant switchover and rollback.
• Kubernetes service selector update — change the service selector from version:v1 to
version:v2. Instant, but requires both versions' pods to run simultaneously.
Drawbacks: requires 2x infrastructure cost during the transition. Database migrations must be
backward-compatible with both versions.

13.3 Canary Deployment

A canary deployment routes a small percentage of traffic to the new version while the majority
continues to receive the current version. This limits the blast radius of a bad deployment.
Traffic splitting mechanisms:
• Kubernetes replica ratio — 1 canary pod + 9 stable pods = ~10% canary traffic. Coarse-
grained.
• Istio VirtualService weighted routing — precise percentage-based splits (see Chapter 7).

Page 44 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

• Header-based routing — internal users or opted-in beta users get the canary version.
• Argo Rollouts — Kubernetes controller that implements progressive delivery with automated
analysis.

13.3.1 Automated Canary Analysis

A canary deployment is only valuable if you automatically detect when the canary is behaving
worse than the baseline. Argo Rollouts integrates with Prometheus and Datadog to evaluate
analysis metrics:
apiVersion: [Link]/v1alpha1
kind: AnalysisTemplate
spec:
metrics:
- name: success-rate
interval: 5m
successCondition: result[0] >= 0.95
failureLimit: 3
provider:
prometheus:
query: |
sum(rate(http_requests_total{
status!~"5..",
deployment="payments-canary"
}[5m]))
/
sum(rate(http_requests_total{
deployment="payments-canary"
}[5m]))

13.4 Feature Flags

Feature flags (also called feature toggles) decouple deployment from release. Code is deployed to
production in a disabled state and enabled for specific users, groups, or percentages via a
configuration system—without a new deployment.
This enables:
• Dark launches — deploy and test in production before enabling for users.
• Percentage rollouts — gradually increase the enabled percentage.
• Kill switches — instantly disable a feature without a rollback.
• A/B testing — compare two implementations in production.
Popular feature flag systems: LaunchDarkly, Unleash, Flagsmith, ConfigCat, AWS AppConfig.

Page 45 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

13.5 Database Migrations in Zero-Downtime Deployments

The hardest part of zero-downtime deployments is database schema changes. The golden rule:
database migrations must be backward-compatible with both the old and new version of the
application.
The expand/contract pattern:
89. Expand phase — add the new column/table/index without removing old ones. Both old and
new application versions can run against this schema.
90. Migrate data — backfill the new column. Can run in parallel with both application versions.
91. Deploy new application — the new version uses the new column.
92. Contract phase — (after all old pods are gone) remove the old column in a separate
migration.
Never: add a NOT NULL column without a default in a single migration. Always: add as nullable
first, populate data, then add the constraint.

Page 46 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 14: Observability — Metrics, Logs, and

Traces
Observability is the ability to understand the internal state of a system from its external outputs. The
three pillars—metrics, logs, and traces—form the foundation of production operations for a principal
engineer.

14.1 The Three Pillars

Term / Component Description / Detail
Metrics Numeric measurements aggregated over time. Examples: request rate,
error rate, latency percentiles, CPU usage, memory usage. Low
cardinality, high efficiency. Used for alerting and dashboards.
Logs Timestamped, structured records of discrete events. Examples: HTTP
request logs, application errors, audit events. High cardinality, detailed
context. Used for debugging specific incidents.
Traces End-to-end records of requests as they propagate through distributed
services. Each trace consists of spans. Used to understand system
behaviour across service boundaries and identify latency bottlenecks.

14.2 Metrics with Prometheus

14.2.1 Prometheus Data Model
Every metric in Prometheus is a time series identified by a metric name and a set of key-value
labels:
http_requests_total{service="payments", method="POST", status="200"} 12453
http_request_duration_seconds{service="payments", quantile="0.99"} 0.342
Four metric types:
• Counter — monotonically increasing value. Resets to 0 on process restart. Example: total
requests, total errors.
• Gauge — value that can go up or down. Example: current goroutine count, queue depth,
memory usage.
• Histogram — samples observations into configurable buckets; also tracks sum and count.
Used for latency and size distributions.
• Summary — like histogram, but computes quantiles on the client side. Avoid in multi-instance
deployments (quantiles cannot be aggregated across instances).

14.2.2 The RED Method

For every service, track these three signals (the RED method):

Page 47 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

• Rate — requests per second.

• Errors — error rate (% of requests returning 5xx or business errors).
• Duration — latency distribution (p50, p90, p99, p99.9).
# Rate
rate(http_requests_total{service="payments"}[5m])

# Error Rate
rate(http_requests_total{service="payments",status=~"5.."}[5m])
/ rate(http_requests_total{service="payments"}[5m])

# p99 Latency
histogram_quantile(0.99,
rate(http_request_duration_seconds_bucket{service="payments"}[5m]))

14.3 Distributed Tracing

Distributed tracing follows a request through every service it touches, creating a timeline (trace) of
all operations. This is essential for understanding microservices performance.

14.3.1 Trace and Span Structure

A trace is a collection of spans. Each span represents a unit of work:
• Trace ID — unique 128-bit ID shared across all spans for a single request.
• Span ID — unique 64-bit ID for each individual span.
• Parent Span ID — links child spans to their parent, forming the trace tree.
• Operation name — human-readable name (e.g., '[Link]', '[Link]').
• Start time and duration.
• Tags/attributes — key-value metadata (HTTP method, status code, user ID, error flag).
• Logs/events — timestamped events within the span (e.g., 'cache miss', 'retry attempt 2').

14.3.2 OpenTelemetry
OpenTelemetry (OTel) is the CNCF standard for instrumentation. It provides vendor-neutral APIs,
SDKs, and a collector. Key components:
• API — language-specific API for creating traces, metrics, and logs (no-op by default).
• SDK — configurable implementation of the API. Configure exporters and processors.
• Collector — a standalone binary that receives, processes, and exports telemetry data to
backends (Jaeger, Zipkin, Tempo, Honeycomb, Datadog).
• Auto-instrumentation — Java agent, Python sitecustomize, Go compile-time — automatically
instruments popular frameworks.

Page 48 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

14.4 Structured Logging

Logs in a microservices environment must be structured (JSON) and contain consistent fields.
Every log entry must include:
• timestamp — ISO 8601 with microsecond precision.
• level — debug, info, warn, error.
• service — the service name and version.
• trace_id and span_id — for correlation with distributed traces.
• request_id — the unique request ID from the API gateway.
• user_id — if applicable and not PII-restricted.
• message — the human-readable description.
• error — structured error object with code, message, stack trace.

{
"timestamp": "2024-11-15T10:32:45.123456Z",
"level": "error",
"service": "payments",
"version": "v2.3.1",
"trace_id": "7f4b2a1c8e3d5f6a",
"span_id": "3d1a9b4c",
"request_id": "req_01HGZ4KWFJ5XN7",
"message": "Database connection failed",
"error": {
"code": "DB_CONNECTION_ERROR",
"message": "dial tcp [Link]:5432: connect: connection refused"
}
}

14.5 SLOs, SLAs, and Error Budgets

A principal engineer must think in terms of reliability targets.
Term / Component Description / Detail
SLI (Service Level Indicator) A quantitative measure of service behaviour. Example: the proportion
of HTTP requests that return a 2xx or 3xx response within 500ms.
SLO (Service Level An internal target for an SLI. Example: 99.9% of requests succeed
Objective) within 500ms, measured over a rolling 30-day window.
SLA (Service Level A contractual commitment to customers, with financial penalties for
Agreement) violation. Always more lenient than the internal SLO.
Error Budget The allowed amount of unreliability: 100% - SLO%. 99.9% SLO = 0.1%
error budget = 43.8 minutes of downtime per 30 days.

Page 49 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Error budgets enable objective decisions about risk: if the error budget is healthy, teams can move
fast. If the error budget is depleted, teams slow down and focus on reliability.

Page 50 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 15: Security in Depth — Zero Trust

Architecture
Zero Trust is the security model that assumes no implicit trust—every request must be
authenticated, authorized, and validated regardless of network location. The perimeter security
model ('trusted internal network') has been definitively broken by cloud adoption, remote work, and
insider threats.

15.1 Zero Trust Principles

• Never trust, always verify — authenticate and authorise every request, including internal
service-to-service calls.
• Least privilege — grant the minimum permissions required. Regularly audit and prune
permissions.
• Assume breach — design systems as if an attacker already has internal network access.
Enforce mTLS, encrypt data at rest, and segment blast radius.
• Explicit verification — use multiple signals for authorisation: identity, device health, network
location, time of day, behaviour.

15.2 Network Policies

Kubernetes NetworkPolicies restrict pod-to-pod traffic at the L3/L4 level. By default, all pods can
communicate with all other pods. NetworkPolicies implement a default-deny posture:
# Default deny all ingress and egress in production namespace
apiVersion: [Link]/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: production
spec:
podSelector: {} # Applies to all pods
policyTypes:
- Ingress
- Egress

# Allow payments to accept from order-service on port 8080

apiVersion: [Link]/v1
kind: NetworkPolicy
metadata:
name: allow-orders-to-payments
namespace: production
spec:

Page 51 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

podSelector:
matchLabels:
app: payments
ingress:
- from:
- podSelector:
matchLabels:
app: order-service
ports:
- protocol: TCP
port: 8080

15.3 Pod Security

15.3.1 Pod Security Standards
Kubernetes 1.25+ deprecated PodSecurityPolicy in favour of Pod Security Standards (PSS):
Term / Component Description / Detail
Privileged No restrictions. Avoid for production workloads.
Baseline Prevents known privilege escalations. Allows some Linux capabilities.
Minimum standard for production.
Restricted Strict hardening. Requires non-root, disables privilege escalation,
requires seccomp. Goal for security-sensitive workloads.

15.3.2 Security Context Best Practices

securityContext:
runAsNonRoot: true
runAsUser: 65534
runAsGroup: 65534
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault

15.4 Supply Chain Security

Securing the software supply chain means verifying the integrity and provenance of everything that
runs in your cluster.

Page 52 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

• Image signing — sign container images with Cosign (Sigstore). Verify signatures in the
admission controller.
• SBOM (Software Bill of Materials) — generate and attest a list of all dependencies in each
image. Required for vulnerability tracking and licence compliance.
• CVE scanning — scan images for known vulnerabilities in registries (ECR enhanced
scanning, Trivy) and at admission time.
• Distroless images — base images that contain only the runtime (no shell, no package
manager, no debugging tools). Dramatically reduces attack surface.
• Image pinning — reference images by digest (sha256:...) not tag. Tags are mutable; digests
are immutable.

15.5 RBAC Best Practices

Kubernetes RBAC controls who can perform what actions on which resources. Common mistakes
and their fixes:
Term / Component Description / Detail
Mistake: Granting cluster- Fix: Create a specific Role that can only update Deployments in the
admin to CI/CD target namespace.
Mistake: Using the default Fix: Create a dedicated ServiceAccount per application with minimal
service account permissions.
Mistake: Using Fix: Prefer namespace-scoped RoleBindings to limit blast radius.
ClusterRoleBinding when
RoleBinding suffices
Mistake: Granting wildcard Fix: List specific resources and verbs. Audit with 'kubectl auth can-i --
resource permissions list'.
Mistake: Never rotating Fix: Use bound tokens (TokenRequest API) which expire automatically.
ServiceAccount tokens

Page 53 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 16: CI/CD and GitOps

Continuous Integration and Continuous Delivery are the operational practices that allow teams to
ship changes frequently and safely. GitOps extends this by using Git as the single source of truth
for both application code and infrastructure state.

16.1 CI Pipeline Stages

A mature CI pipeline for a microservice includes:
93. Code checkout and dependency installation
94. Static analysis (linting, SAST with Semgrep or SonarQube)
95. Unit tests with coverage report
96. Integration tests against a real database (Testcontainers)
97. Container image build (multi-stage Dockerfile)
98. Image vulnerability scanning (Trivy)
99. Image signing (Cosign)
100. Image push to registry (ECR, GCR, GHCR)
101. Helm chart packaging and push to chart repository
102. Notification (Slack, PagerDuty)

16.2 GitOps with Argo CD

Argo CD is a declarative GitOps continuous delivery tool for Kubernetes. It continuously watches a
Git repository and ensures the cluster state matches the desired state in the repository.

16.2.1 Argo CD Application

apiVersion: [Link]/v1alpha1
kind: Application
metadata:
name: payments
namespace: argocd
spec:
project: production
source:
repoURL: [Link]
targetRevision: main
path: services/payments/overlays/production
destination:
server: [Link]
namespace: production
syncPolicy:

Page 54 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

automated:
prune: true # Delete resources not in Git
selfHeal: true # Fix manual kubectl changes
syncOptions:
- CreateNamespace=true

16.3 Multi-Environment Promotion

A mature GitOps setup includes multiple environments with explicit promotion gates:
103. Dev — automatic deployment on every merge to main. No approval required.
104. Staging — automatic deployment on every merge to main. Integration and E2E tests run.
May require manual approval for promotion.
105. Production — triggered by creating a Git tag or merging a PR that bumps the image tag in
the production overlay. Requires approval from two engineers.
Image promotion strategy: the CI pipeline updates only the image tag in the specific environment's
overlay (Kustomize or Helm values file), creating a Git commit. Argo CD detects the change and
syncs.

16.4 Dockerfile Best Practices

Container image quality directly affects startup time, security, and operational cost.
# Multi-stage build for a Go service
FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY [Link] [Link] ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o payments ./cmd/payments

FROM [Link]/distroless/static-debian12:nonroot
COPY --from=builder /app/payments /payments
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/payments"]
Key practices: multi-stage to minimise final image size; distroless base image; non-root user;
minimal layers; pinned base image digest.

Page 55 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 17: Data Layer — Databases, Caches, and

Storage
The data layer is typically the bottleneck and the hardest component to scale. This chapter covers
database patterns, caching strategies, and storage primitives that a principal engineer must master.

17.1 Database Patterns for Microservices

17.1.1 Database-per-Service
Each microservice owns its data store. This is a core microservices principle that enables
independent deployment, technology choice, and scaling.
• Each service has its own schema (or separate database instance for stronger isolation).
• No direct cross-service database queries. Data is shared via APIs or events.
• Services can choose the appropriate database type: relational (PostgreSQL), document
(MongoDB), time-series (InfluxDB), graph (Neo4j), key-value (Redis), wide-column
(Cassandra).

17.1.2 CQRS (Command Query Responsibility Segregation)

CQRS separates the write model (commands) from the read model (queries). Different data stores,
schemas, and scaling strategies can be used for each:
• Write side — normalised relational database optimised for transactional integrity.
• Read side — denormalised, projected views optimised for query patterns. Can be a separate
read replica, Elasticsearch index, Redis cache, or materialised view.
• Synchronisation — events or change data capture (CDC) keep the read side up to date.

17.2 Connection Pooling

Database connections are expensive: each TCP connection requires a file descriptor, memory, and
authentication overhead. Connection pooling reuses connections across requests.

17.2.1 PgBouncer for PostgreSQL

PgBouncer is a lightweight connection pooler for PostgreSQL. It runs as a sidecar or dedicated
service and multiplexes many application connections onto a smaller pool of real database
connections.
• Session mode — each client gets a dedicated server connection for the duration of the
session. Simplest but doesn't reduce connection count.
• Transaction mode — server connection is assigned per transaction. Most efficient;
incompatible with prepared statements and SET variables.
• Statement mode — server connection is assigned per statement. Rarely used.

Page 56 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Configuration: pool_size=20, max_client_conn=1000 means 1000 application connections map to

20 real database connections.

17.3 Caching Strategies

17.3.1 Cache Aside (Lazy Loading)
The application reads from cache first; on a miss, reads from the database, populates the cache,
and returns the result. The cache is only populated with data that is actually requested.

17.3.2 Write-Through
The application writes to the cache and the database simultaneously (or the cache writes through to
the database). Ensures cache consistency at the cost of write latency.

17.3.3 Write-Behind (Write-Back)

The application writes to the cache; the cache asynchronously writes to the database. Low write
latency but risk of data loss if cache fails before flush.

17.3.4 Read-Through
The cache handles the database read on a miss, transparently to the application. The application
always reads from cache; the cache manages invalidation.

17.4 Redis in Production

Redis is the dominant in-memory data structure store used for caching, session storage, pub/sub
messaging, rate limiting, and distributed locks.

17.4.1 Redis Sentinel vs Redis Cluster

Term / Component Description / Detail
Redis Sentinel High availability for a single shard. One primary, multiple replicas.
Sentinel processes monitor health and perform automatic failover. No
horizontal scaling of writes.
Redis Cluster Horizontal sharding across 3–1000+ shards. Data is automatically
partitioned using hash slots. Each shard has primary and replicas.
Scales writes and storage. More complex client requirements.

17.4.2 Distributed Locking with Redis (Redlock)

The Redlock algorithm uses multiple Redis instances to implement a distributed lock:
106. Acquire the lock on N/2+1 Redis instances within a timeout.
107. If successful (quorum achieved), the lock is held.

Page 57 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

108. Perform the critical section.

109. Release the lock on all instances.
Martin Kleppmann's critique of Redlock is important: it is not safe under clock skew or GC pauses.
For strong safety guarantees, use ZooKeeper or etcd (raft-based) for distributed locking.

Page 58 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 18: Principal Engineer Decision

Framework
This final chapter synthesises the technical knowledge from all previous chapters into decision
frameworks, anti-patterns to avoid, and principles that define principal-level engineering judgment.

18.1 Architectural Decisions at the Principal Level

A principal engineer is not just technically deep—they make decisions that affect the entire
organisation. Key decision areas:

18.1.1 Build vs Buy vs Adopt Open Source

Term / Component Description / Detail
Build Maximum control; maximum cost. Build only when the capability is a
core differentiator. Never build: auth, payments, monitoring, CDN,
DNS.
Buy (SaaS) Fastest to value; vendor dependency; potential data sovereignty
concerns. Buy: productivity tools, CRM, ERP, email delivery, SMS.
Adopt Open Source Community support; no licence cost; operational burden. Adopt for:
Kubernetes, Prometheus, Kafka, PostgreSQL, Redis, Istio.

18.1.2 When to Use a Monolith vs Microservices

The industry over-corrected toward microservices. The principal engineer's responsibility is to
choose the right decomposition:
• Monolith first — start with a well-modularised monolith. Extract services when you have clear
bounded contexts, independent scaling requirements, or team ownership boundaries.
• Service granularity — a service should be as small as it can be without requiring synchronous
cross-service calls for its primary use cases.
• Distributed system tax — every service boundary adds: network latency, serialisation
overhead, operational complexity, a deployment pipeline, and a new failure mode. This tax
must be justified by the benefit.

18.2 The Eight Fallacies of Distributed Computing

These assumptions, when made by engineers, lead to serious production failures:
110. The network is reliable. (Networks partition, packets are dropped, NICs fail.)
111. Latency is zero. (Cross-region latency is 100ms+; even cross-AZ is 1–5ms.)
112. Bandwidth is infinite. (Large payloads, fan-out patterns, and streaming can saturate links.)
113. The network is secure. (Zero trust: assume the network is compromised.)

Page 59 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

114. Topology doesn't change. (Pods, nodes, and IPs change constantly in Kubernetes.)
115. There is one administrator. (Multiple teams own different layers; coordination is required.)
116. Transport cost is zero. (Serialisation, encryption, and network I/O have CPU and memory
costs.)
117. The network is homogeneous. (Different services may use different protocols, MTUs, and
network segments.)

18.3 Common Anti-Patterns

18.3.1 The Distributed Monolith
Services that are deployed separately but are tightly coupled via synchronous RPC chains or
shared databases. Worst of both worlds: the operational complexity of microservices without the
independence benefits.

18.3.2 Chatty Services

A single user action triggers N+1 synchronous RPC calls. Each call adds latency and failure
probability. Solution: batch requests, use GraphQL/BFF aggregation, or switch to async patterns.

18.3.3 Shared Mutable State

Multiple services write to the same database table. Eliminates service autonomy and creates
hidden coupling. Solution: event-driven architecture with each service owning its data.

18.3.4 Missing Circuit Breakers

A downstream service becomes slow. The calling service's thread pool fills up waiting for
responses. The caller becomes slow, causing its callers to fill up their thread pools. Cascading
failure takes down the entire system in minutes.

18.3.5 Synchronous Everything

Using synchronous REST calls for operations that are inherently asynchronous (sending email,
generating a report, processing a payment). Solution: use message queues for operations where
the user doesn't need an immediate result.

18.4 The Principal Engineer's Checklist

Before launching a new service or major feature to production:

18.4.1 Reliability
• SLOs defined and monitored for the service
• Health checks (/healthz, /readyz) implemented and returning meaningful status
• Circuit breakers configured for all downstream dependencies
• Retry logic with exponential backoff and idempotency keys

Page 60 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

• Graceful shutdown handling SIGTERM and draining connections

• Horizontal Pod Autoscaler configured with tested load
• PodDisruptionBudget preventing simultaneous disruption of too many replicas

18.4.2 Security
• Container runs as non-root with read-only root filesystem
• All capabilities dropped
• Secrets injected via volume mounts (not env vars) from Vault or external secrets operator
• NetworkPolicy implementing default-deny with explicit allow rules
• RBAC configured with least privilege
• mTLS enforced via Istio PeerAuthentication in STRICT mode
• Image scanned for CVEs and signed

18.4.3 Observability
• Structured JSON logging with trace_id and span_id correlation
• RED metrics exposed via /metrics Prometheus endpoint
• OpenTelemetry distributed tracing instrumented
• Alerting rules configured for SLO breach
• Runbooks written for all alerts
• On-call rotation configured

18.4.4 Deployment
• Rolling update strategy with maxUnavailable: 0
• Canary deployment configured for traffic shifting
• Feature flags for high-risk code paths
• Database migrations are backward-compatible (expand/contract)
• Rollback procedure tested and documented
• Load tested to 2x expected peak traffic

18.5 Closing Thoughts

The systems described in this document represent the accumulated knowledge of the industry's
best practitioners. No single organisation implements all of these patterns perfectly—every system
is a set of deliberate trade-offs.
As a principal engineer, your job is to understand the trade-offs deeply enough to make the right
choices for your context: your team's size, your traffic volume, your regulatory requirements, and
your organisation's risk tolerance.

Page 61 of 62
Modern Distributed Systems Architecture — Principal Engineer Reference

The engineer who blindly adopts every pattern in this document will create an over-engineered
nightmare. The engineer who ignores these patterns will create a fragile, unmaintainable system.
Wisdom lies in the middle: understanding every pattern, knowing when to apply each one, and
building systems that are as simple as possible but no simpler.

The best system architecture is the simplest one that meets your requirements today and can be
extended to meet your requirements tomorrow without being rewritten.

Page 62 of 62