Modern Distributed Systems Architecture — Principal Engineer Reference

MODERN DISTRIBUTED
SYSTEMS
ARCHITECTURE
A Principal Engineer's Complete Reference
From Browser Keystroke to Microservice Response — Every Layer, Every Protocol,
Every Pattern

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Topics Covered
• DNS Resolution & Traffic Management
• TLS 1.3 Handshake & Certificate Lifecycle
• CDN Architecture, Caching & Edge Computing
• L4/L7 Load Balancing Algorithms & Health Checking
• API Gateway: AuthN/AuthZ, Rate Limiting & Routing
• Service Mesh: Istio, Envoy, mTLS & SPIFFE Identity
• Kubernetes: Control Plane, Scheduling & Pod Lifecycle
• Kubernetes Networking: CNI, eBPF, kube-proxy & Services
• Inter-Service Communication: gRPC, Kafka & Saga Pattern
• Secrets Management: Vault, IRSA & External Secrets
• Workload Identity: SPIRE, OIDC & mTLS Authentication
• Deployment Strategies: Canary, Blue/Green & GitOps
• Observability: Prometheus, OpenTelemetry & SLOs
• Zero Trust Security & Supply Chain Hardening

Page 1 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

• Data Layer: Databases, Caching & Connection Pooling

• Principal Engineer Decision Frameworks & Anti-Patterns
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
~200+ pages • Production-grade depth • Real configurations • Annotated code samples

Page 2 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 1: The Complete Anatomy of a Web

Request
Every time a user types a URL and presses Enter, they set off a cascade of dozens of precisely
coordinated events across multiple physical data centres, distributed software systems, and
networking layers. The entire journey—from keystroke to rendered pixels—typically completes in
under half a second. Understanding every step in this chain is the foundation on which all principal-
level system design knowledge rests.
This chapter traces the complete lifecycle of a request, explains why each layer exists, and
establishes the mental model that the remainder of this document expands upon with exhaustive
depth.

1.1 Before the Browser Sends a Byte

1.1.1 URL Parsing and Security Checks
When a user presses Enter on [Link] the browser
performs several pre-flight operations before any network activity occurs:
1. URL parsing — the browser tokenises the input into scheme (https), host ([Link]),
path (/checkout), query string (?cart=abc123), and optional fragment (#section). Invalid
characters are percent-encoded.
2. Scheme determination — HTTPS triggers the TLS code path. HTTP/2 is negotiated during the
TLS handshake (ALPN extension). HTTP/3 (QUIC) may be used if the server advertised Alt-
Svc in a previous response.
3. HSTS preload check — the browser checks its HSTS (HTTP Strict Transport Security)
preload list. If [Link] is on this list, the browser immediately upgrades any http://
request to https:// before sending a single packet.
4. Mixed content check — if the page is being loaded over HTTPS, the browser blocks sub-
resources loaded over HTTP (images, scripts) by default.
5. Cookie retrieval — the browser assembles all cookies matching the domain and path,
excluding HttpOnly cookies from JavaScript access, and respects SameSite policy.

1.1.2 The Browser's Connection Cache

Modern browsers maintain a connection pool. Before initiating a DNS lookup, the browser checks:
• HTTP/2 connection coalescing — if there is an existing HTTP/2 connection to the same IP
and the server certificate covers the new hostname, the browser reuses the connection,
saving the full TLS handshake.
• HTTP/3 session resumption — if a prior 0-RTT token exists for the destination, the browser
can send the request in the first packet.

Page 3 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

• Service Worker intercept — if a service worker is registered for the scope, it intercepts the
fetch event and can respond from a cache, forward to network, or modify the request.

1.2 The Twelve Layers of a Request

A production request traverses these layers in order. Each is expanded across dedicated chapters;
here we establish the complete picture:

Layer Component Primary Responsibility Typical Latency

Added
1 DNS Resolution Translate hostname to IP address; geo-route 0–50ms (0ms if
to nearest PoP cached)
2 TLS Handshake Establish encrypted, authenticated channel 10–30ms TLS 1.3
(0ms resumption)
3 CDN Edge Serve cached assets; WAF; DDoS absorption; 1–5ms on cache hit
edge logic
4 Origin Shield Second-tier CDN cache; collapse origin 2–8ms on cache hit
requests
5 Load Balancer Distribute connections; health check; 0.1–2ms
(L4/L7) terminate TLS
6 API Gateway AuthN/AuthZ; rate limit; route; transform; 2–15ms
observe
7 Service Mesh mTLS; policy enforcement; distributed tracing 0.2–1ms
Ingress start
8 Kubernetes Virtual IP to pod IP translation via kube- < 0.1ms (in-kernel)
Service proxy/eBPF
9 Target Pod / Business logic execution 10–500ms
Service
10 Inter-service Downstream service communication 5–100ms per hop
calls
11 Data Layer Database, cache, object store reads/writes 1–50ms
12 Response Path Reverse journey: compression, chunked Same as forward
encoding, streaming path

1.3 Latency Budget Analysis

The latency budget is the total time budget allocated for a request. Principal engineers must reason
about this budget explicitly. Below is a detailed p99 budget for a typical e-commerce checkout
request:

Page 4 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Stage p50 p95 p99 Notes

DNS (first visit) 25ms 45ms 80ms Subsequent requests: 0ms from OS
cache
TLS 1.3 handshake 15ms 25ms 40ms With session ticket: 0ms; 0-RTT
possible
CDN edge (cache 3ms 8ms 20ms Cache hit: < 2ms from SSD
miss)
WAN transit (CDN → 30ms 60ms 90ms Depends heavily on geography
origin)
Load balancer 0.5ms 1ms 2ms HAProxy / ALB
API Gateway (JWT 3ms 6ms 12ms Cached public key; local validation
validation)
Service mesh sidecar 0.3ms 0.6ms 1ms Envoy filter chain
(ingress)
Order service 20ms 50ms 120ms DB read: 5ms, cache: 1ms
business logic
Inventory service call 8ms 18ms 40ms gRPC; same cluster
(sync)
Payment service call 15ms 35ms 80ms External PSP: 60ms p99
(sync)
DB write (order 3ms 8ms 20ms PostgreSQL fsync
commit)
Response 1ms 3ms 8ms Protobuf/JSON encoding
serialisation
TOTAL ~124ms ~259ms ~513ms Target: < 500ms p99

⚑ Principal Engineer Note

Every layer you add to a request path costs latency at every percentile and introduces a new failure
mode. Before adding a layer, calculate its latency contribution at p99 and p99.9, not just p50. A 5ms
average cost can be a 50ms p99 cost under load.

1.4 The Response Path

The response follows the reverse path through the same layers. Key considerations:
• Streaming responses — for large payloads, chunked transfer encoding allows the client to
start rendering before the full response is ready. HTTP/2 DATA frames enable this with
backpressure control.

Page 5 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

• Compression — gzip and Brotli compression are applied at the edge or origin. Brotli achieves
15–25% better compression than gzip at equivalent CPU cost. The client advertises support
via Accept-Encoding: br, gzip.
• CDN caching on response — the CDN caches the response based on Cache-Control
headers. The origin has full control over what gets cached and for how long.
• Connection reuse — HTTP/2 and HTTP/3 multiplex multiple requests over a single
connection. The connection is kept alive by PING frames.
• Head-of-line blocking — HTTP/1.1 has HOL blocking at the HTTP layer (only one request per
connection at a time). HTTP/2 solves this at the HTTP layer but not TCP. HTTP/3 (QUIC)
eliminates TCP HOL blocking entirely.

1.5 Failure Modes at Each Layer

Understanding failure modes is as important as understanding the happy path. A principal engineer
designs for failure at every layer:

Layer Failure Mode Client Experience Mitigation

DNS NXDOMAIN, Browser error: Low TTL on health-
timeout, stale 'DNS_PROBE_FINISHED_NXDOMAIN' checked records;
cache anycast DNS
TLS Certificate Browser: Automated renewal;
expired, wrong 'NET::ERR_CERT_DATE_INVALID' OCSP stapling; cert
hostname, monitoring
revoked
CDN Origin Stale content or 502/504 Origin shield; stale-
unreachable, while-revalidate; CDN
cache poisoning, failover
PoP failure
Load All backends Connection refused or timeout Multi-AZ LBs; health
Balancer unhealthy, LB check tuning; circuit
itself fails breaking
API Auth service 401/403/429 with Retry-After header Auth cache; fallback to
Gateway down, rate limit local validation; rate
limit headers
Service OOM, panic, 502 from LB; timeout Resource limits;
deadlock liveness probes;
graceful shutdown
Database Connection 500; stale data; timeout Connection pooling;
exhaustion, read replicas; query
replica lag, timeouts; retries
deadlock

Page 6 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 2: DNS Resolution — Deep Dive

DNS (Domain Name System) is the internet's distributed naming infrastructure. It translates human-
readable hostnames into network addresses. DNS is far more than a simple lookup—it is the first
tier of traffic management, geo-routing, failover, and content delivery. A principal engineer must
understand DNS at the protocol level to diagnose resolution failures, tune TTLs for deployments,
and design resilient multi-region architectures.

2.1 The DNS Protocol

2.1.1 Message Format
DNS messages are transmitted over UDP port 53 (fallback to TCP for large responses > 512 bytes,
or always TCP for zone transfers). The DNS message format (RFC 1035):

2.1.2 The Full Resolution Chain (Recursive)

When a browser needs to resolve [Link] and no cache entry exists anywhere:
6. Stub resolver (OS) sends a recursive query to the configured resolver (from /etc/[Link] or
DHCP). Typically [Link], [Link], or the corporate DNS server.
7. Recursive resolver checks its own cache. On a miss, it begins iterative resolution.
8. Recursive resolver queries one of the 13 root name server clusters ([Link] through
[Link]). These are anycast clusters with hundreds of physical servers. The root
responds with a referral to the .com TLD servers.
9. Recursive resolver queries the .com TLD servers ([Link] through [Link]-
[Link]), which respond with the authoritative NS records for [Link].
10. Recursive resolver queries the authoritative name server for [Link] (e.g.,
[Link] for Route 53). This server responds with the actual A/AAAA record.
11. Recursive resolver caches the response with the TTL from the authoritative server and
returns it to the stub resolver.
12. OS resolver caches the result and returns it to the browser. The browser caches it for
min(TTL, browser-internal-max-TTL).

⚑ Deep Dive
The 13 root 'servers' are actually 13 anycast prefixes, each served by dozens to hundreds of
physical servers distributed across 6 continents. ICANN coordinates root zone updates. The root
zone itself is tiny (~2MB) and rarely changes. The real work is in TLD and authoritative servers.

2.1.3 DNS Caching Hierarchy

Page 7 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

DNS responses are cached at every level of the hierarchy. Understanding cache interaction is
critical for deployment planning:

Cache Level Location Controlled By How to Flush

Browser DNS Browser process Browser (Chrome: max chrome://net-internals/#dns
cache memory 60s) → Clear host cache
OS DNS cache OS resolver (nscd, OS TTL, /etc/hosts systemd-resolve --flush-
systemd-resolved) caches; ipconfig /flushdns
Router / CPE Home/office router ISP or router firmware Reboot router or wait for TTL
ISP recursive ISP data centre Record TTL Cannot flush externally; TTL
resolver controls staleness
Public resolver Google/Cloudflare Record TTL TTL controls; some
([Link]) PoP providers offer flush APIs
CDN DNS resolver CDN PoP CDN-internal TTL CDN console cache flush
Authoritative Your DNS provider You control the records Update the record; TTL
server (Route 53) controls propagation speed

2.2 DNS Record Types — Comprehensive Reference

Record Type RFC Description Production Use Case
A RFC Maps hostname to IPv4 address. [Link] → [Link]
1035 Multiple A records enable round-robin
load balancing.
AAAA RFC Maps hostname to IPv6 address. [Link] →
3596 Return both A and AAAA for dual-stack 2001:db8::1
clients.
CNAME RFC Canonical name alias. The DNS www → [Link] →
1035 resolver follows the chain. Cannot be at [Link]
zone apex.
ALIAS/ANAME Vendor- Behaves like CNAME but resolves at [Link] → [Link]-
specific query time; can be at zone apex. Route [Link]
53 ALIAS.
NS RFC Authoritative name servers for a zone. [Link] NS
1035 Delegation record. [Link]
MX RFC Mail exchanger. Priority value Priority 10: mail1, Priority 20:
1035 determines preference. mail2
TXT RFC Arbitrary text. Multiple strings SPF, DKIM, DMARC, ACME
1035 concatenated. challenge, domain
verification
SRV RFC Service location: _service._proto.name Kubernetes headless
2782 → priority weight port target. services, Consul, SIP, XMPP

Page 8 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Record Type RFC Description Production Use Case

PTR RFC Reverse DNS. IP → hostname. In- Required for SMTP
1035 [Link] zone. deliverability; security
auditing
CAA RFC CA Authorization. Restricts which CAs 0 issue '[Link]'; 0
6844 can issue certs for a domain. issuewild ''; 0 iodef
TLSA RFC DANE: pins a TLS certificate to a DNS Certificate pinning without
6698 name. Requires DNSSEC. browser trust store
DS RFC Delegation Signer. Links parent zone to DNSSEC chain of trust
4034 child zone DNSKEY for DNSSEC.
DNSKEY RFC Stores public key for DNSSEC zone ZSK and KSK for the zone
4034 signing.
SOA RFC Start of Authority. Metadata about the Required first record in every
1035 zone: primary NS, admin email, serial, zone
refresh, retry, expire, minimum TTL.

2.3 DNS TTL Strategy — Production Playbook

2.3.1 TTL as a Traffic Management Knob
TTL (Time To Live) is the number of seconds a resolver may cache a DNS answer. It is the primary
lever for balancing resolution freshness against query volume and failover speed. Getting TTL
strategy wrong is a common source of prolonged outages.

Scenario Recommended Rationale

TTL
Normal production A record 300–3600s (5– Balance between cache efficiency and failover
60 min) speed
Pre-migration (1 week 300s (5 min) Start lowering TTL well before the change so
before cutover) caches expire before cutover
Day of migration cutover 60s Allow quick propagation of new record; accept
higher query volume
Post-migration (stable) 3600s Raise TTL after confidence in new config to
reduce query load
CDN CNAME (pointing to 300s CDN manages its own internal resolution and
CDN edge) anycast routing
Health-checked failover 30–60s Must be short enough that clients fail over within
record your RTO
Email MX records 3600s Email servers cache MX aggressively; slow
changes acceptable
TXT records (SPF, DKIM) 3600s Infrequently changed; long TTL acceptable

Page 9 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Scenario Recommended Rationale

TTL
Kubernetes in-cluster DNS 30s (CoreDNS Pod IPs change frequently; short TTL prevents
default) stale cache

⚑ Common Mistake
Engineers often change a DNS record and wonder why clients are still hitting the old server 2 hours
later. The answer: the previous TTL was 7200s. To avoid this, lower TTL to 60s at least 2 × (current
TTL) before the cutover. This ensures all resolvers that cached the record at the start of the grace
period have also expired it.

2.4 Advanced DNS Features

2.4.1 DNS-Based Traffic Management (Route 53)
AWS Route 53 supports sophisticated routing policies that operate at the DNS layer—before a
single TCP packet reaches your infrastructure:

Routing Mechanism Use Case Limitation

Policy
Simple Returns single Single endpoint, no No health checking; no weighted
record failover needed splits
Weighted Returns records Canary deployments Sticky sessions not guaranteed;
with statistical (10/90 weight split); A/B weights are probabilistic per query
probability testing at DNS level
proportional to
weights
Latency-based Returns record for Multi-region APIs; Based on resolver IP, not client IP;
region with lowest global user base VPN users may get wrong region
measured latency
to resolver IP
Failover Active-Passive: DR failover; hot Binary switch; no traffic shaping
primary always standby during failover
used; secondary
only when primary
fails health check
Geolocation Returns record Content localisation; Resolver IP ≠ client IP; may need
based on regulatory compliance catch-all 'default' record
continent/country (GDPR, data
of resolver IP residency)
Geoproximity Like geolocation Traffic shifting between More complex to reason about
but with adjustable regions for cost than pure geolocation
bias radius optimisation

Page 10 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Routing Mechanism Use Case Limitation

Policy
Multivalue Returns up to 8 Simple HA without a Client behaviour with multiple
Answer records; each load balancer for small- records is not standardised
health-checked; scale services
client performs
client-side load
balancing
IP-based Returns record Corporate network Requires knowing client CIDR
based on client routing; ISP peering blocks in advance
CIDR block (not optimisation
resolver IP)

2.4.2 DNSSEC — DNS Security Extensions

DNSSEC adds cryptographic authenticity and integrity to DNS responses, preventing cache
poisoning and man-in-the-middle attacks. However, it does not provide confidentiality—DNS
queries and responses are still plaintext (see DNS over HTTPS/TLS for that).
DNSSEC chain of trust:
13. ICANN signs the root zone with the Root KSK (Key Signing Key). The root zone KSK public
key is hardcoded into all validating resolvers.
14. The root zone publishes DS (Delegation Signer) records for TLD zones, signed by the root
zone.
15. Each TLD (.com, .org) publishes DS records for delegated second-level domains.
16. The authoritative server for [Link] signs all its records with its Zone Signing Key
(ZSK), and publishes its DNSKEY.
17. Validating resolvers verify the entire chain from root to the leaf record.
DNSSEC record types:
• DNSKEY — stores the public key used to sign zone records. Two key types: ZSK (Zone
Signing Key, rotated frequently) and KSK (Key Signing Key, rotated rarely).
• RRSIG — Resource Record Signature. Every signed record set has an RRSIG covering it.
• NSEC/NSEC3 — authenticated denial of existence. Proves that a record does NOT exist
without allowing zone enumeration (NSEC3 uses hashing).
• DS — Delegation Signer. Published in the parent zone; links to the child zone's KSK.

2.4.3 DNS over HTTPS (DoH) and DNS over TLS (DoT)
Traditional DNS queries are plaintext UDP/TCP on port 53, visible to anyone on the network path
(ISP, corporate firewall, coffee shop router). DoH and DoT encrypt DNS queries:
• DNS over TLS (DoT, RFC 7858) — wraps DNS in TLS on port 853. Easy to block by firewall;
provides confidentiality and integrity.

Page 11 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

• DNS over HTTPS (DoH, RFC 8484) — DNS queries inside HTTPS on port 443.
Indistinguishable from regular HTTPS traffic; very hard to block. Used by browsers (Chrome,
Firefox) with trusted resolvers (Cloudflare [Link], Google [Link]).
Implication for enterprise: DoH in browsers may bypass corporate DNS resolvers (which often
enforce split-horizon DNS, content filtering, or DLP). Enterprises can configure their authoritative
DoH endpoint to reclaim resolution control.

2.5 Kubernetes DNS In Depth

2.5.1 CoreDNS Architecture
CoreDNS is the default DNS server for Kubernetes (since 1.11, replacing kube-dns). It runs as a
Deployment in kube-system and is exposed as a ClusterIP service (typically [Link]). The
kubelet configures each pod's /etc/[Link] at creation time:

2.5.2 DNS Query Flow Inside Kubernetes

When the payments pod calls [Link] the following happens:
18. The hostname 'inventory' has 0 dots. ndots:5 means the search path is tried first.
19. OS resolver sends query: [Link] → CoreDNS.
20. CoreDNS checks its Kubernetes cache (populated by watching the API server). It finds the
Service 'inventory' in namespace 'production' with ClusterIP [Link].
21. CoreDNS returns the A record: [Link] → [Link].
22. The OS resolver returns [Link] to the application. The application connects to
[Link]:80.
23. kube-proxy has programmed iptables/eBPF rules: DNAT [Link]:80 → one of the
inventory pod IPs:8080.
Cross-namespace calls require the full FQDN to avoid ambiguity and the 4-step search path
traversal:

⚑ Performance Tip
The ndots:5 setting causes up to 5 DNS lookups for every short name query. In a high-throughput
microservice making hundreds of service calls per second, this can add 5× the DNS query volume
to your CoreDNS pods. Use fully-qualified service names (with namespace) to reduce DNS queries
per request from 5 to 2.

2.5.3 Headless Services and SRV Records

A headless service (clusterIP: None) returns all Ready pod IPs as individual A records. DNS also
returns SRV records for each named port:

Page 12 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

StatefulSet pods always get the same ordinal hostname (kafka-0, kafka-1) even if rescheduled to a
different node. This stable identity is critical for Kafka's partition leadership assignments and
Zookeeper's quorum membership.

Page 13 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 3: TLS — Encrypting the Channel

Transport Layer Security (TLS) is the cryptographic protocol that provides confidentiality, integrity,
and authentication for communications over the internet. Every HTTPS request, every gRPC call
between services, and every connection to a database should use TLS. A principal engineer must
understand TLS at the handshake level, the certificate lifecycle, and the operational implications of
misconfigurations.

3.1 Cryptographic Foundations

3.1.1 Symmetric vs Asymmetric Cryptography
TLS uses a hybrid cryptographic approach: asymmetric cryptography (slow, used only for key
exchange and authentication) to establish a shared secret, then symmetric cryptography (fast) for
the bulk data encryption.

Property Asymmetric (RSA, ECDSA, ECDH) Symmetric (AES-GCM, ChaCha20-

Poly1305)
Key distribution Public key is freely shared; private Shared key must be securely
key is secret distributed first
Performance 1000× slower than symmetric Hardware-accelerated; ~10 Gbps on
modern CPUs
Use in TLS Key exchange (ECDH), Bulk data encryption after
authentication (ECDSA/RSA) handshake
Key sizes RSA 2048–4096 bits; EC 256–384 128–256 bits
bits
Forward secrecy RSA key exchange does NOT N/A (symmetric key is ephemeral by
provide FS design with ECDHE)

3.1.2 Key Exchange: Diffie-Hellman and ECDH

The Diffie-Hellman key exchange allows two parties to establish a shared secret over an insecure
channel without ever transmitting the secret. Elliptic Curve Diffie-Hellman (ECDH) achieves
equivalent security to DH with much smaller key sizes (256-bit EC ≈ 3072-bit DH).
The key exchange works on the mathematical property that (g^a)^b = (g^b)^a = g^ab mod p. Each
side generates an ephemeral key pair; they exchange public keys; they each compute the same
shared secret independently. An eavesdropper who sees both public keys cannot compute the
shared secret without solving the discrete logarithm problem.
Ephemeral key pairs (ECDHE) are generated fresh for each TLS session, providing Perfect
Forward Secrecy (PFS). Even if the server's long-term private key is later compromised, recorded
past sessions cannot be decrypted because the ephemeral keys were discarded after the session.

Page 14 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

3.2 TLS 1.3 Handshake — Step-by-Step

3.2.1 Full Handshake (1-RTT)
TLS 1.3 (RFC 8446) reduced the handshake from 2 round-trips (TLS 1.2) to 1, eliminated obsolete
cipher suites, and made forward secrecy mandatory.

3.2.2 Session Resumption and 0-RTT

TLS 1.3 introduces pre-shared key (PSK) resumption. After a full handshake, the server issues a
NewSessionTicket containing a PSK (encrypted session state). On the next connection, the client
presents the PSK to skip the key exchange:
• 1-RTT resumption — client presents PSK in ClientHello. Server can verify and skip certificate
exchange. Saves one round-trip vs full handshake.
• 0-RTT Early Data — client sends application data in the first packet alongside the ClientHello,
using the PSK-derived key. TTFB drops to zero additional round-trips.

⚑ Security Warning
0-RTT early data is vulnerable to replay attacks. An attacker who captures the first packet can
replay it to any server in a cluster. Never use 0-RTT for POST, PUT, DELETE, or any operation that
is not idempotent. Reject early data for any state-changing endpoint. The server must signal
acceptance of early data with an early_data extension in EncryptedExtensions.

3.2.3 Cipher Suites in TLS 1.3

TLS 1.3 dramatically simplified cipher suite selection by eliminating all weak and legacy options.
Only 5 cipher suites are defined, all providing AEAD (Authenticated Encryption with Associated
Data):

Cipher Suite AEAD Hash Notes

Algorithm
TLS_AES_256_GCM_SHA384 AES-256 in SHA-384 Preferred; hardware-
GCM mode accelerated on AES-NI
CPUs
TLS_CHACHA20_POLY1305_SHA256 ChaCha20- SHA-256 Better performance on
Poly1305 devices without AES
hardware (mobile, IoT)
TLS_AES_128_GCM_SHA256 AES-128 in SHA-256 Faster than 256-bit AES;
GCM mode adequate for most use
cases
TLS_AES_128_CCM_SHA256 AES-128 in SHA-256 For constrained
CCM mode environments (IoT)

Page 15 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Cipher Suite AEAD Hash Notes

Algorithm
TLS_AES_128_CCM_8_SHA256 AES-128 in SHA-256 Reduced authentication
CCM-8 mode tag; constrained IoT only

3.3 X.509 Certificates

3.3.1 Certificate Structure
An X.509 v3 certificate is an ASN.1-encoded data structure containing:

Field Description Example

Version Always v3 for modern certs 3
Serial Number Unique number from CA; used 0x0A4123B8C9D2...
for revocation
Signature Algorithm Algorithm CA used to sign this sha256WithRSAEncryption or ecdsa-
cert with-SHA256
Issuer CA's Distinguished Name (DN) CN=Let's Encrypt R3, O=Let's Encrypt,
C=US
Validity (Not Before) Cert becomes valid at this time 2024-01-01T00:00:00Z
Validity (Not After) Cert expires at this time 2024-03-31T23:59:59Z
Subject The entity the cert is issued to CN=[Link]
Subject Public Key Info The public key and algorithm EC public key (prime256v1)
Extensions: SAN Subject Alternative Names — all DNS:[Link],
valid hostnames for this cert DNS:*.[Link]
Extensions: Key Usage What the key can be used for Digital Signature, Key Encipherment
Extensions: Extended More specific usage constraints TLS Web Server Authentication, TLS
Key Usage Web Client Authentication
Extensions: OCSP URL for certificate revocation [Link]
Stapling status
Extensions: Certificate Policies governing this cert; Policy OID [Link].2.1 (Domain
Policies includes CP and CPS URLs Validated)
Signature CA's signature over all of the 256-byte RSA signature or 64-byte
above ECDSA signature

3.3.2 Certificate Chain Validation

When a client receives a server certificate, it must validate the entire chain from the leaf certificate
to a trusted root:

Page 16 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

24. Signature verification — verify the leaf certificate's signature using the intermediate CA's
public key.
25. Intermediate validation — verify the intermediate certificate's signature using the root CA's
public key.
26. Root trust — check that the root CA certificate is in the operating system's or browser's trust
store.
27. Validity period — check that each certificate in the chain is not yet expired and not yet valid
(not before).
28. Key Usage and Extended Key Usage — verify the certificate is authorised for TLS server
authentication.
29. Subject Alternative Name match — verify the server's hostname matches one of the SANs
(exact match or wildcard).
30. Revocation check — check OCSP (Online Certificate Status Protocol) or download the CRL
(Certificate Revocation List). OCSP Stapling improves performance by having the server pre-
fetch and cache the OCSP response.
31. CAA check (optional, at HTTPS level) — browsers do not check CAA; CAs check CAA
before issuance. CAA records restrict which CAs can issue for a domain.

3.3.3 OCSP Stapling

OCSP (Online Certificate Status Protocol) allows a client to check whether a certificate has been
revoked. Without stapling, the client must make an additional HTTP request to the CA's OCSP
responder for every TLS connection—adding latency and creating a privacy issue (the CA learns
which websites you visit).
OCSP Stapling resolves this: the server periodically fetches a signed OCSP response from the CA
and 'staples' it to the TLS handshake in the CertificateStatus message. The client receives the
revocation status in the handshake itself, with no additional round-trip.

3.4 Certificate Lifecycle Management

3.4.1 The ACME Protocol
ACME (Automatic Certificate Management Environment, RFC 8555) is the protocol used by Let's
Encrypt and other CAs to automate certificate issuance and renewal without human intervention.
The ACME flow for an HTTP-01 challenge:
32. The ACME client (certbot, cert-manager) generates a key pair for the account and registers
with the ACME server ([Link]
33. The client requests a certificate for the domain [Link]. The ACME server
responds with a list of challenges.
34. The client chooses HTTP-01. The server provides a token (random string).
35. The client places the token at [Link]
with content: <token>.<key_authorisation_thumbprint>.

Page 17 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

36. The client notifies the ACME server to validate. The server fetches the URL and verifies the
content.
37. The server marks the challenge as valid. The client submits a CSR (Certificate Signing
Request) with the public key.
38. The ACME server issues the certificate and returns it. The client stores the certificate and
private key.
39. Before expiry (typically 30 days before the 90-day Let's Encrypt cert expires), the process
repeats automatically.

3.4.2 cert-manager in Kubernetes — Complete Configuration

cert-manager is the de facto standard for certificate automation in Kubernetes. It watches Certificate
custom resources and manages the full lifecycle including ACME challenges, renewal, and storage
in Kubernetes Secrets.

3.5 TLS in the Service Mesh

3.5.1 Internal Certificate Authority
In a service mesh, every workload needs a short-lived certificate for mTLS. Istio's istiod acts as an
internal CA, issuing certificates with 24-hour TTLs. These certificates are not trusted by the public
internet—they use a private root CA.
The Istio certificate hierarchy:

3.5.2 Certificate Rotation

Istio sidecars automatically rotate workload certificates before they expire. The rotation process:
40. The Envoy sidecar (via the SDS - Secret Discovery Service API) watches for certificate
expiry.
41. When the certificate is within a configurable percentage of its lifetime (default: 50%), the
sidecar requests a new certificate from istiod.
42. istiod generates a new key pair inside the pod (via the Envoy SDS protocol), creates a CSR,
signs it, and returns the new certificate.
43. Envoy atomically replaces the old certificate with the new one for new TLS connections.
Existing connections use the old certificate until they close naturally.
This rotation happens without pod restarts, application code changes, or manual intervention. The
24-hour TTL ensures that a compromised certificate has a maximum blast radius of 24 hours.

Page 18 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 4: Content Delivery Networks —

Architecture and Operations
A Content Delivery Network is a globally distributed platform of servers designed to serve content
with high performance, availability, and security. Modern CDNs are not merely caches—they are
comprehensive edge computing platforms that perform TLS termination, run application logic,
enforce security policies, and absorb volumetric attacks, all before a single packet reaches your
origin infrastructure.

4.1 CDN Architecture

4.1.1 Points of Presence (PoPs)
A CDN operates hundreds to thousands of PoPs (Points of Presence) distributed across internet
exchange points (IXPs), carrier-neutral colocation facilities, and ISP networks. Each PoP contains:
• Edge servers — high-memory servers with large SSD caches (tens to hundreds of terabytes
per PoP). Serve cached content directly.
• TLS termination hardware — often SmartNICs or dedicated hardware SSL offload cards that
handle the asymmetric cryptography of TLS handshakes at line rate.
• Anycast BGP announcements — all PoPs announce the same IP prefix via BGP. Client DNS
queries and TCP connections are automatically routed to the geographically nearest PoP by
BGP path selection.
• WAN accelerators — proprietary protocols between PoP and origin (Cloudflare Argo, Fastly
Signal Sciences) that optimise the PoP-to-origin path using persistent connections,
multiplexing, and optimal routing.

4.1.2 Request Flow Through the CDN

A cache-miss request flows through three tiers in a typical CDN:
44. Edge PoP (Tier 1) — the client's nearest PoP. Checks L1 cache (DRAM). Miss → checks L2
cache (SSD). Miss → forwards to origin shield.
45. Origin Shield (Tier 2) — a designated PoP (or regional cluster) that aggregates misses from
all edge PoPs before hitting the origin. Prevents thundering herd: if 10,000 users
simultaneously request an uncached object, only one request reaches the origin; the other
9,999 wait for the shield to populate and then serve from shield cache.
46. Origin — your infrastructure. The CDN maintains a persistent connection pool to the origin to
avoid TCP/TLS handshake overhead on each cache miss.

4.2 Caching Mechanics

Page 19 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

4.2.1 Cache Hierarchy and Object Lifetime

The lifetime of a cached object is governed by Cache-Control headers in the origin response. CDN
interpretation:

4.2.2 Cache Key Design

The cache key is the hash used to look up cached responses. The default cache key is the URL
(scheme + host + path + query string). Incorrect cache key design leads to:
• Cache pollution — too many unique cache keys; effective cache size is zero.
• Cache poisoning — an attacker injects a malicious response under a key that other users will
hit.
• Incorrect personalisation — caching personalised content and serving it to the wrong user.
Cache key optimisation patterns:

Pattern Technique Example Risk

Strip tracking Remove UTM params, /page?utm_source=email Must not strip params
parameters ad click IDs from cache → /page that affect content
key
Include Vary Cache separate copies Vary: Accept-Language, Multiplies cache
header values per Accept-Language, Accept-Encoding entries by number of
Accept-Encoding values
Normalise query Sort query params ?b=2&a=1 treated same Requires CDN
string alphabetically before as ?a=1&b=2 configuration
hashing
Custom cache key Include X-Device-Type X-Device-Type: mobile Must be set upstream,
header in cache key for → separate cache entry before CDN
responsive pages
Ignore Cookie Do not include Cookie in Strip Cookie before Must verify no
header cache key for public cache lookup personalised content is
content cached

4.2.3 Cache Invalidation Strategies

Cache invalidation is the hardest problem in CDN management. Each strategy has different trade-
offs:

Strategy How it Works Speed Granularity Use Case

URL purge Purge a specific < 1s Single URL Immediate correction of
URL from all edge (Cloudflare/Fastly) specific cached content
PoPs
Surrogate- Tag responses < 1s for Fastly; 1– Logical Product page update:
Key / Cache- with logical keys; 5s for others group purge all URLs tagged
Tag purge 'product-123'

Page 20 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Strategy How it Works Speed Granularity Use Case

purge all tagged
objects at once
Prefix purge Invalidate all URLs 5–30s depending Path prefix Deploy: purge all
matching a path on provider /static/v2.3/ URLs
prefix
Full cache Invalidate all 30s–5 min All content Major incident recovery;
flush cached content in avoid in production
all PoPs
TTL-based Wait for natural Minutes to hours Passive Normal content refresh;
expiry TTL expiry low-urgency changes
Stale-while- Serve stale Zero client impact Automatic Non-critical content that
revalidate immediately; can briefly be stale
refresh in
background
Cache Change URL when Immediate (new Per asset Immutable assets: JS,
busting via content changes URL = cold CSS, images — set TTL
URL (/static/[Link]) cache) to 1 year
versioning

⚑ Principal Engineer Pattern

The optimal caching architecture for a modern web app: (1) API responses: short TTL (60–300s)
with surrogate-key invalidation. (2) HTML pages: stale-while-revalidate with 5-minute TTL. (3) Static
assets (JS/CSS): content-addressed URLs (hash in filename) with 1-year TTL. This eliminates the
need to ever invalidate static assets—changing the file changes the URL.

4.3 CDN Security

4.3.1 DDoS Mitigation Architecture
CDNs are the first and most effective defence against volumetric DDoS attacks. A well-architected
CDN can absorb terabits per second of attack traffic through several mechanisms:
• Anycast absorption — attack traffic is distributed across all PoPs globally. A 1 Tbps attack
hitting 200 PoPs = 5 Gbps per PoP, well within capacity.
• BGP blackholing — for extreme attacks, the CDN can announce a more-specific route that
drops traffic in the ISP's core network, before it reaches PoP infrastructure.
• Layer 3/4 rate limiting — hardware-based rate limiting at line rate; IP reputation blocking; TCP
SYN proxy to absorb SYN floods without the origin seeing them.
• Layer 7 DDoS — HTTP floods, Slowloris attacks, amplification via large responses. Mitigated
by request rate limiting, connection limits per IP, and challenge pages.
• Challenge pages — JavaScript challenge (requires a JavaScript engine; bots without
browsers fail), CAPTCHA, or device fingerprinting to distinguish human browsers from bots.

Page 21 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

4.3.2 Web Application Firewall (WAF)

A WAF operates at Layer 7, inspecting HTTP request and response content for malicious patterns.
CDN-integrated WAFs operate at the edge, blocking attacks before they consume origin resources.
WAF rule categories:

Category Attack Type Example Signature False Positive Risk

OWASP SQL Injection ' OR 1=1--; UNION Medium — legitimate search
A03 SELECT; DROP TABLE queries may contain SQL
keywords
OWASP XSS (Cross-Site <script>, javascript:, Medium — JSON APIs returning
A03 Scripting) onerror=, onload= < or > in data
OWASP Path Traversal ../../../etc/passwd; Low — file path in URL is
A01 ..%2F..%2F uncommon in REST APIs
OWASP Command ;id;, |whoami|, $(curl) Low — rare in well-designed
A05 Injection APIs
OWASP Server-Side [Link] (EC2 Low — metadata IP is never
A08 Request Forgery metadata), [Link] gopher:// legitimate from external clients
OWASP XML External <!ENTITY xxe SYSTEM Low — only relevant if accepting
A04 Entity (XXE) '[Link] XML input
Rate-based Brute force, > 100 login attempts per IP Medium — shared egress IPs
credential stuffing per minute (NAT, corporate proxy)
Geo- Traffic from Source IP in OFAC/EU- Medium — VPNs and Tor exits
blocking sanctioned blocked country list may appear in blocked regions
countries

4.3.3 Bot Management

Modern bot management goes far beyond IP reputation lists. Sophisticated bot detection uses:
• Browser fingerprinting — TLS fingerprint (JA3/JA4), HTTP/2 fingerprint, browser canvas
fingerprint, JavaScript engine behaviour. Bots using headless Chrome often have detectable
differences from real Chrome.
• Behavioural analysis — real users move mice, have irregular typing patterns, scroll, and take
variable time between actions. Bots have regular, programmatic behaviour.
• Device attestation — challenge pages issue challenges that require a real browser to solve
(computing a proof-of-work, executing obfuscated JavaScript).
• Machine learning scoring — request features (rate, timing, fingerprint, header order, user
agent) are scored by an ML model. High-confidence bots are blocked; borderline cases get a
challenge.

Page 22 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

4.4 Edge Computing

4.4.1 Cloudflare Workers
Cloudflare Workers run JavaScript (V8 isolates, not [Link]) at every Cloudflare edge PoP. Each
isolate starts in sub-millisecond time, with no container overhead. Workers can:
Use cases where edge workers dramatically reduce cost and latency:
• A/B testing and feature flags — vary content based on cookie or header without an origin
round-trip.
• Geofencing and compliance — block or redirect users from specific regions based on
[Link] header.
• Request coalescing — deduplicate simultaneous cache-miss requests for the same resource.
• Response transformation — add security headers (CSP, HSTS, X-Frame-Options) globally.
• Dynamic personalisation — modify HTML at the edge to inject personalised content (product
recommendations, pricing) from KV store without origin latency.

Page 23 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 5: Load Balancers — L4/L7 Deep Dive

Load balancers are the core mechanism for achieving horizontal scalability and high availability.
They distribute traffic across a pool of backends, remove unhealthy nodes from rotation, and enable
zero-downtime deployments. Understanding the differences between L4 and L7 load balancing, the
algorithms for distributing load, and the operational nuances of health checking and draining is
fundamental to designing reliable systems.

5.1 Network Model: L4 vs L7

5.1.1 L4 Load Balancing (Transport Layer)
An L4 load balancer (also called a TCP/UDP load balancer) operates on network packets without
inspecting the application protocol. It makes routing decisions based only on:
• Source IP address and port
• Destination IP address and port
• TCP connection state (SYN, established, FIN)
L4 LBs use techniques like NAT (Network Address Translation) or DSR (Direct Server Return) to
forward packets. Because they do not terminate TCP connections or inspect application data, they
have extremely low latency (sub-millisecond) and can handle millions of connections per second on
commodity hardware.
L4 limitations:
• Cannot route based on HTTP host header, URL path, or HTTP method.
• Cannot inspect or validate TLS (passthrough only) — cannot offload TLS.
• Cannot provide HTTP-level features like sticky sessions via cookies, request deduplication, or
response buffering.

5.1.2 L7 Load Balancing (Application Layer)

An L7 load balancer (also called an application load balancer or reverse proxy) terminates TCP and
TLS connections, reads the complete HTTP request, and makes routing decisions based on
application-level content. This requires more CPU and memory but enables a vastly richer feature
set.

Capability L4 L7 Notes
Routing basis IP + Port URL path, L7 enables content-based routing
hostname,
headers,
method, body,
cookies
TLS termination No (passthrough Yes (offload to L7 LB handles cert management
only) LB)

Page 24 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Capability L4 L7 Notes
HTTP/2 and gRPC Pass-through Full support with L7 required for gRPC routing
only multiplexing
WebSocket Pass-through Support with L7 must handle Upgrade header
connection
upgrade
Sticky sessions IP hash (coarse) Cookie-based L7 enables application-level stickiness
(precise per-
user)
Rate limiting No Yes (per-IP, per- L7 sees headers and URLs
user, per-path)
Authentication No JWT validation, L7 can enforce auth policy
OIDC, API keys
Response No Yes (slow client L7 decouples slow clients from
buffering protection) backends
Health check depth TCP connect HTTP 200 from L7 verifies application health
/healthz
Latency overhead < 0.1ms 0.5–5ms L7 reads and parses HTTP
Throughput Millions of Hundreds of L4 has higher raw throughput
conn/s thousands of
req/s

5.2 Load Balancing Algorithms — In Depth

5.2.1 Round Robin
The simplest algorithm: requests are distributed sequentially. Backend 1 gets request 1, Backend 2
gets request 2, ..., Backend N gets request N, then back to Backend 1. Time complexity: O(1).
When to use: homogeneous backends with uniformly short requests. Fails when requests have
variable processing times — one long-running request on Backend 1 causes it to accumulate queue
depth while others are idle.

5.2.2 Weighted Round Robin

Like round robin but backends have weights proportional to their capacity. A backend with weight 3
gets 3× as many requests as a backend with weight 1. Used for:
• Heterogeneous hardware — some nodes have more CPU/memory.
• Canary deployments — 95 weight on v1, 5 weight on v2 routes 5% of traffic to v2.
• Gradual instance warm-up — new instances start with low weight (0 → 10 → 50 → 100) to
avoid sending full traffic to a cold JVM or Python process with empty caches.

Page 25 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

5.2.3 Least Connections

Each new request is routed to the backend with the fewest active connections. For short,
homogeneous requests this is equivalent to round robin. For variable-duration requests (long-
polling, file uploads, video streaming) it provides significantly better load distribution.
Implementation: the LB maintains a connection counter per backend. Increment on connection
open, decrement on close. O(N) to find minimum unless a priority queue is maintained (O(log N)
updates).

5.2.4 Least Request (Power of Two Choices)

Used by Envoy and Nginx Plus. Instead of tracking all backends, randomly sample two backends
and send the request to whichever has fewer active requests. This provides near-optimal load
distribution with O(1) selection:

5.2.5 Consistent Hashing

Consistent hashing maps requests to backends based on a hash of a stable attribute (user ID,
session cookie, customer ID). The same attribute always maps to the same backend (assuming the
backend is healthy). When backends are added or removed, only 1/N of the keys need to be
remapped (N = number of backends).
Implementation: backends are placed at virtual positions on a hash ring (typically by hashing their
IP:port). Each key is hashed to the ring; the request goes to the first backend clockwise from the
key's position. Multiple virtual nodes per backend prevents uneven distribution.
Use cases:
• Session affinity without shared state — user requests always hit the same backend's local
cache.
• Distributed caching — consistent hashing ensures cache shards are stable during scaling
events.
• gRPC streaming — streaming connections must be maintained to the same backend.

5.3 Health Checking — Production Configuration

5.3.1 Health Check Types and Configuration
Health checking removes unhealthy backends from the load balancing pool automatically. The
sensitivity of health checking directly controls your MTTR (Mean Time to Recovery) during backend
failures.

Page 26 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Parameter Description Production Value Effect of Too Aggressive

interval Time between health 10–30 seconds Increased load on
check probes backend; faster detection
timeout Maximum time to wait for 5 seconds False positives if backend
health check response is slow under load
healthy_threshold Consecutive successes 2–3 Slower warm-up; prevents
before marking backend flapping
healthy
unhealthy_threshold Consecutive failures 2–3 Slower detection; too high
before marking backend = prolonged outage
unhealthy
health_check_path HTTP path to probe /readyz (not Wrong path misses app-
/healthz) level issues
expected_status HTTP status codes 200–299 200 only may miss
considered healthy redirect-based health
endpoints

5.3.2 Health Check Endpoint Design

The design of the health check endpoint is as important as the load balancer configuration. Three
types:

5.4 Connection Draining and Graceful Shutdown

Zero-downtime deployments require coordinated graceful shutdown between the application,
Kubernetes, and the load balancer. The failure to implement this correctly is one of the most
common causes of 502 errors during deployments.
The correct shutdown sequence:
47. Kubernetes sends SIGTERM to the container when a pod is being terminated (during rolling
update, scale-down, or node drain).
48. The application's SIGTERM handler sets a flag: 'I am shutting down'. The /readyz endpoint
immediately returns 503.
49. The load balancer and kube-proxy detect the 503 from /readyz. They stop routing new
requests to this pod.
50. The application waits for all in-flight requests to complete (or until
terminationGracePeriodSeconds is reached).
51. The application closes listener sockets and exits cleanly with exit code 0.
52. Kubernetes removes the pod from the Endpoints object. Any iptables/eBPF rules are
cleaned up.

Page 27 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

5.5 AWS Load Balancers — Production Reference

5.5.1 Application Load Balancer (ALB)
ALB is AWS's L7 load balancer. It supports HTTP/1.1, HTTP/2, WebSocket, and gRPC. Key
operational details:

Feature Detail Principal Engineer Consideration

Listener rules Up to 100 rules per listener; Rule ordering matters; most specific rules
evaluated in priority order; should have lower priority numbers (higher
default rule catches all priority)
Target types Instance (EC2), IP (ECS Use IP target type for EKS; avoids NodePort
Fargate, pods), Lambda, ALB hop
(chained)
Sticky sessions App-based (AWSALBAPP App-based stickiness preserves existing
cookie) or duration-based cookie TTL; more flexible
(AWSALB cookie)
Connection Deregistration delay: 0–3600s Set to match your p99 request duration plus
draining (default 300s) buffer; 30–60s typical for APIs
Access logs S3 delivery; 130+ fields Enable for compliance; use Athena for
including TLS version, cipher, analysis; costs ~$0.50/GB for S3 storage
target processing time
WAF integration Attach AWS WAF ACL Costs $5/month per WAF + $0.60 per million
directly; inspect all requests requests
Desync protection HTTP Desync mitigation; Enable in 'strictest' mode for public APIs; may
blocks HTTP request reject legacy clients
smuggling attacks
Request tracing X-Amzn-Trace-Id header Propagate this header to all downstream
injected; integrates with AWS services for end-to-end tracing
X-Ray

5.5.2 Network Load Balancer (NLB)

NLB is AWS's L4 load balancer designed for ultra-high throughput and sub-millisecond latency. Key
differentiators:
• Static IP addresses — one Elastic IP per AZ. Allows firewall whitelisting. ALBs have dynamic
IPs.
• Source IP preservation — client source IP is visible to the backend without X-Forwarded-For.
Essential for IP-based rate limiting, geo-blocking, and compliance logging.
• PROXY Protocol — NLB can prepend a PROXY protocol header to each TCP connection that
carries the original client IP (useful when source IP preservation is not natively supported by
the backend).
• High throughput — millions of connections per second per AZ; sustained bandwidth of
hundreds of Gbps per AZ.

Page 28 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

• TLS offload — NLB can terminate TLS (since 2019). Supports TLS 1.3. Integrates with ACM
for certificate management.
• Zonal DNS — each AZ has its own DNS name (e.g., [Link]).
Use for AZ-affinity in inter-service calls to reduce cross-AZ data transfer costs.

Page 29 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 6: API Gateway — The Front Door of

Your Platform
The API Gateway is the single entry point for all client traffic. It is the layer where cross-cutting
concerns—authentication, rate limiting, routing, transformation, observability—are centralised.
Without an API gateway, each microservice must independently implement these concerns, leading
to inconsistency, security gaps, and duplicated operational overhead.
A principal engineer must understand not only how to configure an API gateway but also the subtle
security implications of each feature, the performance characteristics of different implementations,
and the architectural trade-offs of gateway-heavy vs thin-gateway designs.

6.1 API Gateway Responsibilities

Responsibility Mechanism Without Gateway With Gateway
Authentication JWT validation, Each service Centralised, consistent;
API key lookup, reimplements auth; failed auth never reaches
OIDC session inconsistent enforcement services
Authorization RBAC policy Services check Policy enforcement at
evaluation, scope permissions gateway; services trust the
checking independently; gaps gateway
possible
Rate Limiting Token Each service implements; Global per-client limit; fair
bucket/leaky no global limit use enforcement
bucket counters
Request Routing Path-based, host- Clients must know service Clients call one endpoint;
based, header- addresses gateway routes internally
based routing
SSL/TLS Certificate Each service manages its One cert at gateway;
Termination management, own cert internal traffic can be
protocol handling plaintext or mTLS
Request/Response Header injection, Clients must handle all Standardise API; adapt
Transform body rewrite, service quirks legacy services
protocol translation
Observability Metrics, structured Each service sends Consistent metrics: every
logs, distributed separate data request logged and traced
traces
API Versioning Path prefix (/v1/, Services manage own Gateway routes v1 → old
/v2/), header- versioning service; v2 → new service
based routing
Circuit Breaking Fail-fast when Clients see long timeouts Gateway returns 503
backend unhealthy immediately on backend
failure

Page 30 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Responsibility Mechanism Without Gateway With Gateway

Caching Cache GET Each client caches Shared cache; reduced
responses; serve independently backend load
from cache

6.2 Authentication — Deep Technical Reference

6.2.1 JSON Web Tokens (JWT)
JWT is the dominant mechanism for stateless authentication in distributed systems. Understanding
its internals is critical for implementing it securely.
A JWT consists of three base64url-encoded JSON objects separated by dots:

6.2.2 JWT Validation — Complete Algorithm

Every step of JWT validation is a security control. Skipping any step creates a vulnerability:
53. Extract the JWT from the Authorization header. Reject requests without the header or with
malformed Bearer syntax.
54. Split the JWT on '.' separators. Reject tokens with ≠ 3 parts.
55. Base64url-decode the header. Parse as JSON. Reject if not valid JSON.
56. Read the 'alg' field. Reject 'none' algorithm entirely — this is the original JWT algorithm
confusion attack. Maintain an allowlist of accepted algorithms (RS256, ES256). Never accept
both RS256 and HS256 for the same key, as they use the key differently.
57. Read the 'kid' (key ID). Fetch the corresponding public key from the JWKS endpoint
([Link] Cache JWKS responses per the Cache-
Control header (typically 1 hour). Reject tokens with unknown kid.
58. Verify the signature using the fetched public key and the stated algorithm. The signature
covers the [Link] string (not the decoded objects). Reject on signature mismatch.
59. Decode and parse the payload. Verify 'exp' > current time. Reject expired tokens.
60. Verify 'nbf' ≤ current time. Reject tokens not yet valid.
61. Verify 'iss' matches the expected issuer exactly. Reject on mismatch.
62. Verify 'aud' contains the expected audience (your API's identifier). Reject on mismatch.
63. Check 'jti' against a seen-token cache (Redis set with TTL = token lifetime) to detect replay
attacks. This is optional but required for high-security applications.
64. Extract claims (sub, roles, scope, tenant_id) and inject them as X-User-* headers for
downstream services.

⚑ Security Alert

Page 31 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

The #1 JWT vulnerability is improper algorithm validation. If your verification code trusts the 'alg'
field from the token without cross-checking against your allowlist, an attacker can forge a token by
setting alg='none' and omitting the signature, or by switching from RS256 (asymmetric) to HS256
(symmetric) and signing with the server's public key as the HMAC secret.

6.2.3 JWKS Endpoint and Key Rotation

The JWKS (JSON Web Key Set) endpoint exposes the public keys used to verify JWTs. It returns a
JSON document:

6.3 Rate Limiting — Implementation Reference

6.3.1 Token Bucket Algorithm
The token bucket is the most widely used rate limiting algorithm. A bucket with capacity B holds
tokens; tokens are added at rate R per second. Each request consumes one token. When the
bucket is empty, requests are rejected.

6.3.2 Rate Limit Response Headers

The API gateway must include standard rate limit headers in every response to allow clients to self-
throttle:

6.3.3 Multi-Dimensional Rate Limiting

Production rate limiting requires multiple dimensions simultaneously:

Dimension Key Limit Example Purpose

Per IP address ip:<client_ip> 100 req/min Basic DDoS
protection;
anonymous user
throttle
Per user:<user_id> 1000 req/min Prevent one user from
authenticated monopolising capacity
user
Per API key / tenant:<tenant_id> 10000 req/min Enforce plan-based
tenant quotas
Per endpoint endpoint:<path>:<method>:<user> 10 req/min for Protect expensive
POST /orders operations
Per geographic region:<country_code> 50000 req/min Traffic shaping for
region from CN compliance or cost

Page 32 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Dimension Key Limit Example Purpose

Global backend global:<service_name> 50000 req/s total Protect backend from
protection total overload
regardless of source

6.4 Request Routing

6.4.1 Path-Based Routing

6.5 API Gateway Products — Detailed Comparison

Product Architecture Strengths Weaknesses Best For
Kong Nginx + Lua Massive plugin Complex HA Large orgs with
Gateway (OpenResty); ecosystem (200+); setup; Lua diverse plugin
plugin-based; DB- battle-tested; high plugins require needs; existing
mode or DB-less performance expertise; Nginx expertise
(declarative) Enterprise
features locked
behind paid tier
AWS API Fully managed; Zero operational Cold starts with AWS-native;
Gateway regional; overhead; native IAM Lambda; limited serverless
(HTTP API) Lambda/ALB/VPC auth; $1/million routing logic; no architectures;
Link integrations requests request body simple REST APIs
inspection for
routing
Envoy + Envoy data plane; Best-in-class L7 proxy; High operational Kubernetes-
Contour Contour control gRPC-first; deep complexity; native; gRPC-
plane; Kubernetes integration YAML verbosity; heavy; teams with
Kubernetes-native Contour lags Envoy expertise
CRDs Envoy features
Traefik v3 Go binary; auto- Zero-config for Less mature Startups; K8s-first
discovery from Kubernetes; fast enterprise teams; fast setup
Docker/K8s iteration; excellent features than priority
labels; Let's docs Kong; limited
Encrypt built-in auth options
Apigee X Managed; policy- Enterprise analytics; Expensive; slow Enterprises
(Google based XML monetisation; to configure; exposing APIs to
Cloud) configuration; OpenAPI-first; multi- XML policies are partners/public;
developer portal cloud verbose API monetisation
included
Azure API Managed; policy Deep Azure Azure lock-in; Microsoft shops;
Management XML; Azure AD ecosystem integration; XML policies; Azure-native
hybrid cloud support architectures

Page 33 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Product Architecture Strengths Weaknesses Best For

integration; self- slower release
hosted gateway cadence

Page 34 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 7: Service Mesh — Istio, Envoy, and

mTLS
A service mesh is a dedicated infrastructure layer for managing service-to-service communication in
a microservices architecture. It addresses the fundamental problem that in a distributed system with
50+ services, each service needs to implement retry logic, circuit breaking, observability, mutual
authentication, and traffic management. Without a service mesh, this logic is duplicated in every
language-specific client library, leading to inconsistency, security gaps, and operational nightmares.
The service mesh moves this logic out of application code and into the infrastructure layer—
specifically into a sidecar proxy running alongside each service instance. This makes network
behaviour language-agnostic, centrally configurable, and uniformly observable.

7.1 Service Mesh Architecture

7.1.1 Data Plane vs Control Plane
A service mesh has two distinct planes:
• Data Plane — the set of sidecar proxies (Envoy) deployed alongside every pod. The data
plane handles actual traffic: it intercepts all inbound and outbound packets, applies policy,
collects metrics, and enforces mTLS. The data plane operates in the critical path of every
request.
• Control Plane — the management component (istiod in Istio) that configures the data plane.
The control plane translates high-level policies (VirtualService, DestinationRule,
AuthorizationPolicy) into Envoy xDS configuration and pushes it to each sidecar. The control
plane is NOT in the critical request path—it configures the data plane asynchronously.

7.1.2 How Traffic is Intercepted

The key to understanding how a service mesh works is the traffic interception mechanism. When a
pod is injected with the Istio sidecar, an init container (istio-init) runs before the application container
and modifies the pod's iptables rules:

7.2 Envoy Proxy — Internal Architecture

7.2.1 Envoy's Core Abstractions
Envoy's configuration model uses four core abstractions that map to the lifecycle of a connection:

Page 35 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Abstraction xDS API Description Example

Listener LDS (Listener Defines how Envoy accepts Listen on [Link]:15006;
Discovery Service) connections: IP, port, filter apply HTTP Connection
chain Manager filter
Route RDS (Route Discovery HTTP routing rules: match Match /api/v2/payments →
Service) path/header/method, forward route to cluster
to cluster 'payments|8080|production'
Cluster CDS (Cluster Upstream service definition: payments service:
Discovery Service) load balancing, circuit LEAST_REQUEST LB;
breaker, TLS settings max 100 connections;
mTLS with SPIFFE ID
Endpoint EDS (Endpoint Actual IP:port addresses for a [Link]:8080 (healthy,
Discovery Service) cluster's backends zone=us-east-1a),
[Link]:8080 (healthy)

7.2.2 Envoy Filter Chain

Every request processed by Envoy passes through a filter chain. Filters are composable
middleware that can inspect, modify, or terminate requests. The standard Istio sidecar filter chain:
65. metadata_exchange — exchanges workload metadata (service name, namespace, version)
between sidecars via HTTP headers. Used for telemetry attribution.
66. stats — emits per-request metrics (request count, latency, bytes, response code) to
Prometheus.
67. jwt_authn (optional) — validates JWTs in the Authorization header at the sidecar level.
68. ext_authz (optional) — calls an external authorisation service for fine-grained policy
decisions.
69. http_connection_manager — the core HTTP L7 filter. Handles HTTP/1.1 and HTTP/2;
applies routing; manages connections.
70. rbac — enforces Istio AuthorizationPolicy rules based on source principal (SPIFFE ID),
source namespace, HTTP method, and path.
71. router — forwards the request to the selected upstream cluster based on route match.

7.2.3 Circuit Breaker in Envoy

Envoy's circuit breaker (called 'outlier detection' in Envoy terminology) ejects unhealthy endpoints
from the load balancing pool based on observed error rates:

7.3 Istio Traffic Management

7.3.1 VirtualService — Comprehensive Reference

Page 36 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

A VirtualService defines the traffic routing rules for a host. It is evaluated by the sidecar proxy of the
CALLING service, not the receiving service. This is critical: the VirtualService is applied at the
egress of the caller.

7.4 Mutual TLS (mTLS) — Complete Implementation

7.4.1 SPIFFE Identity Model
mTLS in a service mesh requires every workload to have a cryptographic identity. Istio uses the
SPIFFE (Secure Production Identity Framework for Everyone) standard. Each pod's identity is
expressed as a SPIFFE URI embedded in the X.509 certificate's SAN field:

7.4.2 Certificate Issuance Flow

When a pod is created with the Istio sidecar, the certificate issuance process:
72. The Envoy sidecar starts and connects to the local SDS (Secret Discovery Service) Unix
socket exposed by the istio-agent process running in the same pod.
73. istio-agent reads the pod's Kubernetes service account token from the projected volume at
/var/run/secrets/[Link]/serviceaccount/token.
74. istio-agent generates an EC key pair locally (private key never leaves the pod).
75. istio-agent creates a CSR (Certificate Signing Request) with the SPIFFE ID in the SAN field.
76. istio-agent authenticates to istiod using the service account token (Kubernetes
TokenReview validation).
77. istiod signs the CSR with the mesh's intermediate CA and returns the certificate.
78. istio-agent provides the certificate to Envoy via the SDS API.
79. Envoy uses the certificate for all inbound and outbound TLS connections.
80. Before the certificate expires (default: 24 hours), istio-agent automatically initiates rotation
by repeating steps 3–8.

7.4.3 mTLS Modes

Mode Description Use Case Migration Path
DISABLE No TLS between Not recommended; only N/A
sidecars. Plain TCP. for debugging
PERMISSIVE Sidecar accepts both Migration phase: allows Enable globally; verify all
plaintext and mTLS. mixed mesh and non- services migrate; then switch
Default mode. mesh services to coexist to STRICT
STRICT Sidecar accepts Production target; zero- All callers must have sidecars;
ONLY mTLS. Plaintext trust enforcement use PeerAuthentication to
enforce per-namespace

Page 37 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Mode Description Use Case Migration Path

connections are
rejected.

Page 38 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 8: Kubernetes — Container

Orchestration
Kubernetes is the industry-standard platform for container orchestration. It provides the APIs,
controllers, and runtime infrastructure to run containerised workloads at scale with high availability,
efficient resource utilisation, and declarative configuration. A principal engineer must understand
Kubernetes deeply—not just how to use it, but how its control loops work, where it can fail, and how
to design workloads that operate reliably within its constraints.

8.1 Kubernetes Control Plane — Component Deep Dive

8.1.1 kube-apiserver
The API server is the heart of Kubernetes. Every cluster operation—from kubectl apply to a
controller watching for resource changes—flows through the API server. It is the only component
that communicates directly with etcd.
API server request processing pipeline:
81. Authentication — the API server identifies the caller using: client certificates (X.509), bearer
tokens (JWT or service account tokens), OIDC tokens, webhook token authentication, or
proxy auth. Anonymous requests are allowed only if explicitly enabled.
82. Authorization — after authentication, the API server checks whether the authenticated
identity is allowed to perform the requested operation (verb) on the specified resource (kind) in
the specified namespace. The default authorizer is RBAC; Webhook and ABAC are also
supported.
83. Admission Control — the request passes through a chain of admission controllers before
being persisted. Two types:
• MutatingAdmissionWebhook — can modify the resource (e.g., inject sidecar containers, add
default labels, inject secret references).
• ValidatingAdmissionWebhook — can accept or reject the resource (e.g., OPA Gatekeeper
policy enforcement, PodSecurity enforcement).
84. API object validation — the API server validates the resource against its OpenAPI schema.
85. Persistence — the resource is serialised and written to etcd. The API server then notifies
watchers via a list-watch mechanism.

8.1.2 etcd — The Source of Truth

etcd is a distributed key-value store using the Raft consensus algorithm. It stores all Kubernetes
cluster state: pod specs, service definitions, ConfigMaps, Secrets (encrypted), RBAC policies, and
custom resources.

Page 39 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Raft consensus ensures that writes are accepted only when a majority (quorum) of etcd members
acknowledge them. For a 3-member cluster, 2 acknowledgements are required; for 5 members, 3
are required.

Cluster Fault Tolerance Quorum Required Recommended For

Size
1 member 0 failures tolerated 1/1 Development only; no HA
3 members 1 failure tolerated 2/3 Standard HA production cluster
5 members 2 failures tolerated 3/5 Critical production; multi-AZ spread
7 members 3 failures tolerated 4/7 Rarely needed; write latency increases
with size

⚑ Production Warning
etcd is the most critical component in your Kubernetes cluster. Losing etcd quorum means the entire
cluster becomes read-only—no new pods can be scheduled, no changes can be made.
Requirements: NVMe SSDs (etcd is latency-sensitive; p99 write latency should be < 10ms),
dedicated nodes (never co-locate etcd with workloads), encrypted storage, continuous backups with
tested restoration procedure.

8.1.3 kube-scheduler — Scheduling Algorithm

The scheduler assigns unscheduled pods to nodes through a two-phase process:
Phase 1 — Filtering (Predicates): eliminate nodes that cannot run the pod. Filters include:
• NodeResourcesFit — node must have enough allocatable CPU and memory for the pod's
requests.
• NodeAffinity — node must match the pod's required node affinity rules (matchExpressions on
node labels).
• PodAffinity / PodAntiAffinity — node must (or must not) have pods matching the affinity rules.
• TaintToleration — node's taints must be tolerated by the pod's tolerations.
• VolumeBinding — node must have the required PersistentVolumes available (or support
dynamic provisioning).
• NodeUnschedulable — skip nodes marked as unschedulable (cordon).
• TopologySpreadConstraint — enforce maximum skew between zones/nodes.

Phase 2 — Scoring (Priorities): score remaining nodes and pick the highest. Scorers include:
• LeastAllocated — prefer nodes with the most unused capacity (bin-spreading).
• MostAllocated — prefer nodes with the least unused capacity (bin-packing; reduces number of
nodes needed).
• BalancedResourceAllocation — prefer nodes where CPU and memory usage are balanced
relative to each other.

Page 40 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

• NodeAffinity — higher score for nodes that match preferred (not required) affinity rules.
• ImageLocality — prefer nodes that already have the pod's container image pulled.

8.2 Pod Lifecycle — Complete Reference

8.2.1 Pod Creation — Every Step

8.2.2 Probes — Complete Configuration Reference

Kubernetes probes determine pod health and readiness. Incorrect probe configuration is one of the
most common causes of production incidents.

Probe Type Purpose Failure Action Configuration Guidance

startupProbe Allow slow-starting Container Set failureThreshold *
containers to initialise restarted after periodSeconds to maximum
without being killed by failureThreshold expected startup time (e.g., 30 *
livenessProbe failures 10s = 5 minutes for JVM)
livenessProbe Detect Container killed Check ONLY process health.
stuck/deadlocked and restarted Never check dependencies here
processes that are — one flapping DB would restart
running but not making all pods.
progress
readinessProbe Determine if pod Pod removed from Check all dependencies. Return
should receive traffic Service Endpoints 503 if DB is unreachable, cache is
(not killed) unavailable, or circuit breakers
are open.

8.3 Resource Management

8.3.1 Requests, Limits, and Quality of Service
Understanding how Kubernetes manages CPU and memory resources is critical for production
stability:

QoS Class Condition Scheduling Behaviour Eviction Priority

BestEffort No requests or limits Lowest priority in Evicted first under memory
set on any container scheduling; no pressure
guaranteed resources
Burstable Requests < Limits, or Guaranteed minimum Evicted before Guaranteed;
only requests OR (requests); can burst to based on usage/request ratio
limits set limit

Page 41 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

QoS Class Condition Scheduling Behaviour Eviction Priority

Guaranteed Requests == Limits Dedicated resource Evicted last; only if BestEffort
for all containers reservation; never and Burstable are all evicted
(CPU and Memory) overcommitted

8.3.2 Vertical Pod Autoscaler (VPA)

VPA automatically adjusts CPU and memory requests/limits based on observed usage. It prevents
over-provisioning (cost waste) and under-provisioning (OOMKills, throttling). VPA has three modes:
• Off — compute recommendations but do not apply them. Use for initial tuning.
• Initial — apply recommendations only at pod creation. Running pods are not modified.
• Auto — evict and recreate pods when recommendations change significantly. Causes brief
disruption; not suitable for single-replica deployments.

8.4 Workload Controllers

8.4.1 Deployment Controller
The Deployment controller manages ReplicaSets and implements rolling updates. Understanding
the update algorithm is critical for zero-downtime deployments:

Page 42 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 9: Kubernetes Networking — CNI,

kube-proxy, and eBPF
Kubernetes networking is one of the most technically complex topics in the cloud-native ecosystem.
Unlike traditional networking where the topology is static and manually configured, Kubernetes
networking is dynamic—pods are created and destroyed constantly, IPs change, and the network
must automatically track these changes and route traffic correctly at all times.
This chapter explains, from first principles, how network packets travel between pods, how
Kubernetes Services work at the kernel level, and how modern CNI plugins like Cilium use eBPF to
replace iptables with faster, more flexible network programming.

9.1 The Kubernetes Networking Model

Kubernetes mandates a flat networking model with four fundamental requirements:
• Every pod gets a unique IP address within the cluster. No two pods share an IP, even across
nodes.
• Pods on different nodes can communicate directly using pod IPs, without NAT. The packet a
pod sends has the same source IP that the receiving pod sees.
• Nodes can communicate with pods using pod IPs directly.
• The IP address space for pods is separate from the host network but must be routable within
the cluster.
This flat model simplifies application design—services can bind to well-known ports without
collision, and there is no port-mapping configuration. The complexity is moved into the CNI
(Container Network Interface) plugin, which implements the networking model on each node.

9.2 Linux Networking Primitives

9.2.1 Network Namespaces
Linux network namespaces provide isolated network stacks. Each namespace has its own:
• Network interfaces (lo, eth0, veth pairs)
• IP routing table
• iptables/nftables rules
• Socket connections
• Network statistics
Every Kubernetes pod runs in its own network namespace. All containers within the same pod
share one network namespace (and therefore share the same IP, the same ports, and can
communicate via localhost). The pause container (also called the sandbox or infra container) is the

Page 43 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

container that holds the network namespace—it does nothing except exist, so the namespace
survives even if the application container is restarted.

9.2.2 veth Pairs

A veth (virtual ethernet) pair is a pair of virtual network interfaces linked together. What goes into
one end comes out the other. CNI plugins use veth pairs to connect a pod's network namespace to
the host's root namespace:

9.3 Cross-Node Pod Communication

9.3.1 VXLAN Overlay (Flannel)
VXLAN (Virtual Extensible LAN) is a Layer 2 encapsulation protocol that tunnels Ethernet frames
inside UDP packets. Flannel uses VXLAN to create an overlay network where pod IPs are
reachable across nodes even when the underlying network doesn't know about pod CIDRs.

9.3.2 Native BGP Routing (Calico)

Calico can operate in native routing mode using BGP to advertise pod CIDRs directly to the
network, eliminating the VXLAN overlay and its overhead:

9.4 eBPF Networking with Cilium

9.4.1 eBPF Fundamentals
eBPF (extended Berkeley Packet Filter) is a revolutionary Linux kernel technology that allows safe,
sandboxed programs to run in the kernel without modifying kernel source code or loading kernel
modules. eBPF programs are attached to kernel hooks and executed when those hooks are
triggered (e.g., when a network packet arrives, when a system call is made, when a tracepoint
fires).
eBPF programs are verified by the kernel's eBPF verifier before execution, ensuring they cannot
crash the kernel, access invalid memory, or loop infinitely. They communicate with userspace via
eBPF maps (shared memory data structures).

9.4.2 How Cilium Replaces iptables

Cilium uses eBPF programs attached to the TC (Traffic Control) hook at the network interface level.
This allows Cilium to make forwarding decisions in the kernel, on the fastest path, without the
overhead of the iptables subsystem:

Page 44 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

9.4.3 Cilium Network Policies — L7 Aware

Unlike Kubernetes NetworkPolicy (which is L3/L4 only), Cilium extends network policy to Layer 7,
allowing policies based on HTTP method, URL path, gRPC service, and Kafka topic:

9.5 Kubernetes Services — Internal Implementation

9.5.1 kube-proxy iptables Mode — Detailed
When you create a Service with ClusterIP [Link] and port 80 targeting pods on port 8080,
kube-proxy creates these iptables rules:

9.5.2 EndpointSlices
Kubernetes 1.21+ replaced Endpoints with EndpointSlices. A single Endpoints object contained all
pod IPs for a service; as a service scales to hundreds of pods, this object becomes very large and
every pod IP change triggers a full update to all nodes.
EndpointSlices shard a service's endpoints into slices of up to 100 endpoints each. Each slice
update is smaller, reducing API server load and kube-proxy update time. For a service with 1,000
pod endpoints:
• Old: one 1,000-entry Endpoints object; every pod IP change = full 1,000-entry write and
broadcast
• New: 10 EndpointSlices of 100 entries; one pod IP change = update only the relevant 100-
entry slice

Page 45 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 10: Inter-Service Communication

Patterns
How services communicate with each other is the most consequential architectural decision in a
microservices system. Every communication pattern carries trade-offs in latency, consistency,
coupling, resilience, and operational complexity. A principal engineer must be able to choose the
right pattern for every interaction, understand the failure modes of each, and design systems that
remain correct under partial failure conditions.

10.1 Synchronous Communication

10.1.1 HTTP/REST — Production-Grade Implementation
REST over HTTP is the most common protocol for synchronous service-to-service communication.
Despite its ubiquity, there are many ways to implement it incorrectly in a microservices context.
Critical HTTP client configuration for service-to-service calls:

10.1.2 gRPC — Deep Technical Reference

gRPC is an open-source RPC framework from Google that uses HTTP/2 as its transport and
Protocol Buffers as its serialisation format. It is the preferred protocol for high-performance,
strongly-typed inter-service communication.
gRPC communication patterns:

Pattern Client Server Use Case Example

Streams Streams
Unary RPC Single Single Standard request- GetUser(UserRequest) →
request response response UserResponse
Server-side Single Stream of Long-running SearchProducts(Query) →
streaming request responses queries; large stream Product
datasets
Client-side Stream of Single Bulk uploads; stream Event →
streaming requests response aggregation BatchResult
Bidirectional Stream of Stream of Chat; real-time stream Message ↔ stream
streaming requests responses collaboration; live Message
feeds

10.1.3 gRPC Deadline Propagation

Page 46 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

One of gRPC's most important features is end-to-end deadline propagation. Unlike timeouts (which
reset at each hop), deadlines are absolute timestamps that propagate across the entire call chain.
When a deadline expires, all downstream work is automatically cancelled:

10.1.4 gRPC Interceptors (Middleware)

gRPC interceptors are middleware functions that wrap every RPC call. They are the idiomatic way
to add cross-cutting concerns (logging, tracing, auth, retry) to gRPC services without modifying
business logic:

10.2 Asynchronous Communication

10.2.1 Apache Kafka — Deep Dive
Kafka is a distributed commit log designed for high-throughput, durable, and replayable event
streaming. Unlike traditional message queues (RabbitMQ, SQS), Kafka retains messages after
consumption, enabling multiple consumers to read the same events independently and at different
rates.
Kafka core concepts:

Concept Description Principal Engineer Consideration

Topic A named, ordered, Choose topic names carefully—they are API contracts.
partitioned log of Use a schema registry (Confluent, Karapace) to enforce
records. Producers schema evolution.
append; consumers
read at their own pace.
Partition A topic is divided into N Partition count is mostly immutable after creation in older
ordered partitions. Kafka. Plan for growth. More partitions = more
Each partition is an parallelism but higher metadata overhead. 10–100
immutable, append- partitions per topic is typical.
only log stored on disk.
Partitions are the unit of
parallelism.
Offset A monotonically Offset commit strategy matters: auto-commit risks at-
increasing integer least-once; manual commit after processing is safer.
identifying each Commit after idempotent processing.
record's position within
a partition. Consumer
groups commit offsets
to track progress.
Consumer A named group of Scaling consumers: you can have at most as many
Group consumers that active consumers as partitions. Extra consumers sit idle.
together consume a Rebalancing on consumer add/remove causes a pause.
topic. Each partition is
assigned to exactly one
consumer in the group.

Page 47 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Concept Description Principal Engineer Consideration

Replication Each partition has 1 Use RF=3 for production. [Link]=2 ensures
Factor leader and RF-1 data is on 2 brokers before ack. This tolerates 1 broker
followers on different failure without data loss.
brokers. Followers
replicate from the
leader.
Retention Messages are kept for Infinite replay within retention window enables
a configurable time debugging, reprocessing, and consumer lag recovery. 7-
([Link]) or size day retention is typical; cost vs. reprocessing value
([Link]), trade-off.
regardless of
consumption.
Log For log-compacted Used for change data capture (CDC), materialised views,
Compaction topics, Kafka retains and event sourcing snapshots. The topic represents the
only the latest value per current state, not history.
key. Older values are
garbage collected.

10.2.2 Producer Configuration

10.2.3 The Saga Pattern — Implementation Patterns

The Saga pattern manages distributed transactions across multiple services without two-phase
commit. There are two saga orchestration styles:
Choreography-based Saga (event-driven):

10.3 Resilience Patterns

10.3.1 Circuit Breaker — State Machine

10.3.2 Bulkhead Pattern

The bulkhead pattern isolates resources by consumer category to prevent one slow consumer from
starving others. Named after the compartments of a ship's hull that prevent one flooded section
from sinking the whole ship.

Page 48 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 11: Secrets Management — Vault,

IRSA, and External Secrets
Secrets management is the practice of securely creating, storing, distributing, rotating, auditing, and
revoking sensitive credentials. It is one of the most critical—and most commonly mishandled—
aspects of production system security. A single leaked secret can result in a full database
compromise, customer data breach, or complete infrastructure takeover.
A principal engineer must design secrets management systems that are: secure by default,
operationally simple, automatically rotating, comprehensively audited, and recoverable from
compromise within minutes.

11.1 Threat Model for Secrets

Before choosing a secrets management approach, understand the threat model:

Threat Attack Vector Impact Mitigation

Secrets in Developer commits All secrets in the file Pre-commit hooks (detect-secrets,
source code .env file; Git history exposed; historical gitleaks); Git history scanning in CI;
exposed via GitHub commits reveal rotated never commit secrets
secrets too
Secrets in Developer bakes Image pushed to public Multi-stage builds; scan images for
container API key into registry; anyone can secrets (Trivy, Snyk); use build
images Dockerfile or ARG pull and inspect layers secrets (Docker buildkit --secret)
Secrets in Process listing (ps Any process on the Mount secrets as files in tmpfs
environment auxe), same node can read volumes; never put secrets in env
variables /proc/<pid>/environ, env vars of processes it vars in K8s
crash dumps, can inspect
debug endpoints
Secrets in Direct etcd access All cluster secrets Enable etcd encryption at rest with
etcd if encryption at rest exposed KMS provider; strict RBAC; use
(Kubernetes is not enabled; external secrets operator
Secrets) RBAC
misconfiguration
allowing Secret
reads
Secrets in Application logs Log aggregation Structured logging with secrets
logs printed API keys in systems (ELK, Splunk) scrubbing; never log Authorization
error messages or store secrets; log headers or request bodies without
request logs access is often less scrubbing
controlled than secret
stores
Long-lived Stolen from Credential valid until Short-lived dynamic credentials via
static memory, traffic manual rotation (often Vault; automatic rotation; minimum
credentials never)

Page 49 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Threat Attack Vector Impact Mitigation

interception, social 90-day rotation maximum for static
engineering creds
Over- Compromised Blast radius extends Least privilege; per-service
privileged service accesses beyond the credentials; Vault namespaces and
service secrets it doesn't compromised service policies
accounts need; lateral
movement

11.2 Kubernetes Secrets — Internals and Security

11.2.1 How Kubernetes Secrets Work Under the Hood
Kubernetes Secrets store base64-encoded values in etcd. Base64 is encoding, not encryption. The
security of Kubernetes Secrets entirely depends on:
• etcd encryption at rest — without this, any process with etcd access reads all secrets in
plaintext
• RBAC policies — who can get/list/watch Secrets
• Audit logging — who accessed what secret and when
• Network policy — which pods can reach the API server

11.2.2 etcd Encryption at Rest with KMS

11.3 HashiCorp Vault — Production Architecture

11.3.1 Vault Internals
Vault is an identity-based secrets management platform. Its security model is fundamentally
different from Kubernetes Secrets: all stored data is encrypted using envelope encryption with the
master key, access is governed by token-based policies, and every access is comprehensively
audited.

Component Description Production Configuration

Storage Where Vault persists Integrated Raft (embedded consensus; no external
Backend encrypted data. All data Consul dependency); 3 or 5 nodes
is encrypted before
writing.
Seal / Unseal Vault starts sealed. Auto-unseal with AWS KMS (seal stanza in [Link]);
Unsealing requires eliminates manual key holders
reconstructing the

Page 50 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Component Description Production Configuration

master key using
Shamir's Secret
Sharing.
Auth Methods How clients prove their kubernetes (pod service accounts), aws (IAM), oidc
identity to Vault (SSO), approle (CI/CD)
Secret Engines Plugins that generate kv-v2 (versioned KV), database (dynamic creds), pki
or store secrets (cert authority), aws (IAM creds), transit (encryption)
Policies HCL rules defining Separate policy per service; wildcards forbidden; path-
what an authenticated based and capability-based
token can do on which
paths
Leases All dynamic secrets Short TTLs (1h for DB creds); configure max_lease_ttl;
have a TTL (lease). monitor lease renewal failures
Clients renew or the
secret is auto-revoked.
Audit Devices Immutable audit log of File + syslog; forward to SIEM; alert on policy
every Vault operation violations

11.3.2 Vault Database Secret Engine — Dynamic Credentials

The database secret engine generates unique, short-lived database credentials for each application
instance. This is the gold standard for database authentication in a microservices environment.

11.3.3 Vault Agent Sidecar Injector

The Vault Agent Injector is a Kubernetes MutatingWebhook that automatically injects a Vault Agent
sidecar into annotated pods. The agent authenticates to Vault, retrieves secrets, writes them to a
shared in-memory volume, and continuously renews leases:

11.4 External Secrets Operator (ESO)

The External Secrets Operator synchronises secrets from external stores (Vault, AWS Secrets
Manager, GCP Secret Manager, Azure Key Vault, 1Password) into Kubernetes Secrets. This allows
applications to consume secrets as native Kubernetes Secrets while the actual storage and rotation
happens in the external system.

Page 51 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 12: Workload Identity and Pod

Authentication
Authentication in a microservices system occurs at multiple levels simultaneously: users
authenticate to services, services authenticate to each other, and pods authenticate to cloud
infrastructure. This chapter covers every authentication mechanism a principal engineer must
understand, with particular depth on the modern zero-trust workload identity patterns that replace
static credentials.

12.1 Kubernetes Service Accounts — Complete Reference

12.1.1 Service Account Token Lifecycle
Every pod in Kubernetes has a service account, and every service account has one or more tokens.
Understanding the token lifecycle is essential for designing secure workload authentication.
Kubernetes has two token mechanisms:

Token Type Kubernetes Characteristics Security Properties

Version
Legacy static < 1.21 Never expire; stored in Poor: long-lived, not audience-
tokens Secrets; mounted bound, not pod-bound.
automatically into all pods Compromise is permanent until
using the SA manual rotation.
Bound Service >= 1.21 Expire (default 1h); Good: short-lived, scoped,
Account Tokens (default) audience-bound; pod- automatically revoked when pod
(TokenRequest bound; automatically is deleted or SA is deleted.
API) rotated by kubelet

12.2 AWS IAM Roles for Service Accounts (IRSA)

12.2.1 Complete IRSA Flow
IRSA is AWS's mechanism for granting Kubernetes pods access to AWS services (S3, DynamoDB,
SQS, Secrets Manager, etc.) without embedding long-term AWS access keys. It uses the OIDC
federation standard to exchange Kubernetes service account tokens for AWS temporary
credentials.

12.3 SPIFFE and SPIRE — Universal Workload Identity

Page 52 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

12.3.1 SPIRE Architecture

SPIRE (SPIFFE Runtime Environment) provides workload identity across any cloud, on-premise
data centre, or edge environment. It is the most comprehensive and portable solution for workload
identity.

12.3.2 SPIRE Registration Entries

12.4 Service-to-Service Authentication Patterns

12.4.1 Comparison of Authentication Approaches
Approach Mechanism Security Level Operational Best For
Complexity
Shared secret Static token in Low: long-lived, Low: simple to Internal tools,
/ API key header hard to rotate, implement prototypes, non-
no identity sensitive APIs
mTLS X.509 cert with Very High: short- High: requires Production
(SPIFFE/Istio) SPIFFE ID in TLS lived, service mesh or microservices; zero
handshake cryptographic SPIRE trust environments
identity, auto-
rotated
JWT (OAuth2 Short-lived JWT High: signed, Medium: Cross-org API calls;
client from auth server expiring, requires auth partner integrations
credentials) audience-bound server; token
refresh logic
IRSA / Cloud IAM High: short-lived, Low-Medium: Pods accessing
Workload temporary cloud-native, cloud managed; cloud services (S3,
Identity credentials via audit-logged annotation- RDS, etc.)
OIDC based
Network- IP allowlisting, Low: IPs can be Low: firewall Legacy systems; last
based trust VPC security spoofed; no rules resort
groups service-level
identity

12.4.2 Token Exchange for Cross-Cluster Authentication

When service A in cluster-1 needs to call service B in cluster-2, and both clusters have different
trust roots, OAuth 2.0 Token Exchange (RFC 8693) provides a standard mechanism:

Page 53 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 13: Deployment Strategies and

Release Engineering
Deployment strategy is the bridge between development and production. The goal is to ship
changes frequently, safely, and with the ability to reverse course quickly when things go wrong. A
principal engineer must understand the full spectrum of deployment patterns, the operational tooling
that implements them, and the interplay between deployment strategy and database schema
evolution.

13.1 Deployment Strategies Comparison

Strategy Traffic Split Infrastructure Rollback Speed Risk Level Complexity
Cost
Recreate Old down; 1× Redeploy old High: full Very Low
new up version downtime
(minutes)
Rolling Gradual pod ~1.3× peak kubectl rollout Medium: Low
Update replacement undo (seconds) mixed
versions
briefly
Blue/Green Instant 2× (both envs) Switch back Low: clean Medium
0%→100% (seconds via LB) cutover
switch
Canary % split (5% → 1.1–1.5× Reduce canary Very Low: Medium-
25% → 100%) to 0% limited blast High
radius
Shadow/Dark 100% to 2× (shadow Remove shadow Near-zero: High
Launch stable; copy to receives all routing shadow not
shadow traffic) serving users
Feature Flag Per- 1× (same Toggle flag off Very Low: Low (for
user/group binary) (milliseconds) per-feature app);
targeting control Medium (for
flag system)
A/B Test Split by user 1.1–1.5× Route all to A Low High
attribute (requires
analytics)

13.2 Argo Rollouts — Progressive Delivery

13.2.1 Canary Rollout with Automated Analysis

Page 54 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

13.3 Database Migrations — The Hardest Part

13.3.1 The Expand/Contract Pattern
Database migrations in zero-downtime deployments require that any schema change be compatible
with both the current and new version of the application simultaneously. The expand/contract (or
parallel change) pattern achieves this through a three-phase process:

13.3.2 Blue/Green Database Migrations

For larger schema changes that cannot be done with expand/contract, blue/green at the database
layer is required:
86. Create a replica (blue = old schema; green = new schema destination).
87. Use Debezium CDC (Change Data Capture) or logical replication to stream all changes from
blue to green, applying schema transformations.
88. Once green is caught up (lag < 1 second), switch application connections from blue to
green. Application must support reconnect.
89. Keep blue running for 24-48 hours as a rollback option.
This approach requires database-level replication capability and careful orchestration. Tools: AWS
DMS, Flyway, Liquibase, pglogical.

13.4 GitOps with Argo CD — Production Configuration

Page 55 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 14: Observability — Metrics, Logs,

Traces, and SLOs
Observability is the ability to understand the internal state of a distributed system from its external
outputs. In a microservices architecture with dozens of services, thousands of pods, and millions of
requests per day, observability is not a nice-to-have—it is a prerequisite for operating the system at
all. A principal engineer designs observability in from day one, not as an afterthought.

14.1 The Four Signals (Google SRE)

Google's Site Reliability Engineering book identifies four golden signals that, if properly monitored,
are sufficient to catch most production problems:

Signal Description Alert Condition Example Metric

Latency Time to serve a p99 > 500ms for histogram_quantile(0.99,
request. > 2 minutes rate(http_request_duration_seconds_bucket[5m]))
Distinguish
between
successful and
failed request
latency.
Traffic Request rate. Traffic drops > rate(http_requests_total[5m])
Shows demand 30% suddenly
on the system (indicates
and helps upstream
correlate other problem or
signals. incident)
Errors Rate of failed Error rate > rate(http_requests_total{status=~'5..'}[5m]) /
requests (5xx, 0.1% (SLO rate(http_requests_total[5m])
timeouts, failed breach) or > 5%
health checks). (page
immediately)
Saturation How full the CPU > 80% process_cpu_seconds_total;
service is. CPU, sustained; jvm_memory_used_bytes; db_pool_active /
memory, queue depth > db_pool_size
connection 1000; conn pool
pool, queue > 90% full
depth.

14.2 Prometheus — Production Setup

14.2.1 Prometheus Architecture

Page 56 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Prometheus is a pull-based monitoring system. It scrapes HTTP /metrics endpoints at configurable

intervals, stores time-series data in its own TSDB (time-series database), and evaluates alerting
rules against the stored data.

14.2.2 Alerting Rules — Production Patterns

14.3 Distributed Tracing with OpenTelemetry

14.3.1 OpenTelemetry Collector Pipeline

14.4 SLOs, Error Budgets, and Alerting

14.4.1 Defining SLIs and SLOs

Page 57 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 15: Security — Zero Trust and Supply

Chain
Security in a cloud-native environment requires a fundamentally different mindset from traditional
perimeter-based security. The Zero Trust model assumes that the network is always hostile, that
identity must be verified cryptographically for every request, that every component operates with the
minimum necessary privileges, and that compromise of any individual component does not
compromise the entire system.

15.1 Zero Trust Architecture Principles

Principle Traditional Zero Trust Approach Implementation
Approach
Network Trust all traffic No implicit trust; verify mTLS between all services;
perimeter inside the firewall every request never rely on network location
as identity
Lateral Once inside, move Micro-segmentation; NetworkPolicy default-deny;
movement freely every hop requires auth Istio AuthorizationPolicy;
separate namespaces per
sensitivity level
Credential Long-lived Short-lived; IRSA, SPIFFE/SPIRE, Vault
lifecycle passwords; automatically rotated; dynamic secrets; max 24h
manual rotation bounded scope credential TTL
Access control Role-based on Attribute-based; OPA/Cedar policies; time-
network location or context-aware; just-in- bounded access; request-level
group membership time decisions
Observability Firewall logs only Every request logged; Comprehensive audit logs;
anomaly detection behaviour analytics; SIEM
integration

15.2 Pod Security — Hardening Reference

15.3 Supply Chain Security

15.3.1 Sigstore and Cosign
Sigstore is a set of tools and services for cryptographically signing software artifacts. Cosign signs
container images with OIDC-based keyless signing—no private key material to manage.

Page 58 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

15.3.2 SBOM (Software Bill of Materials)

15.4 OPA Gatekeeper — Policy as Code

Page 59 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 16: CI/CD Pipeline and GitOps

Continuous Integration and Continuous Delivery form the engineering backbone of rapid, safe
software delivery. GitOps extends this by treating the entire desired state of the system—not just
application code, but infrastructure configuration, Kubernetes manifests, and policy—as version-
controlled, auditable, declarative Git content.

16.1 CI Pipeline — Complete Stage Reference

A mature CI pipeline for a production microservice includes these stages, executed in order with
fail-fast semantics:

16.2 Multi-Stage Dockerfile Best Practices

Page 60 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 17: Data Layer — Databases,

Caching, and Storage
The data layer is the final destination of most requests and the hardest layer to scale. Unlike
stateless services, which can be horizontally scaled with minimal coordination, stateful systems—
databases, caches, queues, object stores—require careful design decisions that are very difficult
and expensive to change later. A principal engineer must make these decisions with a clear
understanding of the CAP theorem, consistency models, access patterns, and operational trade-
offs.

17.1 Database Patterns for Microservices

17.1.1 Database-per-Service with Change Data Capture
The database-per-service pattern is a core microservices principle. Each service owns its schema
and data. No other service may directly query another service's database. Data sharing happens
via APIs or events.
Change Data Capture (CDC) enables event-driven data sharing without tight coupling:

17.2 PostgreSQL — Production Configuration

17.2.1 Connection Pooling with PgBouncer

17.2.2 Read Replicas and Query Routing

PostgreSQL supports streaming replication: the primary WAL is streamed to replicas in near-real-
time (typically < 100ms lag). Read-heavy workloads should route read queries to replicas:

17.3 Redis — Production Architecture

17.3.1 Redis Cluster Data Sharding

17.4 Caching Patterns — Advanced

17.4.1 Cache-Aside with Distributed Lock (Cache Stampede Prevention)

Page 61 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

17.5 Data Consistency Patterns

17.5.1 Outbox Pattern (Transactional Messaging)
The outbox pattern solves the dual-write problem: how to atomically update a database AND
publish an event to Kafka, when these are two different systems and distributed transactions are
not available.

Page 62 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Chapter 18: Principal Engineer Decision

Framework
The most valuable skill of a principal engineer is not the mastery of any individual technology—it is
the ability to make sound architectural decisions under uncertainty, communicate the trade-offs
clearly to stakeholders, and build systems that are correct today and evolvable tomorrow. This
chapter synthesises the technical knowledge from all previous chapters into decision frameworks,
anti-patterns, and the principles that define principal-level engineering judgment.

18.1 The Architecture Decision Record (ADR)

Every significant architectural decision should be documented as an ADR. An ADR captures the
context, decision, alternatives considered, and consequences. This creates an institutional memory
that prevents organisations from cycling through the same decisions repeatedly.

18.2 The Eight Fallacies of Distributed Computing

These fallacies, identified by Peter Deutsch and James Gosling at Sun Microsystems, are the most
common false assumptions engineers make when designing distributed systems:

Fallacy Reality Engineering Implication

The network is Networks partition; packets All remote calls must have timeouts. All
reliable are dropped; NICs fail; BGP operations that matter must have retries with
routes flap; cables are cut idempotency. Design for partial failure, not just
complete failure.
Latency is zero Cross-region: 100-200ms; Never make a synchronous remote call in a hot
cross-AZ: 1-5ms; cross-pod: loop. Batch requests. Cache aggressively.
0.1-1ms + serialisation Design data access patterns to minimise hops.
Bandwidth is WAN: 100Mbps-1Gbps Be aware of large payload sizes. Protobuf beats
infinite shared; intra-cluster: 10- JSON 3-10x. Pagination prevents unbounded
100Gbps but shared responses. Compression helps on WAN.
The network is Assumption: internal mTLS everywhere; zero trust; encrypt all data in
secure network is trusted. Reality: transit and at rest; never trust the caller's
lateral movement is the claimed identity without cryptographic proof.
primary enterprise attack
pattern
Topology doesn't Pods are created/destroyed Design for dynamic topology. Use service
change constantly; nodes are discovery (DNS/Envoy EDS). Do not hardcode
replaced; AZs fail; clusters IPs. Implement graceful degradation when
are migrated endpoints disappear.

Page 63 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Fallacy Reality Engineering Implication

There is one 50+ teams own 200+ Explicit contracts between services (OpenAPI,
administrator services; cloud providers protobuf). Versioned APIs. SLOs and error
manage underlying infra; budgets. Runbooks for cross-team incidents.
3rd parties own CDN/LB
Transport cost is Serialisation, TLS, TCP Profile the full request chain. Measure sidecar
zero overhead, DNS lookups, overhead. Batch small messages. Choose
sidecar processing all have efficient serialisation formats.
CPU and memory cost
The network is Mix of bare metal, VMs, Test across all network paths. Understand MTU
homogeneous containers; IPv4 and IPv6; (VXLAN overhead reduces effective MTU). Use
different MTUs, QoS explicit protocol version negotiation.
policies, firewall rules

18.3 Anti-Patterns — The Principal Engineer's List

18.3.1 The Distributed Monolith
A distributed monolith has the worst of both worlds: the operational complexity of microservices
(separate deployments, network calls, distributed tracing needed) but the tight coupling of a
monolith (services cannot be deployed independently; database schema changes require
coordinating 10 teams; a single service going down takes the entire system with it).
Symptoms: circular dependencies between services; shared database tables; synchronous call
chains 5+ hops deep; a 'platform team' that must be involved in every deployment.
Fix: properly define service boundaries using Domain-Driven Design. Services should be aligned
with bounded contexts. If service A cannot be deployed without also deploying service B, they are
not actually separate services.

18.3.2 The Chatty Service

A single user action triggers N downstream synchronous RPC calls in sequence. Each adds latency
and failure probability. With 5 sequential calls of 50ms each, p99 latency is at minimum 5 × 50ms =
250ms before any business logic.
Fix: parallel fan-out (concurrent calls); API composition at the gateway (BFF pattern); GraphQL to
let clients request exactly what they need; async patterns where real-time response is not required.

18.3.3 Missing Idempotency

An operation that creates money, charges a payment, or sends an email is called with retry logic
but no idempotency key. The network times out; the client retries; the operation executes twice. The
customer is charged twice or receives two confirmation emails.

Page 64 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Fix: every mutating operation must accept an idempotency key (client-generated UUID). The server
stores the key and result; duplicate requests return the cached result without re-executing.
Implement this at the API gateway layer for consistent enforcement.

18.4 Technology Selection Framework

Decision Questions to Ask Default Recommendation Override Conditions
Synchronous Does the caller need an Synchronous gRPC for real- Choose async when
vs Async immediate result? Can time; Kafka for decoupled latency spike tolerance
the caller proceed if the processing > 5 seconds or callee
callee is down? SLA < caller SLA
SQL vs Do you need PostgreSQL unless you Cassandra for write-
NoSQL transactions? Joins? have a specific, well- heavy, wide-column;
Flexible schema? What understood reason for DynamoDB for
are the read patterns? NoSQL serverless + key-value;
MongoDB for
document model with
flexible schema
Service mesh How many languages? Service mesh (Istio) for Skip mesh if < 3
vs library How important is uniform production; libraries for services; if sub-1ms
policy enforcement? proof of concept latency is required and
Envoy overhead is
measurable
Kubernetes vs What is the operational Managed K8s Self-managed only if
managed expertise of the team? (EKS/GKE/AKS) unless cloud lock-in is a hard
service strong rationale for self- requirement or if on-
managed premise with no cloud
Monolith vs How large is the team? Modular monolith first; Start with
microservices Are bounded contexts extract services when team microservices only if
clear? What are the and scaling demand it team is > 50 engineers
scaling requirements? and bounded contexts
are crystal clear

18.5 The Principal Engineer's Pre-Production Checklist

18.5.1 Reliability
• SLIs and SLOs defined, documented, and implemented in Prometheus alert rules
• Multi-window burn rate alerts configured (1h and 5m windows)
• startupProbe, livenessProbe, and readinessProbe implemented and tested
• Graceful shutdown with SIGTERM handler and preStop hook
• PodDisruptionBudget configured (minAvailable: 2 or maxUnavailable: 1)
• HorizontalPodAutoscaler with appropriate min/max replicas and metrics

Page 65 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

• Circuit breakers configured for all downstream dependencies (via Envoy DestinationRule)
• Retry policy with exponential backoff and idempotency keys for all mutating calls
• Database connection pool sized correctly (PgBouncer pool_size = pod_count × 2)
• Tested at 2× expected peak load with k6 or Gatling
• Chaos engineering test: what happens when the primary database is unavailable?

18.5.2 Security
• Pod security context: runAsNonRoot, readOnlyRootFilesystem, drop ALL capabilities
• seccompProfile: RuntimeDefault or custom profile
• Secrets mounted as read-only files, not environment variables
• No secrets in Dockerfile, environment variables at build time, or Git history
• Image signed with Cosign; signature verified in admission controller
• Image scanned for CVEs; no HIGH/CRITICAL unmitigated vulnerabilities
• NetworkPolicy: default-deny with explicit allow rules
• Istio PeerAuthentication: STRICT mTLS in the namespace
• Istio AuthorizationPolicy: only authorised callers can reach each endpoint
• RBAC: service account has minimum required permissions; no cluster-admin
• Vault dynamic credentials for database access; static credentials prohibited
• WAF rules reviewed and tested with OWASP ZAP or similar

18.5.3 Observability
• Structured JSON logging with trace_id, span_id, request_id on every log line
• Prometheus /metrics endpoint exposing RED metrics per endpoint
• OpenTelemetry SDK instrumented; traces exported to Jaeger/Tempo
• Dashboard created in Grafana showing SLI compliance and RED metrics
• Runbook written for every alert rule
• On-call rotation configured with escalation policy
• Synthetic monitoring (Blackbox Exporter or Uptime Robot) from external perspective

18.5.4 Operations
• Helm chart or Kustomize overlay per environment
• Argo CD Application configured with automated sync for non-prod, manual for prod
• Database migration strategy documented (expand/contract for schema changes)
• Rollback procedure documented and tested
• Runbook covers: how to scale up/down; how to restart; how to roll back; how to access logs;
how to connect to database for emergency queries

Page 66 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

• Capacity estimate: how many pods needed at 10× current traffic?

• Cost estimate: monthly compute, storage, and data transfer costs

18.6 Closing Principles

A principal engineer operates at the intersection of deep technical expertise, systems thinking, and
organisational influence. The technical knowledge in this document is necessary but not sufficient.
The following principles, harder to learn than any technology, define the practice:
• Optimise for the P99 case, not the average. Systems fail at the tail; design for it.
• Make failure modes explicit. A system that fails loudly and detectably is better than one that
silently degrades.
• Simplicity is not laziness. The simplest solution that meets requirements is almost always the
best solution. Every component you add is a component that can fail, be misconfigured, or be
misunderstood.
• Design for the team you have, not the team you wish you had. A system too complex for the
team to operate reliably will fail in production regardless of its theoretical elegance.
• Operability is a feature. If the on-call engineer cannot understand what is wrong at 3 AM from
dashboards and logs, the system is not ready for production.
• Write the runbook before the code. If you cannot explain how to diagnose and recover from
failures before writing the feature, you do not understand the failure modes well enough.
• Technical debt is a business risk. Quantify it (how much does it slow delivery? how many
incidents per quarter does it cause?) and communicate it in business terms.
• Architectural decisions age. Revisit major decisions every 6–12 months. What was the right
choice at 100 engineers and 10 million users may not be right at 500 engineers and 100
million users.

⚑ Final Thought
The best system architecture is the simplest one that meets your reliability, security, and
performance requirements today—and that your team can understand, operate, and evolve
tomorrow. Every pattern in this document is a tool. A principal engineer knows which tools to use,
which to avoid, and crucially, which to defer until the problem actually demands them.

Page 67 of 68 | © 2024 Principal Engineer Reference

Modern Distributed Systems Architecture — Principal Engineer Reference

Page 68 of 68 | © 2024 Principal Engineer Reference