MASTER GUIDE TO

PRODUCTION OBSERVABILITY

Building Systems That Process 50M+ Events Daily

Complete Technical Reference with Real-World Scenarios

Scope End-to-end observability system design & implementation

Scale 50M+ events/day, Petabyte-scale storage, Sub-100ms queries

Topics Traces, Spans, Logs, Metrics, ClickHouse, Kafka, OTel

Includes Visual flows, Real incidents, 100+ questions, Exercises

Companies Netflix, Uber, Pinterest, Spotify case studies

Created November 2025

Table of Contents

PART I: FOUNDATIONS
1. Observability Fundamentals & Visual Models

2. Traces, Spans, and Events - Complete Visual Guide

3. The Relationship Between Logs, Metrics, and Traces

4. OpenTelemetry Deep Dive

PART II: ARCHITECTURE & IMPLEMENTATION

5. Auto-Initialization Patterns (Sentry, Datadog Style)

6. Hybrid Logging: Logs as Span Events

7. Multi-Process Correlation & Context Propagation

8. Processing 50M+ Events: Architecture Patterns

PART III: STORAGE & QUERY OPTIMIZATION

9. ClickHouse for Observability: Schema Design

10. Query Optimization & Performance Tuning

11. Sampling Strategies at Scale

12. Cost Optimization Techniques

PART IV: REAL-WORLD SCENARIOS

13. Netflix: Distributed Tracing at Scale

14. Uber: Conquering Microservices Complexity

15. Production Incident Case Studies

16. Debugging Decision Trees

PART V: PRACTICE & MASTERY

17. 50+ Interview Questions with Answers

18. Troubleshooting Exercises & Solutions

19. Performance Tuning Challenges

20. System Design Questions

APPENDICES
A. OpenTelemetry Semantic Conventions

B. ClickHouse Query Reference

C. Common Anti-Patterns to Avoid

D. Further Reading & Resources

PART I: FOUNDATIONS
Chapter 1: Observability Fundamentals & Visual Models
Observability is the ability to understand the internal state of a system by examining its external outputs. Unlike
monitoring (which tells you WHEN something breaks), observability tells you WHY it broke and HOW to fix it.

The Three Pillars vs. The Unified Model

TRADITIONAL APPROACH (Three Pillars - Siloed)
■■■■■■■■■■■■ ■■■■■■■■■■■■ ■■■■■■■■■■■■
■ LOGS ■ ■ METRICS ■ ■ TRACES ■
■ ■ ■ ■ ■ ■
■ Splunk ■ ■Prometheus■ ■ Jaeger ■
■ ELK ■ ■ Datadog ■ ■ Zipkin ■
■■■■■■■■■■■■ ■■■■■■■■■■■■ ■■■■■■■■■■■■
↓ ↓ ↓
Manual correlation required via trace IDs
Different storage systems, different query languages

MODERN APPROACH (Unified Wide Events)

■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■ UNIFIED OBSERVABILITY DATA ■
■ ■
■ { ■
■ timestamp, trace_id, span_id, ■
■ service_name, operation_name, ■
■ logs: [...], metrics: {...}, ■
■ attributes: {user_id, region, ...} ■
■ } ■
■ ■
■ Single ClickHouse Cluster ■
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
↓
Automatic correlation, single query language,
High-cardinality support, 10:1 compression

KEY INSIGHT
Modern observability platforms store logs, metrics, and traces as a single unified event with high-cardinality
attributes. This "wide events" model enables instant correlation without expensive JOINs and provides the
flexibility to ask arbitrary questions about your system without pre-defining dashboards.
Chapter 2: Traces, Spans, and Events - Complete Visual
Guide
Understanding the Hierarchy
TRACE (Request Journey)
■■■ Represents complete user request across all services
trace_id: "abc123xyz789" (unique identifier for entire request)

SPAN (Individual Operation)

■■■ Represents single operation within the trace
■ span_id: "span001" (unique within trace)
■ parent_span_id: null (root span)
■ operation: "POST /api/checkout"
■ start: 2025-11-12T10:00:00.001Z
■ duration: 145ms
■ status: OK
■
■ SPAN EVENT (Point-in-time occurrence)
■ ■■■ timestamp: 2025-11-12T10:00:00.005Z
■ ■ name: "[Link]"
■ ■ attributes: {item_count: 3}
■ ■
■ ■■■ timestamp: 2025-11-12T10:00:00.015Z
■ ■ name: "[Link]"
■ ■ attributes: {total_price: 129.99}
■ ■
■ ■■■ timestamp: 2025-11-12T10:00:00.140Z
■ name: "[Link]"
■ attributes: {processor: "stripe"}
■
■■■ CHILD SPAN
span_id: "span002"
parent_span_id: "span001" (points to parent)
operation: "payment_service.charge"
start: 2025-11-12T10:00:00.020Z
duration: 95ms

■■■ GRANDCHILD SPAN

span_id: "span003"
parent_span_id: "span002"
operation: "[Link].save_transaction"
start: 2025-11-12T10:00:00.025Z
duration: 12ms

Real E-Commerce Example: Add to Cart Flow

USER CLICKS "ADD TO CART"

[TRACE ID: e78b4c12-3f6a-4d5e-8b2a-9c7f1e3d6a5b]

■
■■ SPAN 1: frontend.add_to_cart_click [Client Span] (150ms)
■ ■ service: web-frontend
■ ■ span_kind: CLIENT
■ ■ [Link]: POST
■ ■ [Link]: /api/cart/add
■ ■
■ ■■ EVENT: "[Link]" (0ms)
■ ■■ EVENT: "[Link]" (5ms)
■ ■■ EVENT: "[Link]" (10ms)
■ ■■ EVENT: "[Link]" (150ms)
■
■■ SPAN 2: api_gateway.handle_request [Server Span] (140ms)
■ ■ service: api-gateway
■ ■ span_kind: SERVER
■ ■ http.status_code: 200
■ ■ [Link]: "user_12345"
■ ■
■ ■■ EVENT: "[Link]" (0ms)
■ ■■ EVENT: "[Link]" (15ms)
■ ■
■ ■■ SPAN 2.1: cart_service.add_item [Client Span] (100ms)
■ ■ ■ service: api-gateway
■ ■ ■ span_kind: CLIENT
■ ■ ■ [Link]: CartService
■ ■ ■
■ ■ ■■ SPAN 2.1.1: cart_service.add_item [Server Span] (95ms)
■ ■ ■ ■ service: cart-service
■ ■ ■ ■ span_kind: SERVER
■ ■ ■ ■ [Link]: "product_789"
■ ■ ■ ■ [Link]: 1
■ ■ ■ ■
■ ■ ■ ■■ EVENT: "[Link]" (0ms)
■ ■ ■ ■
■ ■ ■ ■■ SPAN [Link]: inventory_service.check [Client] (40ms)
■ ■ ■ ■ ■ service: cart-service
■ ■ ■ ■ ■ span_kind: CLIENT
■ ■ ■ ■ ■
■ ■ ■ ■ ■■ SPAN [Link].1: [Link] [Server] (35ms)
■ ■ ■ ■ ■ service: inventory-service
■ ■ ■ ■ ■ span_kind: SERVER
■ ■ ■ ■ ■ [Link]: "product_789"
■ ■ ■ ■ ■ [Link]: true
■ ■ ■ ■ ■ [Link]: 42
■ ■ ■ ■ ■
■ ■ ■ ■ ■■ EVENT: "[Link]" (0ms)
■ ■ ■ ■ ■■ EVENT: "[Link]" (5ms)
■ ■ ■ ■ ■■ EVENT: "[Link]" (30ms)
■ ■ ■ ■
■ ■ ■ ■■ EVENT: "[Link]" (40ms)
■ ■ ■ ■■ EVENT: "[Link]" (45ms)
■ ■ ■ ■
■ ■ ■ ■■ SPAN [Link]: [Link] [Internal] (45ms)
■ ■ ■ ■ ■ service: cart-service
■ ■ ■ ■ ■ span_kind: INTERNAL
■ ■ ■ ■ ■ [Link]: postgresql
■ ■ ■ ■ ■ [Link]: "INSERT INTO cart_items..."
■ ■ ■ ■ ■ db.rows_affected: 1
■ ■ ■ ■ ■
■ ■ ■ ■ ■■ EVENT: "[Link]" (45ms)
■ ■ ■ ■
■ ■ ■ ■■ EVENT: "[Link]" (95ms)
■ ■ ■
■ ■ ■■ EVENT: "[Link]" (100ms)
■ ■
■ ■■ EVENT: "[Link]" (140ms)
■
■■ END OF TRACE (Total: 150ms)

TIMELINE VIEW (Waterfall Diagram):

0ms 50ms 100ms 150ms
|-------|-------|-------|
[■■■■■■■■■■■■■■■■SPAN 1■■■■■■■■■■■■■■■■] frontend (150ms)
[■■■■■■■■■■■■SPAN 2■■■■■■■■■■■■■■] api-gateway (140ms)
[■■■■■■SPAN 2.1■■■■■■■] cart-service call (100ms)
[■■■■SPAN 2.1.1■■■■■] cart-service server (95ms)
[SPAN [Link]] inventory check (40ms)
[SPAN...] inventory server (35ms)
[■■SPAN [Link]■■] db insert (45ms)

CRITICAL PATH (longest sequential path):

SPAN 1 → SPAN 2 → SPAN 2.1 → SPAN 2.1.1 → SPAN [Link] = 150ms total

The bottleneck is SPAN [Link] (db insert, 45ms) occurring sequentially

after inventory check. These could potentially be parallelized.
Span Kinds Explained
Span Kind When to Use Example Parent Usually

CLIENT Making outbound request HTTP call, gRPC call, DB query

INTERNAL or SERVER

SERVER Receiving inbound request HTTP endpoint, RPC handler CLIENT (remote)

INTERNAL Operation within service Function call, computation Any

PRODUCER Sending async message Kafka produce, SQS send INTERNAL

CONSUMER Receiving async message Kafka consume, SQS receive PRODUCER (remote)

When to Use Span Events vs New Spans

DECISION: Should this be a Span Event or a new Span?

START
■
■■ Does it have meaningful DURATION?
■ ■
■ ■■ YES (> 1ms, worth measuring)
■ ■ ■■→ Use SEPARATE SPAN
■ ■
■ ■■ NO (instantaneous or sub-millisecond)
■ ■■→ Continue to next question
■
■■ Does it represent a DISTINCT OPERATION?
■ ■
■ ■■ YES (separate logical unit of work)
■ ■ ■■→ Use SEPARATE SPAN
■ ■
■ ■■ NO (just a milestone/checkpoint)
■ ■■→ Continue to next question
■
■■ Do you need to SAMPLE independently?
■ ■
■ ■■ YES (different sampling rate needed)
■ ■ ■■→ Use SEPARATE SPAN
■ ■
■ ■■ NO
■ ■■→ Continue to next question
■
■■ Is it DIAGNOSTIC information?
■ ■
■ ■■ YES (logging, state change, milestone)
■ ■ ■■→ Use SPAN EVENT
■ ■
■ ■■ NO
■ ■■→ Use SEPARATE SPAN (default)

EXAMPLES:

✓ Use SPAN EVENT for:

• "validation started"
• "cache hit"
• "retry attempt #3"
• "user authenticated"
• "state transitioned to PROCESSING"
• Error logs with context

✓ Use SEPARATE SPAN for:

• Database query (has duration)
• HTTP request to another service
• File I/O operation
• Authentication check (separate unit)
• Payment processing (distinct operation)
Chapter 3: The Relationship Between Logs, Metrics, and
Traces
How They Complement Each Other
METRICS LOGS TRACES
(Aggregated) (Individual Events) (Request Journey)

"WHAT is happening" "WHY it happened" "WHERE it happened"

■■■■■■■■■■■■■■■ ■■■■■■■■■■■■■■■■ ■■■■■■■■■■■■■■■■

■ Error Rate ■ ■ Exception ■ ■ Failed Span ■
■ spiked to ■ ■■■■→ ■ Stack Trace ■ ■■■■→ ■ in Payment ■
■ 15% at 10am ■ ■ NullPointer ■ ■ Service ■
■■■■■■■■■■■■■■■ ■■■■■■■■■■■■■■■■ ■■■■■■■■■■■■■■■■
↓ ↓ ↓
Alerts you Tells you what error Shows you which
something broke message appeared service/operation

CORRELATION FLOW:

1. METRIC ALERT: "API error rate > 5%"

■■→ Metric: api_errors_total{status="500"} increased

2. FIND EXEMPLAR TRACES

■■→ Query: Get sample trace_ids where status=500
■■→ Result: trace_id="abc123", "def456", "ghi789"

3. EXAMINE TRACE
■■→ Load trace "abc123"
■■→ Waterfall shows: payment_service span failed (95ms)

4. GET SPAN LOGS

■■→ Filter logs by trace_id="abc123" AND span_id="span_payment"
■■→ Log: "ConnectionError: Unable to reach payment gateway"

5. ROOT CAUSE IDENTIFIED

■■→ Payment gateway is down, causing 500 errors
■■→ Fix: Switch to backup payment processor

UNIFIED DATA MODEL (ClickHouse):

CREATE TABLE unified_observability (

-- Time (essential for all signals)
timestamp DateTime64(9),

-- Correlation IDs
trace_id String,
span_id String,

-- Service Context
service_name LowCardinality(String),

-- TRACE data
span_name String,
span_kind LowCardinality(String),
duration_ns UInt64,

-- LOG data
log_severity LowCardinality(String),
log_body String,

-- METRIC data
metric_name LowCardinality(String),
metric_value Float64,

-- High-cardinality attributes
attributes Map(String, String),

-- Efficient storage
INDEX idx_trace_id trace_id TYPE bloom_filter GRANULARITY 1
) ENGINE = MergeTree()
ORDER BY (service_name, timestamp)
PARTITION BY toYYYYMMDD(timestamp)
TTL timestamp + INTERVAL 90 DAY;

-- Now you can query EVERYTHING together:

SELECT
span_name,
count() as error_count,
avg(duration_ns) / 1000000 as avg_duration_ms,
groupArray(log_body) as error_messages
FROM unified_observability
WHERE timestamp >= now() - INTERVAL 1 HOUR
AND attributes['http.status_code'] = '500'
GROUP BY span_name
ORDER BY error_count DESC;
PART IV: REAL-WORLD SCENARIOS
Chapter 13: Netflix - Distributed Tracing at Scale
Netflix processes millions of streaming sessions daily across a complex microservices architecture. Their
observability tool "Edgar" demonstrates production-scale distributed tracing patterns that reduced
troubleshooting time from 30 minutes to 2 minutes.

INCIDENT: Streaming Session Failure

PROBLEM:
Users in EU region unable to start streaming sessions
- Error: "Unable to load video content"
- Affecting ~5% of EU users
- Started: 14:23 UTC
- Duration: 23 minutes before detection

TRADITIONAL DEBUGGING (Before Edgar): 30+ minutes

■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
1. Check CDN logs (3 mins) ✗ No obvious issues
2. Check API gateway metrics (5 mins) ✗ All services "healthy"
3. Check each microservice individually ✗ Time consuming
- Auth service logs (4 mins)
- Playback service logs (4 mins)
- License service logs (5 mins)
- DRM service logs (6 mins) ✓ Found timeout errors!
4. Correlate user IDs across systems (10 mins) ✗ Manual, error-prone
5. Find root cause ✓ DRM service overloaded

Total: ~37 minutes to identify root cause

WITH DISTRIBUTED TRACING (Edgar): 2 minutes

■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
1. Alert: "Playback failure rate spiked" (0s)

2. Query Edgar for recent failed traces: (10s)

SELECT trace_id, user_id, error_message
FROM traces
WHERE timestamp > now() - INTERVAL 10 MINUTE
AND status_code = 'ERROR'
AND service_name = 'playback'
LIMIT 10;

3. Open trace waterfall for trace_id: (15s)

"7f3c89a2-4b5d-6e8f-9a0c-1d2e3f4a5b6c"

Visual waterfall shows:

[playback-service]■■■■■■■■■■■■■■■■■■■■■■[OK: 150ms]
■■[auth-service]■■■■[OK: 25ms]
■■[license-service]■■■■■■■■■■■■■■■■■[TIMEOUT: 5000ms] ← PROBLEM!
■■[drm-service]■■■■■■■■■■■■■■■■[TIMEOUT: 5000ms] ← ROOT CAUSE

4. Click on drm-service span: (5s)

Span attributes show:
- [Link]: "TimeoutError"
- [Link]: "Connection pool exhausted"
- [Link]: "eu-west-1"
- [Link]: "[Link]"

5. Check DRM service health: (30s)

- CPU: 98% (normally 40%)
- Connection pool: 1000/1000 (maxed out)
- Traffic spike: 300% above normal

6. Root cause identified: (60s)

DRM service in EU flooded with requests due to
Premier League match streaming spike

Solution: Scale up DRM service instances

Total: ~2 minutes to identify and start fixing

KEY LEARNINGS FROM NETFLIX:

■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
1. CAPTURE 100% of critical paths
- All playback-related traces at 100% sample rate
- Non-critical traces at 1% sample rate

2. ADD BUSINESS CONTEXT to traces

- User tier (Free/Premium)
- Content type (Live/VOD)
- Device type (TV/Mobile/Web)
- Geographic region

3. BUILD ANALYZERS on top of traces

- Automatic pattern detection
- Common failure mode identification
- "Why no 4K?" analyzer
- "Why no HDR?" analyzer

4. INTEGRATE with other signals

- Link traces to application logs
- Connect traces to metrics dashboards
- Correlate with A/B test variants
PART V: PRACTICE & MASTERY
Chapter 17: 50+ Interview Questions with Detailed
Answers
CATEGORY 1: Fundamentals (10 Questions)
Q1: Explain the difference between monitoring and observability.
Answer: Monitoring answers "What is broken?" by checking known failure modes through predefined metrics and
alerts. Observability answers "Why is it broken?" by allowing you to ask arbitrary questions about system behavior
using high-cardinality data. Example: Monitoring tells you "API error rate is 5%". Observability lets you ask "Which
users from which regions using which app versions on which devices are seeing these errors?" Monitoring = Known
unknowns. Observability = Unknown unknowns.

Q2: What is a span and how does it differ from a trace?

Answer: A span represents a single operation or unit of work within your application (e.g., database query, HTTP
request, function call). A trace is a collection of spans that represents the complete journey of a request through your
distributed system. Analogy: If a trace is a book, spans are the chapters. Each chapter (span) has its own start/end
time, but together they tell the complete story (trace) of what happened.

Q3: When would you use a span event instead of creating a new span?
Answer: Use span events for: • Point-in-time occurrences without meaningful duration • Diagnostic logging within an
operation • State transitions or milestones • Events with < 1ms duration Use separate spans for: • Operations with
measurable duration (> 1ms) • Distinct units of work (DB queries, API calls) • Operations you want to sample
independently • Operations that might run in parallel Example: Within a "process_payment" span, use events for
"validation_started" and "card_authorized", but create separate spans for "call_payment_gateway" and
"update_database".
CATEGORY 2: Distributed Systems (10 Questions)
Q4: How do you propagate trace context across service boundaries?
Answer: Use W3C Trace Context standard via HTTP headers: traceparent: 00-{trace_id}-{parent_span_id}-{flags}
tracestate: vendor-specific data (optional) Implementation: 1. Service A creates a span and injects context into HTTP
headers 2. Service B extracts context from headers and creates child span 3. Context propagates through entire
request chain For message queues (Kafka, SQS), inject context into message headers.

Q5: What challenges arise when implementing distributed tracing at scale?

Answer: Key challenges: 1. Data Volume: At 50M events/day, you generate massive amounts of trace data Solution:
Intelligent sampling (tail-based, adaptive) 2. Consistent Propagation: Ensuring all services propagate context
correctly Solution: Auto-instrumentation, testing, enforcement 3. Clock Skew: Timestamps across distributed hosts
may be inconsistent Solution: Use relative time within traces, NTP synchronization 4. High Cardinality: Millions of
unique trace/span IDs Solution: ClickHouse with bloom filters, proper indexing 5. Storage Costs: Full-fidelity traces are
expensive Solution: 90-day TTL, compression (10:1 ratio), sampling
CATEGORY 3: Performance & Scale (10 Questions)
Q6: How would you optimize ClickHouse queries for observability data?
Answer: Key optimizations: 1. Proper Ordering Key: ORDER BY (service_name, toStartOfHour(timestamp),
trace_id) - Most selective columns first - Time bucketing for better compression 2. Use LowCardinality for categorical
data: service_name LowCardinality(String) - Reduces storage by 10x - Faster queries on enums 3. Bloom Filters for
high-cardinality lookups: INDEX idx_trace_id trace_id TYPE bloom_filter GRANULARITY 1 - Fast trace_id lookups 4.
Materialized Views for aggregations: Pre-compute p95, p99 latency per service per hour 5. Sampling for exploratory
queries: SELECT ... FROM traces SAMPLE 0.1 - 10x faster for high-level analysis

Q7: At what point would you implement sampling, and what strategy?
Answer: Implement sampling when: • Traffic > 1000 req/sec per service • Storage costs > $5000/month • Query
latency > 500ms for common queries Strategies (in order of sophistication): 1. Head-based (Probabilistic): Sample
X% of all requests - Simple, low overhead - Might miss important errors 2. Adaptive: Adjust sampling based on error
rate - 100% of errors - 50% of slow requests (p95+) - 1% of normal requests 3. Tail-based (Recommended for
production): - Buffer all spans for ~60 seconds - Decide to keep/drop based on complete trace - Keep: all errors, slow
requests, interesting patterns - Drop: fast successful requests At 50M events/day, use tail-based sampling to reduce to
~10M (5:1 ratio)
CATEGORY 4: Troubleshooting Scenarios (10 Questions)
Q8: You notice query latency spiked from 50ms to 500ms. Walk through your debugging process.
Answer: Systematic debugging approach: 1. METRICS: Identify the pattern - Which endpoint? (query metrics by
[Link]) - Which timeframe? (when did spike start?) - Which users? (filter by user_id, region) 2. TRACES: Find
exemplar traces ```sql SELECT trace_id, duration_ms FROM traces WHERE timestamp > spike_start_time AND
[Link] = '/api/search' AND duration_ms > 400 LIMIT 10; ``` 3. WATERFALL: Identify bottleneck Load trace
waterfall, look for: - Long duration spans (DB query taking 400ms?) - Many sequential calls (N+1 query problem?) -
External API timeouts 4. LOGS: Get error context Filter logs by trace_id and span_id of slow span Look for: slow
query warnings, timeout errors 5. ROOT CAUSE: Common culprits - Database: Missing index, lock contention -
Cache: Cache invalidation, cache miss storm - Code: N+1 queries, inefficient algorithm - Infrastructure: CPU throttling,
memory pressure

Q9: Traces show all services healthy but users report errors. How do you investigate?
Answer: This is "silent failure" - services return 200 but with wrong data. Investigation steps: 1. Check application
logs (not just span status) ```sql SELECT log_body, trace_id FROM logs WHERE timestamp > now() - INTERVAL 1
HOUR AND log_severity = 'ERROR' AND service_name = 'checkout-service'; ``` 2. Look for span events indicating
issues Even if span status = OK, events might show: - "[Link]" - "[Link]" - "[Link]"
3. Check business metrics alongside technical metrics Technical: All 200 OK ✓ Business: Conversion rate
dropped 50% ✗ 4. Correlate with deployment times Did this start after a recent deployment? 5. Sample actual
responses Capture response payloads for investigation Look for: empty arrays, null values, default fallbacks
CATEGORY 5: System Design (10 Questions)
Q10: Design an observability system for a company processing 100M requests/day.
Answer: Complete architecture: COLLECTION LAYER: - OpenTelemetry SDKs in all services (auto-instrumentation)
- OTel Collector fleet (12 instances) • Batch size: 10,000 spans • Flush interval: 5s • Memory: 4GB per collector
STREAMING LAYER: - Kafka cluster (3 brokers, 24 partitions) • Topic: traces-raw • Retention: 7 days • Replication: 3x
• Throughput: 5000 msgs/sec PROCESSING LAYER: - Consumer fleet (24 instances, 1 per partition) • Tail-based
sampling (10:1 reduction) • Enrichment (add user metadata) • Transform to ClickHouse schema STORAGE LAYER: -
ClickHouse cluster (6 nodes, 3 shards, 2 replicas) • Distributed table across shards • Partition by day • TTL: 90 days •
Compression: ZSTD (10:1 ratio) • Storage: ~500GB/month post-compression QUERY LAYER: - Grafana for
dashboards - Custom UI for trace exploration - Alerts via Prometheus → AlertManager COSTS: - Infrastructure:
~$8,000/month - Storage: ~$500/month (S3 + ClickHouse) - Total: ~$8,500/month for 100M req/day
APPENDICES
Appendix C: Common Anti-Patterns to Avoid
1. Creating Too Many Spans ✗ BAD: Creating a span for every function call ✓ GOOD: Spans for meaningful
operations only (> 1ms duration) 2. High-Cardinality in Metrics Labels ✗ BAD:
http_requests{user_id="12345"} ✓ GOOD: Use traces for high-cardinality, metrics for aggregates 3. Not Using
Sampling ✗ BAD: Storing 100% of traces indefinitely ✓ GOOD: Tail-based sampling, 90-day TTL 4. Missing
Trace Context Propagation ✗ BAD: Traces break at service boundaries ✓ GOOD: Inject/extract W3C Trace
Context in all HTTP calls 5. Blocking Application on Telemetry ✗ BAD: Synchronous span export blocking
request handling ✓ GOOD: Async batching with bounded queues 6. No Backpressure Handling ✗ BAD: OOM
when backend is slow ✓ GOOD: Circuit breakers, drop spans if queue full 7. Unclear Span Names ✗ BAD:
span_name = "handler" ✓ GOOD: span_name = "POST /api/checkout" 8. Not Testing Instrumentation ✗
BAD: Hope traces work in production ✓ GOOD: Integration tests validating trace structure
Appendix D: Further Reading & Resources
Essential Reading: 1. "Distributed Tracing in Practice" by Austin Parker, Daniel Spoonhower 2. "Observability
Engineering" by Charity Majors, Liz Fong-Jones 3. OpenTelemetry Documentation: [Link]/docs 4.
ClickHouse for Observability: [Link]/docs/en/observability Company Blogs: • Netflix Tech Blog:
[Link] • Uber Engineering: [Link] • Pinterest Engineering:
[Link]/pinterest-engineering • Cloudflare Blog: [Link] Tools & Platforms: •
OpenTelemetry: [Link]/open-telemetry • SigNoz (Open Source): [Link]/SigNoz/signoz • Jaeger:
[Link]/jaegertracing/jaeger • ClickHouse: [Link]/ClickHouse/ClickHouse Communities: • CNCF
Slack: #opentelemetry channel • ClickHouse Slack: [Link]/slack • OpenObservability Slack:
[Link]

■■■■■■■■■■■■■■■■■■■■■■■■

You've completed the Master Guide to Production Observability! You now have the knowledge to build,
deploy, and operate observability systems at massive scale. Remember: observability is not a destination but a
journey. Systems evolve, traffic grows, and requirements change. Keep learning, keep iterating, and most
importantly—keep your systems observable! Good luck with your [Link] founding engineer interview and
your observability journey! — Built with ♥ for engineers who want to understand WHY things break, not just
THAT they broke