MICROSERVICES

Design Patterns — Deep Dive

Real-World Examples | Java & Python Code | 60+ Interview Q&As

Patterns Covered:
Decomposition • Data Management • Communication • Reliability
Security • Observability • Deployment • UI Patterns
1. Microservices Design Patterns — Overview
Design patterns in microservices are proven, reusable solutions to recurring architectural problems.
Unlike OOP patterns (GoF), microservice patterns address the unique challenges of distributed systems:
network partitions, partial failures, data consistency across service boundaries, and operational
complexity.

Category Patterns Core Problem Solved

Decomposition Business Capability, Subdomain, Strangler How to split the system into
Fig, Anti-Corruption Layer services

Data DB per Service, Shared DB, Saga, CQRS, Data consistency & storage
Management Event Sourcing, Outbox isolation

Communication API Gateway, BFF, Service Mesh, How services talk to each other
Messaging, Request-Reply

Reliability Circuit Breaker, Retry, Bulkhead, Timeout, Handling failures gracefully

Rate Limiter, Fallback

Security Access Token, API Key, Service-to-Service Auth & authorization across
Auth, Secrets Mgmt services

Observability Log Aggregation, Distributed Tracing, Health Understanding what the system is
Check, Alerting doing

Deployment Blue/Green, Canary, Rolling Update, Sidecar, Releasing and running services
Service Discovery safely

UI Backend for Frontend, API Composition, Aggregating data for clients

Server-Side Page Fragment

Q What is the most important principle when applying design patterns in microservices?
A Patterns are tools, not rules. Apply them only when they solve a real pain point. The most common
mistake is over-engineering early — adding Saga, CQRS, and Event Sourcing to a system that could
be a simple CRUD app. Start with the simplest solution; add patterns when you hit the specific problem
they solve. Always articulate the trade-off: every pattern adds operational or cognitive complexity in
exchange for a specific benefit.
2. Decomposition Patterns
Decomposition patterns answer the first and most consequential question in microservices: how do you
break a system into services? Getting this wrong is expensive — poorly bounded services lead to chatty
inter-service communication, distributed monoliths, and teams that can't deploy independently.

2.1 Decompose by Business Capability

A business capability is something the business does to create value — stable, high-level functions that
rarely change even as implementation evolves. Each capability becomes a service boundary.

E-commerce Example
Business Capability Service Owns Data Key Operations

Product Management Catalog Service Products, Categories, Add product, Update price,
Attributes Search

Order Management Order Service Orders, Order Items, Place order, Cancel, Track
Status

Inventory Management Inventory Service Stock levels, Reserve, Release, Restock

Warehouses

Payment Processing Payment Service Transactions, Refunds Charge, Refund, Validate

Customer Management Customer Profiles, Addresses, Register, Update,

Service Preferences Authenticate

Shipping & Logistics Shipping Service Shipments, Carriers, Schedule, Track, Notify
Tracking

Notifications Notification Templates, Delivery logs Email, SMS, Push

Service

Reviews & Ratings Review Service Reviews, Ratings, Submit, Approve, Aggregate
Moderation

Q How do you identify business capabilities in a brownfield system?

A Use Event Storming — a workshop technique where domain experts and engineers collaboratively
map all domain events (things that happened), commands (what triggered them), aggregates (entities),
and bounded contexts on a large whiteboard. Events are orange sticky notes ('OrderPlaced',
'PaymentFailed'). Commands are blue ('PlaceOrder'). Aggregates are yellow ('Order'). After 2-3 hours,
natural clusters emerge — those clusters are your service candidates. Also look at organizational
structure (Conway's Law): teams that work independently on different parts of the codebase are strong
signals for service boundaries. Validate by checking: can this capability be changed and deployed
without touching another team's code?
2.2 Decompose by Subdomain (Domain-Driven Design)
DDD categorizes subdomains into three types, which should drive your investment and approach:

Subdomain Type Definition Build Strategy Example (E-commerce)

Core Domain Your unique competitive Build with highest Recommendation Engine,
advantage. Where you quality. Custom Pricing Algorithm, Search
win or lose. microservices. Ranking

Supporting Necessary but not Build custom but Order Management,

Subdomain differentiating. Supports simpler. Can be a Inventory, Shipping
the core. service.

Generic Solved problem. No Buy off-the-shelf or Email (SendGrid), Auth

Subdomain competitive advantage. use SaaS. (Auth0), Payments (Stripe)

Q What is the difference between a subdomain and a bounded context?

A A subdomain is a problem space concept — a part of the business domain. A bounded context is a
solution space concept — the boundary within which a particular domain model is defined and
applicable. In ideal DDD, they map 1:1: one subdomain = one bounded context = one service. In
practice, a large subdomain might be split into multiple bounded contexts (e.g., the 'Order' subdomain
might have separate contexts for Order Creation, Order Fulfillment, and Order Analytics). The key
insight is that the same word ('Customer') can mean different things in different bounded contexts — in
Sales it means a prospect with a CRM record; in Support it means a ticket submitter. Each context
defines its own Customer model independently.

2.3 Strangler Fig Pattern

Named after the strangler fig tree that grows around a host tree until it replaces it. Used for incremental
migration from a monolith to microservices without a 'big bang' rewrite.

Migration Steps
1. Identify a capability to extract — choose a well-defined, low-risk area first (e.g., email
notifications).
2. Build the new microservice in isolation with its own database and deploy it.
3. Add a routing layer (API Gateway or facade) in front of the monolith.
4. Route traffic for that capability to the new service, keeping the monolith as fallback.
5. Monitor, validate correctness, then remove the old code from the monolith.
6. Repeat for the next capability. The monolith 'shrinks' with each iteration.

# Routing layer (e.g., NGINX or API Gateway rules)

# Route /api/notifications/* to new microservice
location /api/notifications/ {
proxy_pass [Link]
}

# Everything else still goes to the monolith

location / {
proxy_pass [Link]
}

Pros Cons
• Zero risk of big-bang failure • Routing layer adds complexity
• Can be done incrementally • Dual writes during transition period
• Monolith keeps running during migration • Data migration between old/new DB
• Learn as you go • Long-lived migration (months/years)
• Deploy new service independently • Team must maintain both systems

Q What is the Anti-Corruption Layer (ACL) pattern and when is it used with Strangler Fig?
A The ACL is a translation layer that sits between a new service and the legacy monolith (or external
system), preventing the legacy model's concepts from 'corrupting' the new service's clean domain
model. When extracting a service from a monolith, the new service often needs data that still lives in
the monolith. Instead of having the new service understand the monolith's schema or APIs directly, you
create an ACL that translates the monolith's concepts into the new service's ubiquitous language. For
example, the monolith might represent a customer with a flat 'CustomerRecord' struct; the ACL
translates this into a proper 'Customer' aggregate with value objects that the new service uses. This is
especially important when the monolith uses a different programming paradigm or has messy,
inconsistent naming.
3. Data Management Patterns
Data management is where most microservice projects encounter their hardest problems. The shift from
shared databases to service-owned data enables autonomy but introduces distributed consistency
challenges.

3.1 Database per Service

Each service exclusively owns and manages its own database. No other service may directly access it
— all data access must go through the service's API.

Polyglot Persistence Example

Service Database Why This DB?

Product Catalog MongoDB (Document) Flexible schema for varied product attributes

Order Service PostgreSQL (Relational) ACID transactions, complex queries with joins

User Session Redis (Key-Value) Nanosecond lookups, TTL-based expiry

Activity Feed Cassandra (Wide- High write throughput, time-series data

Column)

Fraud Detection Neo4j (Graph) Relationship traversal for network analysis

Search Service Elasticsearch Full-text search, faceted navigation

Analytics ClickHouse (OLAP) Columnar storage for aggregate queries

Notification DynamoDB (Document) Managed, scalable, simple access patterns

Common Violation: Shared Database Anti-Pattern

Never let two services share a database directly. If Order Service and Inventory Service
both query the same 'orders' table, you've created a distributed monolith. Schema
changes require coordinating releases. Services are no longer independently deployable.
Fix: expose data via API, events, or replicate read-only views using CDC (Debezium).

Q How do you handle queries that need data from multiple services (JOIN problem)?
A Several strategies depending on latency and consistency needs: (1) API Composition — the caller
(or an API Gateway/BFF) makes parallel requests to each service and aggregates results in memory.
Good for low-data-volume queries. (2) CQRS with a dedicated Query Service — publish events from
each service to a Query Service that maintains a pre-joined, denormalized read model in its own
database. Query the read model directly. High performance but eventually consistent. (3) GraphQL
Gateway — acts as a composition layer, resolving fields by calling individual services. (4) Event-driven
denormalization — Order Service subscribes to ProductUpdated events and stores a local copy of
 product name/price in the order record (accept some duplication). The right choice depends on how
frequently the data changes and how stale you can tolerate it.

3.2 Saga Pattern

A Saga is a sequence of local transactions where each step publishes an event or message to trigger
the next step, and each step has a compensating transaction to undo it if a later step fails. Sagas
replace distributed ACID transactions (2PC) with eventual consistency.

Choreography-Based Saga
Services react to events autonomously — no central coordinator. Each service knows what to do when it
receives a specific event.

// Order Service
@KafkaListener(topics = 'order-commands')
public void createOrder(CreateOrderCommand cmd) {
Order order = [Link](new Order(cmd, [Link]));
[Link](new OrderCreatedEvent([Link](), [Link]()));
}

// Inventory Service — reacts to OrderCreated

@KafkaListener(topics = 'order-events')
public void onOrderCreated(OrderCreatedEvent event) {
try {
[Link]([Link](), [Link]());
[Link](new InventoryReservedEvent([Link]()));
} catch (InsufficientStockException e) {
[Link](new InventoryReservationFailedEvent([Link]()));
}
}

// Order Service — handles failure compensation

@KafkaListener(topics = 'inventory-events')
public void onInventoryFailed(InventoryReservationFailedEvent event) {
[Link]([Link](), [Link]);
[Link](new OrderCancelledEvent([Link](), 'OUT_OF_STOCK'));
}

Orchestration-Based Saga
A central Saga Orchestrator (a dedicated service or workflow engine) tells each service what to do,
tracks state, and handles failures.

// Saga Orchestrator — using Temporal workflow engine

@WorkflowInterface
public interface PlaceOrderSaga {
@WorkflowMethod
OrderResult placeOrder(PlaceOrderRequest request);
}

public class PlaceOrderSagaImpl implements PlaceOrderSaga {

 private final InventoryActivities inventory =
[Link]([Link]);
private final PaymentActivities payment =
[Link]([Link]);
private final OrderActivities orders =
[Link]([Link]);

@Override
public OrderResult placeOrder(PlaceOrderRequest req) {
String orderId = [Link](req);
String reservationId = null;
String paymentId = null;
try {
reservationId = [Link](orderId, [Link]());
paymentId = [Link](orderId, [Link](), [Link]());
[Link](orderId);
return [Link](orderId);
} catch (Exception e) {
// Compensate in reverse order
if (paymentId != null) [Link](paymentId);
if (reservationId != null) [Link](reservationId);
[Link](orderId, [Link]());
return [Link](orderId, [Link]());
}
}
}

Aspect Choreography Orchestration

Coordination Decentralized — each service decides Central orchestrator controls flow

Coupling Lower — services only know about Higher — orchestrator knows all steps
events

Visibility Hard to see the big picture Easy — orchestrator tracks full state

Testing Complex — must simulate full event Simpler — test orchestrator + each step
chain

Failure handling Each service handles its own failures Central failure + compensation logic

When to use Simple flows (2-3 steps), highly Complex multi-step flows, need visibility
decoupled teams

Q How does a Saga handle partial failures and ensure no data is lost?
A Each step in a Saga must be idempotent (safe to retry) and must have a defined compensating
transaction. Partial failure handling: (1) The step fails after DB write but before publishing an event —
use the Outbox Pattern: write event to an outbox table in the same transaction; a relay publishes it. (2)
The event is published but not consumed — message brokers guarantee at-least-once delivery;
consumers deduplicate by tracking processed event IDs. (3) A compensation fails — compensations
must also be idempotent and retried until success. In extreme cases (compensation also fails), use a
dead letter queue and trigger a manual resolution workflow with alerting. Tools like Temporal handle all
of this automatically: they persist workflow state durably and retry activities until they succeed, making
Sagas much more reliable.

Q What is the difference between a Saga and a 2PC (Two-Phase Commit)?

A 2PC uses a distributed transaction coordinator: Phase 1 (Prepare) — all participants vote 'ready';
Phase 2 (Commit) — if all vote ready, coordinator sends commit to all. If any vote 'no' or timeout,
coordinator sends rollback. Problems: the coordinator is a single point of failure; participants hold locks
during the entire protocol causing blocking; it doesn't work well with microservices that use different
databases, message brokers, or external APIs. Saga trades ACID atomicity for eventual consistency.
There are no distributed locks; each service commits locally and publishes events. If a step fails,
compensating transactions roll back previous steps. The system is temporarily inconsistent during
execution but reaches a consistent state (either fully succeeded or fully compensated). Sagas are the
standard for microservices because they work across any technology boundary and are resilient to
partial failures.
3.3 CQRS — Command Query Responsibility Segregation
CQRS separates the write model (commands that change state) from the read model (queries that
return data). They can use different databases, schemas, services, and scaling strategies.

# ── WRITE SIDE (Command Handler) ──────────────────────────────

class CreateOrderCommandHandler:
def __init__(self, order_repo, event_bus):
self.order_repo = order_repo
self.event_bus = event_bus

def handle(self, cmd: CreateOrderCommand) -> str:

# Validate business rules
order = [Link](cmd.customer_id, [Link], cmd.payment_method)
self.order_repo.save(order) # Write to PostgreSQL
self.event_bus.publish(OrderCreated( # Publish event
order_id=[Link],
customer_id=order.customer_id,
items=[Link],
total=[Link]
))
return [Link]

# ── READ SIDE (Event Handler builds denormalized view) ─────────

class OrderReadModelProjection:
def __init__(self, read_db): # Separate MongoDB read DB
[Link] = read_db

@event_handler(OrderCreated)
def on_order_created(self, event: OrderCreated):
# Denormalized view — includes customer name, product names
[Link].insert_one({
'_id': event.order_id,
'customer_id': event.customer_id,
'customer_name': self._fetch_customer_name(event.customer_id),
'items': self._enrich_items([Link]),
'total': [Link],
'status': 'PENDING',
'created_at': [Link]()
})

@event_handler(OrderStatusChanged)
def on_status_changed(self, event: OrderStatusChanged):
[Link].update_one(
{'_id': event.order_id},
{'$set': {'status': event.new_status, 'updated_at': [Link]()}}
)

# ── QUERY SIDE (Fast reads from denormalized DB) ────────────────

class OrderQueryService:
def get_order_summary(self, order_id: str) -> dict:
return self.read_db.orders.find_one({'_id': order_id}) # 1ms lookup

def get_customer_orders(self, customer_id: str) -> list:

return list(self.read_db.[Link](
{'customer_id': customer_id},
sort=[('created_at', -1)], limit=20
))
 Q When should you use CQRS and when is it overkill?
A Use CQRS when: (1) Read and write workloads have very different scaling needs — millions of
reads per second vs thousands of writes. (2) The optimal read data structure (denormalized, pre-
joined) is very different from the write structure (normalized, consistent). (3) You need multiple
specialized read models from the same data (dashboard view, mobile compact view, analytics
aggregate). (4) The system already uses event sourcing. Avoid CQRS when: the system is CRUD-
heavy with simple reads and writes that share the same structure, the team is small and the complexity
overhead isn't justified, or the read and write volumes are similar and neither is a bottleneck. CQRS
adds complexity — two codebases, eventual consistency, projection maintenance, and
synchronization between read and write stores. A typical REST API with a single database is perfectly
appropriate for most services.

3.4 Event Sourcing

Instead of storing only the current state, store the complete sequence of state-changing events. The
current state is derived by replaying all events. The event log is the source of truth.

// Event Store — append-only log of domain events

public class OrderAggregate {
private String id;
private OrderStatus status;
private List<OrderItem> items;
private BigDecimal total;
private List<DomainEvent> uncommittedEvents = new ArrayList<>();

// Reconstitute from event history

public static OrderAggregate rehydrate(List<DomainEvent> history) {
OrderAggregate order = new OrderAggregate();
[Link](order::apply);
return order;
}

// Command: Place Order

public void placeOrder(String customerId, List<OrderItem> items) {
if ([Link] != null) throw new IllegalStateException('Order already exists');
raiseEvent(new OrderPlaced([Link]().toString(), customerId, items));
}

// Command: Cancel Order

public void cancelOrder(String reason) {
if ([Link] == [Link]) throw new
InvalidOperationException('Cannot cancel shipped order');
raiseEvent(new OrderCancelled([Link], reason));
}

// Event handlers — update aggregate state (pure functions)

private void apply(OrderPlaced event) {
[Link] = [Link]();
[Link] = [Link]();
[Link] = [Link]();
[Link] = [Link];
}

private void apply(OrderCancelled event) {

 [Link] = [Link];
}

private void raiseEvent(DomainEvent event) {

apply(event);
[Link](event);
}
}

// Repository saves events, not aggregate state

public class EventSourcedOrderRepository {
public void save(OrderAggregate order) {
List<DomainEvent> events = [Link]();
[Link]([Link](), events, [Link]());
[Link](eventBus::publish); // Notify projections
}

public OrderAggregate findById(String orderId) {

List<DomainEvent> history = [Link](orderId);
return [Link](history);
}
}

Benefits Challenges
• Complete audit trail for free • Querying current state requires replay (use
• Time-travel: query state at any past point snapshots)
• Natural integration with event-driven arch • Schema evolution is complex (immutable
history)
• Rebuild any read model from event history
• Not suitable for simple CRUD
• Debug by replaying what actually
happened • High event volume grows the store fast
• Learning curve for the team

Q How do snapshots work in Event Sourcing and when do you need them?
A As the event log grows (thousands of events per aggregate), replaying all events to get current
state becomes slow. Snapshots are periodic checkpoints: serialize the aggregate's current state into a
snapshot at a specific event version (e.g., every 100 events). On load, fetch the latest snapshot and
only replay events that occurred after it. Implementation: after saving, if event count % 100 == 0, write
a snapshot. On load, find the latest snapshot by aggregate ID and version, deserialize it, then stream
and apply only newer events. Snapshots don't replace the event log — the log remains the source of
truth. You need snapshots when: a) aggregate has many events (>500), b) rehydration latency
exceeds your SLA, c) the aggregate is read frequently (every user request). Most aggregates don't
need snapshots — only high-volume, frequently-accessed aggregates like bank accounts or shopping
carts.
3.5 Outbox Pattern
Solves the dual-write problem: ensuring a database write and a message broker publish are atomic.
Without this pattern, a crash between the two operations leaves the system inconsistent.

-- Schema: Outbox table in the same database as business data

CREATE TABLE outbox_events (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
aggregate_type VARCHAR(255) NOT NULL,
aggregate_id VARCHAR(255) NOT NULL,
event_type VARCHAR(255) NOT NULL,
payload JSONB NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW(),
published_at TIMESTAMPTZ NULL
);

-- Application code: Single transaction writes both tables

BEGIN;
INSERT INTO orders (id, customer_id, status, total)
VALUES ($1, $2, 'PENDING', $3);

INSERT INTO outbox_events (aggregate_type, aggregate_id, event_type, payload)

VALUES ('Order', $1, 'OrderCreated',
jsonb_build_object('orderId', $1, 'customerId', $2, 'total', $3));
COMMIT;
-- If COMMIT fails, BOTH inserts roll back. No inconsistency.

-- Relay (Debezium CDC): streams outbox changes to Kafka automatically

# [Link]
{
'[Link]': '[Link]',
'[Link]': 'postgres',
'[Link]': 'public.outbox_events',
'transforms': 'outbox',
'[Link]': '[Link]',
'[Link]': 'event_type',
'[Link]': 'aggregate_type'
}

Q Why is the Outbox Pattern preferable to simply using a try-catch around DB write + event
publish?
A A try-catch approach has two failure scenarios that cannot be caught: (1) The DB write succeeds
but the process crashes before publishing — the event is lost forever. The catch block never runs
because the process is dead. (2) The DB write succeeds, publish succeeds, but the DB commit then
fails — you have a published event with no corresponding DB state. This is the 'dual-write problem'
and no amount of try-catch logic can solve it because it requires atomic operations across two different
systems (DB + broker) — which by definition are not atomic. The Outbox Pattern solves this by writing
to only one system (the database) atomically, then using a reliable relay (Debezium/CDC or polling
worker) to publish events. The relay guarantees at-least-once delivery; consumers handle
idempotency for any duplicates.
4. Communication Patterns

4.1 API Gateway Pattern

A single entry point for all client requests. Routes requests to appropriate services and handles cross-
cutting concerns centrally.

Responsibility Description Example

Routing Map URL patterns to backend /api/orders/* → Order Service :8081

services

Authentication Validate JWT/OAuth tokens Verify signature, check expiry, extract claims
before forwarding

Rate Limiting Throttle requests per client/IP/API 100 req/min per user, 1000/min per tenant
key

SSL Termination Handle HTTPS, forward HTTP Reduce TLS overhead on each service
internally

Load Balancing Distribute across service Round-robin, least connections, weighted

instances

Request Modify request/response format Add headers, translate JSON schemas

Transformation

Caching Return cached responses for GET Cache product catalog for 5 minutes
requests

API Composition Aggregate responses from Dashboard = User + Orders + Analytics

multiple services

Logging & Tracing Inject Trace-ID, log all requests Centralized access logs, OpenTelemetry

Q What are the trade-offs between a single API Gateway and multiple specialized gateways?
A A single gateway is simpler to operate and provides a consistent cross-cutting concern
implementation. However, it becomes a bottleneck if all teams must coordinate changes to it, and a
misconfiguration can take down all APIs simultaneously. Multiple gateways (per product line or per
client type — mobile, web, B2B) allow team autonomy and fault isolation but multiply operational
overhead. A practical middle ground: one gateway for external traffic (internet-facing), with separate
internal routing for service-to-service. For large organizations, consider a platform team that owns
gateway infrastructure but provides self-service configuration APIs so product teams can manage their
own routes without a central bottleneck.

4.2 Backend for Frontend (BFF) Pattern

Create a dedicated backend tailored for each client type. Each BFF is owned by the frontend team,
optimized for its client's specific needs.
# Mobile BFF — compact payload, offline-friendly
class MobileBFF:
async def get_home_screen(self, user_id: str) -> dict:
# Parallel fetch, compact format
user, cart, featured = await [Link](
self.user_svc.get_name_only(user_id),
self.cart_svc.get_count(user_id),
self.catalog_svc.get_featured(limit=5, img_size='small')
)
return { # Only fields mobile needs
'greeting': f'Hi {user.first_name}',
'cart_count': [Link],
'featured': [{'id':[Link], 'name':[Link], 'thumb':p.small_img, 'price':[Link]}
for p in featured]
}

# Web BFF — richer data, server-side rendering support

class WebBFF:
async def get_home_page(self, user_id: str) -> dict:
user, cart, featured, recommended, promo = await [Link](
self.user_svc.get_full_profile(user_id),
self.cart_svc.get_full_cart(user_id),
self.catalog_svc.get_featured(limit=20, img_size='large'),
self.rec_svc.get_recommendations(user_id, limit=10),
self.promo_svc.get_active_banners()
)
return { # Full data for server-side render
'user': user.to_dict(),
'cart': cart.to_dict(),
'featured_products': featured,
'recommended': recommended,
'promotions': promo
}

Q GraphQL vs BFF — which do you choose?

A GraphQL acts as a flexible API layer where clients declare exactly what data they need, eliminating
over/under-fetching without needing separate BFFs. BFF is an imperative server-side composition
approach where the backend team controls the contract. Choose GraphQL when: clients need high
flexibility in data fetching, you have many diverse clients with unpredictable query patterns, or a single
team owns the API layer. Choose BFF when: client needs are well-known and stable, the frontend
team wants full ownership of their API contract, security requirements are strict (GraphQL can expose
data through introspection), or you need fine-grained caching per client type. Many organizations use
both: GraphQL as the underlying composition layer, with BFF adding client-specific logic like auth,
caching, and response shaping on top.
4.3 Messaging Patterns

Publish-Subscribe (Pub/Sub)
Producer publishes events to a topic. All subscribed consumers receive every event independently.
Producers and consumers are fully decoupled — neither knows about the other.

// Kafka Producer — Order Service

public class OrderEventPublisher {
private final KafkaTemplate<String, Object> kafka;

public void publishOrderCreated(Order order) {

OrderCreatedEvent event = new OrderCreatedEvent(
[Link](), [Link](), [Link](), [Link]()
);
// Key = orderId ensures all events for same order go to same partition (ordering
guarantee)
[Link]('order-events', [Link](), event);
}
}

// Multiple independent consumers

@KafkaListener(topics='order-events', groupId='inventory-service')
public void onOrderCreated(OrderCreatedEvent event) {
[Link]([Link](), [Link]());
}

@KafkaListener(topics='order-events', groupId='notification-service')
public void onOrderCreatedSendEmail(OrderCreatedEvent event) {
[Link]([Link](), [Link]());
}

@KafkaListener(topics='order-events', groupId='analytics-service')
public void onOrderCreatedTrack(OrderCreatedEvent event) {
[Link]([Link](), [Link]());
}

Request-Reply over Messaging

Enables synchronous-style communication over asynchronous messaging. The requester sends a
message with a reply-to queue and correlation ID; the replier sends the response to that queue.

# Request-Reply pattern using RabbitMQ

class PricingServiceClient:
def get_price(self, product_id: str, quantity: int) -> Decimal:
correlation_id = str(uuid.uuid4())
reply_queue = [Link].queue_declare('', exclusive=True).[Link]

[Link].basic_publish(
exchange='',
routing_key='pricing-requests',
body=[Link]({'product_id': product_id, 'quantity': quantity}),
properties=[Link](
reply_to=reply_queue,
 correlation_id=correlation_id,
)
)
# Wait for response (with timeout)
response = self._wait_for_response(reply_queue, correlation_id, timeout=5.0)
return Decimal(response['price'])

Q How do you handle message ordering in Kafka and why does it matter?
A Kafka guarantees ordering within a partition, not across partitions. A topic with 10 partitions
processes messages in 10 parallel streams, each ordered internally. This matters when events must
be processed in sequence for the same entity — e.g., OrderCreated must be processed before
OrderShipped for the same orderId. Solution: use the entity ID as the Kafka message key. Kafka's
partitioner hashes the key and always routes messages with the same key to the same partition. All
events for orderId='ORD-123' land in partition 3 and are consumed in order. The trade-off: all events
for the same order are processed by one consumer, limiting parallelism per entity. However, different
orders (different keys) process fully in parallel across all partitions. For truly globally ordered events
(rare), use a single partition — but this limits throughput to one consumer.
5. Reliability Patterns
Distributed systems fail in ways monoliths don't: network timeouts, partial failures, cascading outages.
These patterns build resilience into your services.

5.1 Circuit Breaker Pattern

Prevents cascading failures by short-circuiting calls to a failing service. Acts like an electrical circuit
breaker — trips when overloaded, resets after recovery.

// Resilience4j Circuit Breaker — Java

@Configuration
public class ResilienceConfig {
@Bean
public CircuitBreaker paymentCircuitBreaker(CircuitBreakerRegistry registry) {
CircuitBreakerConfig config = [Link]()
.slidingWindowType(SlidingWindowType.COUNT_BASED)
.slidingWindowSize(20) // Last 20 calls
.failureRateThreshold(50) // Trip at 50% failure rate
.waitDurationInOpenState([Link](30)) // Wait before probing
.permittedNumberOfCallsInHalfOpenState(5) // 5 test calls in HALF-OPEN
.slowCallRateThreshold(80) // Also trip on 80% slow calls
.slowCallDurationThreshold([Link](2)) // 'Slow' = >2 seconds
.build();
return [Link]('payment-service', config);
}
}

// Usage with fallback

@Service
public class OrderService {
private final CircuitBreaker cb;
private final PaymentServiceClient paymentClient;

public PaymentResult chargeCustomer(String orderId, BigDecimal amount) {

return [Link](
[Link](cb, () -> [Link](orderId,
amount))
).recover([Link], ex -> {
// Circuit is OPEN — return fallback
[Link]('Payment circuit OPEN, queuing for retry: {}', orderId);
[Link](orderId, amount); // Async retry
return [Link](orderId);
}).get();
}
}

Circuit Breaker State Machine

State Behavior Transition Trigger

CLOSED All calls pass through. Failures counted. Failure rate > threshold → OPEN
 State Behavior Transition Trigger

OPEN All calls fail immediately (no network call). Wait duration elapsed → HALF-OPEN
Fallback returned.

HALF-OPEN Limited test calls allowed through. Tests succeed → CLOSED; Tests fail
→ OPEN

Q How do you choose the right threshold values for a Circuit Breaker?
A Start with the service's SLA and observed failure patterns. Sliding window size: enough calls to be
statistically meaningful — 20-50 for high-traffic services. Failure rate threshold: typically 50-60% —
lower than 50% may trip on normal transient errors; higher than 60% means too many failures before
tripping. Wait duration: should exceed the expected recovery time of the downstream service — if a
service typically restarts in 30 seconds, use a 30-60 second wait. Permitted calls in HALF-OPEN: 3-5
calls is enough to determine health without overwhelming a recovering service. Monitor circuit state
metrics (Prometheus gauges for state: 0=CLOSED, 1=OPEN, 2=HALF-OPEN) and tune based on
production data. Also consider slow calls — a circuit should trip on high latency as well as errors, since
a slow service can exhaust thread pools just as badly as a failing one.

5.2 Retry with Exponential Backoff + Jitter

Automatically retry failed operations for transient failures. Always use exponential backoff to avoid
thundering herd, and add jitter (randomness) so retrying clients don't all hit the server simultaneously.

# Python: Retry with exponential backoff + full jitter

import time, random, functools

def retry(max_attempts=3, base_delay=1.0, max_delay=60.0, exceptions=(Exception,)):

def decorator(func):
@[Link](func)
def wrapper(*args, **kwargs):
for attempt in range(max_attempts):
try:
return func(*args, **kwargs)
except exceptions as e:
if attempt == max_attempts - 1:
raise # Last attempt — propagate
# Exponential backoff with full jitter
# Full jitter: sleep = random(0, min(cap, base * 2^attempt))
delay = [Link](0, min(max_delay, base_delay * (2 ** attempt)))
[Link](f'Attempt {attempt+1} failed: {e}. Retrying in
{delay:.2f}s')
[Link](delay)
return wrapper
return decorator

@retry(max_attempts=3, base_delay=1.0, exceptions=(RequestException, Timeout))

def call_inventory_service(order_id: str) -> dict:
response = [Link](f'{INVENTORY_URL}/reserve', json={'order_id': order_id},
timeout=5)
response.raise_for_status()
return [Link]()

# Backoff schedule example (base=1s, cap=60s):

# Attempt 1 failure → sleep random(0, 2) seconds
# Attempt 2 failure → sleep random(0, 4) seconds
# Attempt 3 failure → sleep random(0, 8) seconds
# Attempt 4 failure → sleep random(0, 16) seconds

Critical: Idempotency Before Retrying

NEVER retry non-idempotent operations without idempotency keys. Retrying a
POST /charge endpoint can result in double charges. Make operations idempotent:
1. Client generates idempotency key (UUID) once per logical operation.
2. Send key in header: Idempotency-Key: <uuid>
3. Server caches result for this key (Redis, TTL 24h). Duplicate = return cached result.
4. Now retry is safe — duplicates return the original result, not a new charge.

Q What is the difference between at-most-once, at-least-once, and exactly-once delivery?

A At-most-once: Message is sent once; if it fails, it's dropped. Never duplicated but may be lost. Use
for non-critical metrics, logging where loss is acceptable. At-least-once: Message is retried until
acknowledged. Guaranteed delivery but may result in duplicates. Use for most business operations
with idempotent consumers. Exactly-once: Each message is processed exactly once, no loss and no
duplicates. Theoretically sound but requires distributed coordination overhead. In practice, use at-
least-once delivery with idempotent consumers — it's simpler, more performant, and effectively
achieves exactly-once semantics from the business perspective. Kafka supports exactly-once
semantics (EOS) within a Kafka cluster using transactions and idempotent producers, but cross-
system exactly-once (Kafka + database) still requires the Outbox Pattern and consumer idempotency.

5.3 Bulkhead Pattern

// Resilience4j Bulkhead — isolate thread pools per dependency

@Bean
public ThreadPoolBulkhead inventoryBulkhead() {
ThreadPoolBulkheadConfig config = [Link]()
.maxThreadPoolSize(10) // Max 10 threads for inventory calls
.coreThreadPoolSize(5) // Keep 5 warm
.queueCapacity(20) // Queue up to 20 pending calls
.keepAliveDuration([Link](20))
.build();
return [Link]().bulkhead('inventory', config);
}

@Bean
public ThreadPoolBulkhead paymentBulkhead() {
ThreadPoolBulkheadConfig config = [Link]()
.maxThreadPoolSize(10) // Completely separate pool for payments
.coreThreadPoolSize(5)
.queueCapacity(20)
.build();
return [Link]().bulkhead('payment', config);
}

// If Inventory exhausts its 10+20=30 capacity, Payment is UNAFFECTED

// Without bulkheads: both share the same JVM thread pool — one slow service kills all
Q How do you combine Circuit Breaker, Retry, Bulkhead, and Timeout — in what order?
A The correct order when wrapping a remote call (outermost to innermost): (1) Bulkhead — limits
concurrency; should wrap the whole call including retry. (2) Circuit Breaker — if open, fails fast before
even attempting the call. (3) Retry — retries the underlying call on transient failures. (4) Timeout —
wraps each individual attempt so it doesn't hang. So the call stack is: Bulkhead → Circuit Breaker →
Retry → Timeout → actual HTTP call. Rationale: Bulkhead is outermost because it limits total
concurrent operations regardless of retry loops. Circuit Breaker is next to short-circuit without
consuming bulkhead capacity. Retry is inside Circuit Breaker so failed retries are recorded
(contributing to failure rate). Timeout is innermost so each attempt has its own deadline. In
Resilience4j, apply decorators in this order using the Decorators utility class.
6. Observability Patterns

6.1 Distributed Tracing

A trace represents a single request's journey across all services. Each service adds a span (a unit of
work with start/end time) that is a child of the calling service's span. Together they form a trace tree.

# OpenTelemetry — Python instrumentation

from opentelemetry import trace
from [Link] import TracerProvider
from [Link] import JaegerExporter
from [Link] import BatchSpanProcessor

# Setup (once at startup)

tracer_provider = TracerProvider()
jaeger_exporter = JaegerExporter(agent_host_name='jaeger', agent_port=6831)
tracer_provider.add_span_processor(BatchSpanProcessor(jaeger_exporter))
trace.set_tracer_provider(tracer_provider)
tracer = trace.get_tracer('order-service')

class OrderService:
def place_order(self, request: PlaceOrderRequest) -> str:
with tracer.start_as_current_span('place-order') as span:
span.set_attribute('[Link]', request.customer_id)
span.set_attribute('[Link]', float([Link]))

# Child span for DB write

with tracer.start_as_current_span('[Link]-order'):
order = [Link](request)

# Child span for downstream call (traceId propagated via HTTP headers)
with tracer.start_as_current_span('[Link]-service'):
self.inventory_client.reserve([Link], [Link])

span.set_attribute('[Link]', [Link])
return [Link]

# HTTP headers automatically injected: traceparent: 00-{traceId}-{spanId}-01

# All downstream services extract and continue the trace

6.2 Structured Logging

// Structured JSON logging — every log includes traceId for correlation
import [Link];

public class OrderController {

private static final Logger log = [Link]([Link]);

@PostMapping("/orders")
public ResponseEntity<OrderResponse> placeOrder(@RequestBody PlaceOrderRequest req) {
// traceId injected by OpenTelemetry auto-instrumentation into MDC
// [Link]("traceId") → "abc123xyz"
[Link]("Placing order",
 kv("event", "[Link]"),
kv("customerId", [Link]()),
kv("itemCount", [Link]().size()),
kv("total", [Link]())
);
// Outputs:
// {"timestamp":"2024-01-15T10:30:00Z","level":"INFO","service":"order-service",
// "traceId":"abc123xyz","spanId":"def456","event":"[Link]",
// "customerId":"USR-789","itemCount":3,"total":149.99}
}
}

Q What is the difference between logging, metrics, and tracing and when do you use each?
A They answer different questions: Logs answer 'What happened?' — discrete events with full context
(error messages, request details). Use when debugging a specific incident or auditing what occurred.
Query by traceId to see all events for one request. Metrics answer 'How is the system performing?' —
numeric measurements aggregated over time (request rate, error %, latency percentiles, CPU). Use
for monitoring health, capacity planning, alerting, and SLA reporting. Dashboards show trends; alerts
fire when thresholds are crossed. Traces answer 'Where did the time go?' — a request's journey
across services with timing for each step. Use when debugging latency issues or understanding which
service in a call chain is slow. All three must be correlated by a common trace ID. The modern
standard is OpenTelemetry, which provides a unified API to emit all three signals with automatic
correlation.
7. Deployment Patterns

7.1 Blue/Green Deployment

Maintain two identical production environments. 'Blue' is live; 'Green' has the new version. Switch traffic
atomically. Rollback = switch back instantly.

# Kubernetes: Blue/Green with Service selector switch

# Blue deployment (currently live)
apiVersion: apps/v1
kind: Deployment
metadata:
name: order-service-blue
spec:
replicas: 3
selector:
matchLabels: { app: order-service, version: blue }
template:
metadata:
labels: { app: order-service, version: blue }
spec:
containers:
- image: order-service:v1.2.0

# Green deployment (new version, receives no traffic yet)

apiVersion: apps/v1
kind: Deployment
metadata:
name: order-service-green
spec:
replicas: 3
selector:
matchLabels: { app: order-service, version: green }
template:
metadata:
labels: { app: order-service, version: green }
spec:
containers:
- image: order-service:v1.3.0

# Service points to blue

apiVersion: v1
kind: Service
metadata:
name: order-service
spec:
selector: { app: order-service, version: blue }

# Switch to green (one kubectl command):

kubectl patch service order-service -p '{"spec":{"selector":{"version":"green"}}}'
# Rollback: patch back to blue — instant, zero downtime
7.2 Canary Deployment
Gradually roll out a new version to a small percentage of traffic, monitor it, then ramp up if healthy.

# Kubernetes + Istio: Canary with traffic splitting

apiVersion: [Link]/v1alpha3
kind: VirtualService
metadata:
name: order-service
spec:
http:
- route:
- destination:
host: order-service-stable
port: { number: 80 }
weight: 90 # 90% to stable (v1.2.0)
- destination:
host: order-service-canary
port: { number: 80 }
weight: 10 # 10% to canary (v1.3.0)

# After monitoring for 30 minutes with no errors:

# Update weights: stable=70, canary=30
# After another 30 minutes: stable=0, canary=100
# Promote canary to stable, delete old deployment

# Argo Rollouts automates this progression:

spec:
strategy:
canary:
steps:
- setWeight: 10
- pause: { duration: 30m } # Auto-pause, check metrics
- setWeight: 50
- pause: { duration: 30m }
- setWeight: 100
analysis: # Auto-rollback if error rate > 1%
templates:
- templateName: error-rate-analysis

Q How does a Canary deployment differ from A/B testing?

A Canary is a deployment strategy focused on risk reduction — you're releasing the same feature to a
small subset of users to catch bugs before full rollout. Traffic splitting is typically random (10% gets
new version). The goal is to validate that the new version is stable and correct before committing. A/B
testing is a product experimentation strategy — you're intentionally serving different experiences to
different user segments to measure which performs better (higher conversion, more clicks, longer
session). Traffic is split by user cohort (e.g., users in cohort A get button color red; cohort B gets blue).
The goal is statistical analysis of business metrics, not deployment safety. In practice, both use similar
traffic-splitting infrastructure (Istio VirtualService, feature flags), but the intent, duration, and success
criteria are different. Canary ends when the new version is fully promoted. A/B tests run for statistical
significance and end when a winner is declared.
7.3 Sidecar Pattern
Deploy a secondary container alongside every service instance to handle cross-cutting infrastructure
concerns transparently.

# Kubernetes Pod: Application + Envoy Sidecar

apiVersion: v1
kind: Pod
metadata:
name: order-service
annotations:
[Link]/inject: 'true' # Istio auto-injects Envoy sidecar
spec:
containers:
- name: order-service # Your application
image: order-service:v1.3.0
ports:
- containerPort: 8080
# Envoy sidecar is auto-injected below by Istio:
- name: istio-proxy
image: [Link]/istio/proxyv2:1.20.0
# Handles: mTLS, circuit breaking, retries, tracing, rate limiting
# Application code has ZERO knowledge of these concerns

# All inbound/outbound traffic routes through Envoy

# Application only talks to localhost:8080
# Envoy intercepts and applies policies transparently

7.4 Service Discovery

Approach How It Works Example Pros / Cons

Client-Side Service registers with Netflix Eureka + Pro: Client controls LB algo. Con:
registry. Client queries Ribbon Each client needs discovery
registry, picks instance, library.
calls directly.

Server-Side Client calls load balancer. AWS ELB + Route Pro: Client is simple. Con: LB can
LB queries registry and 53, k8s Service + be a bottleneck/SPOF.
routes. kube-proxy

DNS-Based Services registered as Consul DNS, k8s Pro: Works with any
DNS SRV records. Client CoreDNS language/framework. Con: DNS
resolves via DNS. caching delays updates.

Service Mesh Envoy sidecar handles Istio + Envoy Pro: Fully transparent. Con:
discovery. Control plane Significant operational
(Istio) manages registry. complexity.

Q How does Kubernetes handle service discovery and load balancing internally?
A Kubernetes provides built-in service discovery via DNS and kube-proxy. When you create a Service
resource (kind: Service), Kubernetes: (1) Assigns a stable virtual IP (ClusterIP). (2) Creates a DNS
entry: [Link] resolving to the ClusterIP. (3) kube-proxy (running on
every node) watches the API server for Endpoint changes and programs iptables or IPVS rules to
route ClusterIP traffic to one of the healthy pod IPs (round-robin by default). When a pod starts, it's
added to Endpoints; when it fails liveness/readiness probes, it's removed — automatically removing it
from the load balancer without any service discovery library in your code. For more advanced LB (least
connections, consistent hashing, circuit breaking), use a service mesh like Istio that replaces kube-
proxy's simple round-robin with Envoy's rich traffic management.
8. Master Interview Q&A Bank
These are the most frequently asked, highest-signal questions at senior/architect-level microservices
interviews at FAANG and unicorn companies. Each answer demonstrates depth, trade-off awareness,
and real-world experience.

8.1 Architecture & Design

Q Design a ride-sharing system (Uber/Lyft) using microservices. Walk me through the services
and key patterns.
A Core services: User Service (profiles, auth), Driver Service (driver state, location), Trip Service (trip
lifecycle), Matching Service (driver-rider matching), Pricing Service (dynamic surge pricing), Payment
Service (billing, payout), Notification Service (push/SMS). Key patterns: Driver location uses event
streaming (Kafka) — drivers publish location every 5 seconds; Matching Service consumes this to find
nearby drivers. Trip lifecycle uses Saga — CreateTrip → MatchDriver (with timeout/fallback to next
driver) → ConfirmPickup → CompleteTrip → ChargePayment. Surge pricing is CPU-intensive; extract
it to a dedicated service that uses CQRS: writes go to the pricing write model (current supply/demand
data), reads serve a pre-computed surge multiplier from a cache. Real-time updates use WebSocket
connections managed by a Gateway Service. Geospatial queries use Redis GEOSEARCH or a
specialized geo-index.

Q How would you break apart a 10-year-old e-commerce monolith step by step?
A Step 1: Instrument the monolith — add distributed tracing, profiling, and module-level metrics to
understand coupling and hot paths. Step 2: Identify seams using Event Storming — map domain
events and find bounded contexts. Step 3: Prioritize extraction by: highest value (frequently changed
code), lowest risk (fewest dependencies), fastest feedback. Notification service is usually first — few
inbound dependencies, mostly outbound. Step 4: Extract using Strangler Fig — build the service, add
routing layer, migrate data, validate, remove from monolith. Step 5: Handle data — use dual writes
(write to both monolith DB and new service DB via Outbox) during migration, then cut over. Step 6:
Address cross-cutting concerns — shared auth, logging, tracing must be established early and each
service adopts them. Timeline: plan for 18-36 months for a large monolith. Never do a big-bang
rewrite.

Q What is the CAP theorem and how does it affect your microservices design choices?
A CAP theorem states that a distributed system can guarantee at most two of: Consistency (every
read sees the latest write), Availability (every request gets a response), Partition Tolerance (system
works despite network failures). Since network partitions are unavoidable in distributed systems, you
must choose between CP (consistency + partition tolerance) or AP (availability + partition tolerance).
CP systems (e.g., HBase, etcd, Zookeeper) refuse to serve potentially stale data during a partition —
correct data or no data. AP systems (e.g., Cassandra, DynamoDB, CouchDB) continue serving
potentially stale data during a partition — always available but may be inconsistent. In microservices,
most user-facing services should be AP (high availability, eventual consistency) because downtime is
worse than temporary staleness. Financial systems requiring strict accuracy should be CP. Design
accordingly: use AP databases for product catalog and user sessions; use CP for payment ledgers
and inventory reservations.
8.2 Data & Consistency Deep Dives
Q Explain exactly how you would implement the Saga pattern for a flight booking system.
A Services involved: Booking Service (orchestrator), Flight Inventory Service, Seat Reservation
Service, Payment Service, Notification Service. Using orchestration (Temporal workflow): Step 1 —
Booking Service creates booking record (PENDING) and starts saga. Step 2 — Reserve flight
inventory (check availability, hold seat count for 10 minutes). Compensation: release hold. Step 3 —
Reserve specific seat (assign seat to passenger). Compensation: unreserve seat. Step 4 — Charge
payment (process card). Compensation: refund. Step 5 — Confirm booking (CONFIRMED). Step 6 —
Send confirmation email (fire-and-forget, no compensation needed). Each activity is idempotent: flight
inventory uses a reservationId for deduplication; payment uses an idempotency key. If Step 4
(payment) fails, Temporal automatically executes: unreserve seat (Step 3 compensation), release flight
hold (Step 2 compensation), cancel booking. The workflow is durable — if the process crashes mid-
saga, Temporal replays the workflow from the last completed step.

Q What is read-your-own-writes consistency and how do you implement it with CQRS?

A Read-your-own-writes (RYOW) means: after a user writes data, their subsequent reads immediately
reflect that write. This is violated in CQRS/event-driven systems because the read model updates
asynchronously — a user submits a new address, the write succeeds, but the read model hasn't
projected the change yet. The user refreshes and sees their old address. Solutions: (1) Version tokens
— the write returns a version number. The read request includes this version as a minimum; the read
service waits until the read model has processed at least that version. (2) Return data from the write —
include the new state in the write response so the client doesn't need to read immediately. (3) Sticky
routing — route read requests from the same user session to the same replica, which has already
applied the write. (4) Cache on write — after a command, update the cache directly with the new state
so reads serve from cache. Approach 2 (return data from write) is the simplest and most practical.

Q How do you handle schema evolution in an event-sourced system?

A Events are immutable once written, so you can't change past events when the schema changes.
Strategies: (1) Additive changes (new optional fields) — old consumers ignore unknown fields if using
JSON/Avro with forward-compatible schemas. This is the safest approach; prefer it whenever possible.
(2) Upcasters — when loading old events, apply a transformation function (upcaster) that converts old
event format to the new format on the fly. The event store stores v1 events; the aggregate always
receives v2 events after upcasting. (3) New event types — instead of changing OrderCreated, create
OrderCreatedV2 with the new schema. Handle both in the aggregate. (4) Snapshotting + migration —
snapshot all aggregates in the new format, discard events older than the snapshot. This loses history
but simplifies the codebase. Use a schema registry (Confluent) with compatibility rules
(FULL_TRANSITIVE) to prevent accidental breaking changes in event schemas.

8.3 Reliability & Operations

Q A microservice is experiencing intermittent 503 errors from a downstream service. Walk me
through your debugging process.
A Structured approach: (1) Check distributed traces — find a specific failing request by Trace ID (from
error logs or user report). Examine the trace to identify which span is failing and at what latency. (2)
Check metrics for the downstream service — is its error rate elevated? Is it CPU/memory constrained?
Is the connection pool exhausted? (3) Check network — are the 503s from the service itself or from
the load balancer/ingress? Load balancer 503 = no healthy instances. Service 503 = the service is
rejecting requests (rate limiting, circuit breaker open, thread pool exhausted). (4) Check recent
 deployments — did any service change in the last hour? (5) Check infrastructure — is the node the
service runs on under pressure? Are there network partition events in cloud provider logs? (6)
Mitigation while debugging — if the circuit breaker is open, evaluate if the fallback behavior is
acceptable. Consider scaling up the failing service or temporarily reducing traffic to it via canary
rollback. (7) Root cause — correlate all data and document the failure mode, add alerting to detect it
earlier next time.

Q How do you implement distributed rate limiting across multiple instances of an API
Gateway?
A Single-instance rate limiting (in-memory counters) doesn't work across multiple gateway instances.
Distributed rate limiting requires a shared state store. Implementation with Redis: use a sliding window
counter algorithm. For each request, increment a counter in Redis with a key of
user_id:window_timestamp (e.g., 1-second windows). Use Redis INCR + EXPIRE atomically (via Lua
script or Redis pipeline). If counter > limit, return 429. Redis can handle 100K+ operations/second with
sub-millisecond latency, making this viable even for high-traffic gateways. More sophisticated: token
bucket algorithm with Redis — store remaining tokens and last refill time per user. Use the Lua script
for atomic check-and-consume. For extreme scale, use local rate limiting with a small burst allowance
per instance plus Redis for global rate limiting — check local first (fast path), sync to Redis periodically.
Libraries: redis-rate-limiter, Envoy's built-in rate limiting (calls a gRPC rate limit service).

Q How do you ensure data consistency when a service needs to roll back due to a failed Saga
while some compensating transactions also fail?
A This is the 'non-atomic compensation' problem. Defense-in-depth approach: (1) Design
compensations to be idempotent and retryable — they should always eventually succeed given
enough retries. Use exponential backoff with very long retry horizons (hours, not seconds) for
compensations. (2) Use a reliable workflow engine (Temporal, AWS Step Functions) that durably
persists workflow state — even if your process crashes, compensation retries continue after restart. (3)
For compensations that truly cannot be automated (e.g., a refund to a closed card), implement a dead
letter queue (DLQ) that triggers a manual review workflow with operator notification and tooling to
perform the action manually. (4) Accept that some states require human intervention — design your
system to handle 'stuck' orders gracefully (show them in an admin UI, alert on-call). (5) Implement a
reconciliation job that periodically audits saga-in-progress records and alerts/resolves ones that are
stuck beyond a time threshold. Perfect automated compensation isn't always achievable; the system
must degrade gracefully to manual processes.

8.4 Pattern Selection & Trade-offs

Q When would you choose event-driven choreography vs. orchestration for a Saga?
A Choreography when: (1) The flow has 2-3 steps and is unlikely to grow complex. (2) Services are
owned by different teams who need maximum autonomy — adding a step doesn't require changing a
central coordinator. (3) You already have a robust event streaming platform and teams experienced
with event-driven design. (4) The flow is truly reactive — each service's behavior naturally follows from
events in its domain. Orchestration when: (1) The flow is complex (5+ steps) with conditional branches
and complex compensation logic. (2) You need a clear audit trail of saga state — the orchestrator's
state machine provides this naturally. (3) The flow spans many teams and you need a single place to
understand and modify it. (4) You're using a mature workflow engine (Temporal, Conductor) that
handles durability, retries, and timeouts automatically. In practice: many teams start with choreography
for simplicity, then switch to orchestration as flows grow complex. The switch is painful — prefer
starting with orchestration for any flow with more than 3 steps or complex failure handling.
 Q How do you decide the right granularity for a microservice? What's the cost of services that
are too fine-grained or too coarse-grained?
A Too fine-grained (nano-services): Services so small they don't encapsulate a meaningful capability
— e.g., a separate service just to validate email addresses. Cost: excessive inter-service
communication (chatty), increased latency for every operation, operational overhead multiplies (N
services = N deployments, N monitoring setups, N security configurations). Teams spend more time
managing infrastructure than building features. Too coarse-grained (distributed monolith): Services so
large they contain multiple unrelated capabilities — e.g., one service handles products, orders, AND
payments. Cost: teams are not independent (need to coordinate deployments), the service can't be
scaled for its individual workloads, a failure in one area takes down unrelated capabilities. The right
size: a service should be owned by a team that can develop, deploy, and operate it independently. A
useful heuristic — a service should have one reason to change (SRP applied at service level). In
practice, a 2-pizza team (5-8 engineers) owning 1-3 services is a healthy ratio. If you need to change
two services for one feature routinely, merge them. If one service is changed by two different teams
routinely, split it.

Q What are the key considerations when choosing between REST and event-driven
communication for a new microservice integration?
A Use synchronous REST when: the client needs an immediate response to continue its work (user-
facing request/response), strong consistency is required (must know the result before proceeding), the
operation is naturally request/reply in nature (query a resource, submit a form). Use event-driven
async when: the operation can complete in the background (sending an email, generating a report,
updating a search index), you need to fan out to multiple consumers (audit logging, analytics, caching),
you want to decouple services so they can evolve independently, or the downstream service may be
slow/unavailable and you can't hold the user waiting. A practical guide: if the user is waiting for the
response → REST. If the user doesn't need to wait (fire and forget from user's perspective) → events.
If the system needs to coordinate multiple services atomically → events + Saga. Remember: REST
calls can be made more resilient with retry/circuit breaker but still couple services temporally. Events
decouple services but introduce eventual consistency. Most real-world systems use both — REST for
queries and user-facing writes, events for notifications, projections, and cross-domain workflows.

8.5 Rapid-Fire Conceptual Questions

Q What is an idempotency key and how do you implement it?
A An idempotency key is a unique client-generated identifier (UUID) that the server uses to
deduplicate repeated requests. Implementation: client generates key once per logical operation and
sends it in the header (Idempotency-Key: <uuid>). Server checks Redis/DB for this key. If found, return
the cached result without re-executing. If not found, execute, store result with the key (TTL: 24 hours),
return result. Use Redis SETNX for atomic check-and-set. The key lookup and operation must be
atomic — use a distributed lock or database transaction to prevent two concurrent requests with the
same key from both executing.

Q What is the difference between horizontal and vertical scaling and when do microservices
help?
A Vertical scaling (scale up): add more CPU/RAM to the same server. Simple but has hardware limits
and creates a single point of failure. Horizontal scaling (scale out): add more instances of the same
service. Microservices enable fine-grained horizontal scaling — scale only the services that are under
load. In a monolith, you must scale everything together (wasteful). With microservices, if Order Service
is the bottleneck during a sale, scale it from 3 to 30 instances without touching Payment or Catalog
services. Kubernetes makes this automatic with Horizontal Pod Autoscaler (HPA) based on
CPU/memory/custom metrics.

Q How do you handle service-to-service authentication?

A Several approaches by security level: (1) Mutual TLS (mTLS) — both services authenticate each
other with certificates. Automatic with Istio — no code changes needed. Strongest option, prevents
unauthorized services from calling each other. (2) JWT service accounts — each service has an
identity JWT signed by a central authority; other services validate the signature. Services can express
their identity in claims. (3) API keys — simple shared secret in request header. Easy but hard to rotate
and less granular. (4) Network-level — rely on Kubernetes NetworkPolicy to restrict which pods can
reach which. Good as defense-in-depth but not authentication. In production, combine mTLS (for
transport security and identity) with authorization policies (RBAC) that specify which service identities
can call which operations.

Q What is a Dead Letter Queue (DLQ) and when is it used?

A A DLQ is a special queue/topic where messages are routed after exhausting all retry attempts and
still failing. It prevents bad messages from blocking the main queue and allows inspection and replay.
Scenarios: malformed message that can't be deserialized, a processing bug that causes consistent
failure, or a downstream dependency outage. Implementation: configure max retry attempts (e.g., 3)
with delays. After 3 failures, route message to DLQ. Set up: alerting when DLQ has messages, a
dashboard showing DLQ message details, tooling to replay DLQ messages after fixing the bug. In
Kafka, this is typically implemented by the consumer: on repeated failure, produce to a .DLQ topic. In
SQS, it's built-in — configure MaxReceiveCount and a DLQ ARN on the main queue.

Q How does a service mesh provide mTLS without changing application code?
A The service mesh uses the sidecar pattern: an Envoy proxy is injected into every pod as a sidecar
container. iptables rules (set up by an init container) redirect all inbound and outbound traffic through
Envoy. The application thinks it's making a plain HTTP call to localhost; Envoy intercepts it, performs
the mTLS handshake with the destination's Envoy sidecar, and sends the request over an encrypted
tunnel. The destination Envoy decrypts it and forwards to the local application container — again over
plain HTTP on localhost. From the application's perspective, zero encryption code. The Istio control
plane (istiod) acts as a Certificate Authority, issuing short-lived certificates (e.g., 24-hour SVID —
SPIFFE Verifiable Identity Document) to each Envoy. Certificates rotate automatically. Authorization
policies (which service identity can call which service's endpoint) are also enforced by Envoy
transparently.
9. Pattern Quick Reference

9.1 Pattern Decision Matrix

Problem You Face Pattern to Apply Key Trade-off

How to split the application Decompose by Business Autonomy vs. cross-service communication
Capability / DDD
Subdomain

Migrating a monolith Strangler Fig Long-lived migration, dual maintenance

incrementally

Services sharing a database Database per Service + Data duplication, no joins, eventual
API access consistency

Distributed transaction Saga (Orchestration Eventual consistency, complex

across services preferred for complex compensation logic
flows)

Read/write workloads at CQRS Two codebases, eventual consistency on

different scale reads

Full audit trail required Event Sourcing Complex querying, schema evolution
difficulty

Atomic DB write + event Outbox Pattern Additional table + relay process to operate
publish

Single entry point for clients API Gateway Single point of failure, must be highly
available

Different API shape per client Backend for Frontend Multiple backends to maintain
type (BFF)

Cascading failures from slow Circuit Breaker Requires tuning; false positives trip
service unnecessarily
Resource exhaustion from Bulkhead Thread pool overhead; over-partitioning
one consumer wastes threads

Transient failures on calls Retry + Exponential Only for idempotent ops; adds latency on
Backoff + Jitter failure

Zero-downtime release with Blue/Green Deployment 2x infrastructure cost during deployment

instant rollback

Safe gradual rollout with Canary Deployment Traffic splitting infrastructure required
monitoring

Cross-cutting concerns Service Mesh (Sidecar) Significant operational complexity

(mTLS, tracing)

Fanout to many consumers Pub/Sub (Kafka) Eventual consistency, consumer offset

from one event management

Debugging across service Distributed Tracing Instrumentation effort, trace storage cost
 Problem You Face Pattern to Apply Key Trade-off

boundaries (OpenTelemetry)

9.2 Anti-Patterns to Avoid

Anti-Pattern Description Fix

Shared Database Multiple services read/write the Database per service + API access
same DB tables

Chatty Services Service A calls B, B calls C, C calls Consolidate services or use async
D synchronously for every request events

Distributed Monolith Services are deployed separately Enforce API contracts, separate
but coupled by shared DB/tight API databases
contracts

God Service One service owns too many Decompose using DDD bounded
business capabilities contexts

Synchronous Saga Saga steps use synchronous REST Use async messaging or workflow
calls instead of async events engine

No Idempotency Retried POST requests create Add idempotency keys to all write APIs
duplicate records/charges

Skipping Outbox DB write + direct broker publish Use Outbox Pattern with CDC relay
without transaction

Premature Splitting into microservices before Start modular monolith; extract when
Decomposition domain is understood pain is real

Ignoring Observability No distributed tracing, no structured OpenTelemetry from day one

logs, no metrics

Key Mindset for Interviews & Real Projects

1. Always discuss trade-offs — every pattern solves one problem and creates another.
2. Start simple — most services don't need Saga, CQRS, AND Event Sourcing. Add patterns
incrementally.
3. Consistency model first — decide strong vs eventual consistency before choosing patterns.
4. Failure modes — always ask: what happens when service X is down? Design for it.
5. Conway's Law — team structure drives architecture. Align teams with service boundaries.
6. Operational readiness — a pattern that works in development but can't be operated in production
is useless.
7. Domain understanding — no technical pattern compensates for wrong service boundaries.