Apache Kafka

Complete Guide

Core Concepts & Architecture

with JavaScript/NestJS Examples

The 20% of Kafka knowledge that covers 80% of daily usage in production environments,
technical interviews, and software engineering roles.
Table of Contents
1. Core Kafka Concepts & Architecture

1.1 Basic Components

1.2 Message Anatomy

1.3 Replication

2. Complete Message Lifecycle

3. JavaScript/NestJS Examples

3.1 Producer Examples

3.2 Consumer Examples

3.3 NestJS Integration

1. Core Kafka Concepts & Architecture

1.1 Basic Components

Topics
Definition: A topic is a logical channel or category to which records are published. Think of it as a
named feed of messages, similar to a table in a database or a folder in a file system.

Key Characteristics:

• Logical abstraction: Topics don't physically store data; partitions do

• Named entities: Each topic has a unique name within a cluster (e.g., 'user-events',
'payment-transactions')
• Multi-producer/multi-consumer: Multiple producers can write to a topic, and multiple
consumer groups can read simultaneously
• Append-only: Messages are always appended to the end; they're never modified in place
• Configurable retention: Data can be retained based on time (e.g., 7 days) or size (e.g., 1GB
per partition)

Naming Conventions (Best Practices):

<domain>.<entity>.<event-type>

Examples:
- [Link]
- [Link]
- [Link]-changed

Real-World Example:

E-commerce system topics:

- [Link] (new orders)
- [Link] (shipping updates)
- [Link] (cancellations)
- [Link] (stock changes)
- [Link] (new users)

Partitions
Definition: A partition is the physical unit of storage and parallelism in Kafka. Each topic is divided
into one or more partitions, and each partition is an ordered, immutable sequence of records.

Key Characteristics:

• Ordered Sequence: Within a single partition, messages maintain strict ordering by offset
 • Immutable: Once written, records cannot be modified (only appended or deleted by retention
policy)
• Distributed: Different partitions of the same topic can reside on different brokers
• Unit of Parallelism: Each partition can be consumed by only one consumer within a consumer
group

Why Partitions Matter:

Scalability Example:
Topic: "user-events" with 1 partition → Throughput: 10 MB/s (limited by single partition)
Topic: "user-events" with 10 partitions → Throughput: 100 MB/s (10x parallelism)

Ordering Guarantees:

✓ Guaranteed: Order within a partition

✗ Not guaranteed: Order across partitions

Topic: payments (3 partitions, 3 brokers)

Broker 1: Partition 0 [msg1, msg2, msg3, ...]

Broker 2: Partition 1 [msg4, msg5, msg6, ...]
Broker 3: Partition 2 [msg7, msg8, msg9, ...]

Choosing Partition Count:

Formula:
Partitions = max(Target Throughput / Producer Throughput per Partition,
Target Throughput / Consumer Throughput per Partition)

Example:
Target: 100 MB/s throughput
Producer: 10 MB/s per partition
Consumer: 20 MB/s per partition

Partitions needed = max(100/10, 100/20) = max(10, 5) = 10 partitions

Best Practices:

• Start with 3-6 partitions for small topics

• Use 10-30 partitions for medium workloads
• Consider future growth (can't easily decrease partitions)
• More partitions = more file handles and memory on brokers

Brokers
Definition: A Kafka broker is a server that stores data and serves client requests. It's the
fundamental building block of a Kafka cluster.
Responsibilities:

• Data Storage: Stores partition data on disk, manages log segments, handles retention
• Request Handling: Accepts produce requests, serves fetch requests, responds to metadata
requests
• Replication: Replicates data from leader partitions, serves as leader or follower
• Coordination: Participates in leader election, sends heartbeats to maintain membership

/var/lib/kafka/data/
■■■ payments-0/ (topic: payments, partition: 0)
■ ■■■ [Link] (segment file)
■ ■■■ [Link] (offset index)
■ ■■■ [Link]
■ ■■■ leader-epoch-checkpoint
■■■ payments-1/
■ ■■■ ...
■■■ payments-2/
■■■ ...

Hardware Considerations:

• CPU: Moderate (compression, decompression)

• Memory: 6-64 GB (page cache is critical)
• Disk: Fast I/O (SSD preferred for high throughput)
• Network: High bandwidth (10 Gbps+ for large clusters)

Cluster
Definition: A Kafka cluster is a group of brokers working together to provide high availability,
scalability, and fault tolerance.

Cluster Components:

• Brokers: Multiple servers (typically 3+ for production)

• Controller: Special broker managing cluster state
• ZooKeeper/KRaft: Coordination service (KRaft is the new native mode)

Controller Role:

Elected: One broker is elected as controller

Responsibilities:
• Manages partition leader election
• Handles broker failures
• Propagates metadata changes
• Maintains cluster state

Cluster Coordination:
 Traditional (ZooKeeper):
ZooKeeper Ensemble (3-5 nodes)
↓
Kafka Cluster (3+ brokers)
- Broker 1 (Controller)
- Broker 2
- Broker 3

Modern (KRaft - Kafka 3.0+):

Kafka Cluster with KRaft
- Broker 1 (Controller + Data)
- Broker 2 (Controller + Data)
- Broker 3 (Controller + Data)

No external ZooKeeper needed!

Cluster Benefits:

• High Availability: Broker failures don't cause downtime

• Load Distribution: Partition leaders spread across brokers
• Scalability: Add brokers to increase capacity
• Fault Tolerance: Data replicated across brokers
1.2 Message Anatomy
Every message (record) in Kafka consists of several components. Understanding these is crucial for
effective Kafka usage.

Complete Message Structure:

■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■ KAFKA MESSAGE ■
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■ Key (Optional) ■ Value (Required) ■
■ "user-123" ■ {"event": "purchase", ...} ■
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■ Headers (Optional) ■
■ { ■
■ "source": "mobile-app", ■
■ "trace-id": "abc-123", ■
■ "version": "2.0" ■
■ } ■
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■ Timestamp ■ Offset ■
■ 1706543210000 ■ 12345 ■
■ (Jan 29, 2024 10:00) ■ (within partition) ■
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■ Partition ■ Topic ■
■ 2 ■ "user-events" ■
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■

Key
Definition: An optional identifier used for message routing and log compaction.

Purpose:

• Partitioning: Messages with the same key go to the same partition

• Ordering: Guarantees order for messages with the same key
• Compaction: In compacted topics, keeps only the latest value per key

Key Selection Strategies (JavaScript/NestJS):

// Strategy 1: Entity ID (most common)

// All events for user-123 go to same partition
const message = {
key: 'user-123',
value: [Link]({
action: 'login',
timestamp: new Date().toISOString()
})
};

// Strategy 2: Composite Key

 // Group related entities together
const message = {
key: 'tenant-5:user-123',
value: [Link](orderData)
};

// Strategy 3: No Key (null)

// Round-robin distribution across partitions
const message = {
key: null,
value: [Link]({
log: 'System started'
})
};

// Strategy 4: Event Type

// All orders go to same partition (careful: hot partition risk!)
const message = {
key: 'order',
value: [Link](orderDetails)
};

Partitioning with Keys:

Partition = hash(key) % number_of_partitions

Example:
key = "user-123"
hash("user-123") = 2147483647
2147483647 % 3 = 0

Message goes to Partition 0

Common Use Cases:

Scenario Key Strategy Reason

User activity tracking User ID Maintain order per user

IoT sensor data Device ID Process device events in order

Financial transactions Account ID Ensure transaction ordering

Log aggregation null Distribute load evenly

Change data capture Primary key Latest state per record

Important Notes:
• Changing partition count breaks key-to-partition mapping
• Hot keys can create uneven partition load
• Key size impacts network and storage (keep small)
Value
Definition: The actual message payload - the data you want to transmit.

Characteristics:

• Required: Every message must have a value (unlike key)

• Format-agnostic: Can be any byte array
• Size limit: Default 1MB (configurable via [Link])

Common Value Formats:

1. JSON (Human-readable, flexible)

{
"user_id": "123",
"event_type": "purchase",
"amount": 99.99,
"timestamp": "2024-01-29T10:00:00Z",
"items": [
{"id": "prod-1", "quantity": 2}
]
}
Pros: Easy to debug, schema flexibility
Cons: Larger size, slower parsing, no schema enforcement

2. Avro (Schema-based, compact)

Schema:
{
"type": "record",
"name": "Purchase",
"fields": [
{"name": "user_id", "type": "string"},
{"name": "amount", "type": "double"},
{"name": "timestamp", "type": "long"}
]
}
Binary value: [encoded bytes]
Pros: Compact, schema evolution, type safety
Cons: Requires schema registry, harder to debug

3. Protobuf (Google's format)

message Purchase {
string user_id = 1;
double amount = 2;
int64 timestamp = 3;
}
Pros: Efficient, language support, backward compatible
Cons: Setup complexity

4. Plain String
"User 123 made a purchase of $99.99"
Pros: Simple
Cons: No structure, hard to parse
Headers
Definition: Key-value metadata pairs attached to messages, separate from the key and value.

Common Use Cases:

// 1. Tracing & Correlation

headers: {
'trace-id': 'abc-123-def-456',
'span-id': 'span-789',
'parent-span-id': 'span-456'
}

// 2. Message Routing
headers: {
'destination': 'email-service',
'priority': 'high'
}

// 3. Schema Information
headers: {
'schema-version': '2.5',
'content-type': 'application/avro'
}

// 4. Source Metadata
headers: {
'source-system': 'mobile-app',
'source-version': '1.2.3',
'client-ip': '[Link]'
}

// 5. Business Context
headers: {
'tenant-id': 'acme-corp',
'region': 'us-east-1',
'environment': 'production'
}

Best Practices:

• Keep headers small (< 1KB total)

• Use consistent naming conventions
• Don't duplicate data from key/value
• Useful for cross-cutting concerns (security, tracing)

Timestamp
Definition: The time associated with a message, indicating when it was created or ingested.

Two Timestamp Types:

 • CreateTime (Default): Set by producer when creating the message; reflects when the event
occurred
• LogAppendTime: Set by broker when the message is written; reflects when Kafka received it

Timestamp Uses:
1. Time-based Retention
Delete messages older than 7 days: [Link]=604800000

2. Time-based Seeking
// Seek to messages from specific time
const timestamp = new Date('2024-01-29T10:00:00Z').getTime();
await [Link]({ topic, partition, offset: timestamp });

3. Out-of-Order Detection
let lastTimestamp = 0;
for (const message of messages) {
if ([Link] < lastTimestamp) {
[Link]('Out-of-order message detected!');
}
lastTimestamp = [Link];
}

Offset
Definition: A unique, sequential identifier for each message within a partition. It's Kafka's primary
mechanism for tracking message position.

Key Characteristics:

• Unique per partition: Offset 100 in partition 0 ≠ offset 100 in partition 1

• Sequential: Increases by 1 for each new message
• Immutable: Once assigned, never changes
• Zero-indexed: Starts at 0 for each partition
• Never reused: Even after deletion, offsets continue incrementing

Topic: orders, Partition: 0

Offset ■ Message
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0 ■ {"order": "A"}
1 ■ {"order": "B"}
2 ■ {"order": "C"}
3 ■ {"order": "D"}
4 ■ {"order": "E"}
■
↓ (new messages appended)

Consumer Offset Management:

■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■ Partition (orders-0) ■
■ ■
■ Offset: 0 1 2 3 4 5 6 7 8 9 ■
■ ↑ ↑ ↑ ■
■ ■ ■ ■ ■
■ Earliest Committed Latest ■
■ Offset Offset Offset ■
■ ■
■ Committed: 5 (processed up to here) ■
■ Current: 7 (processing in progress) ■
■ Lag: 2 messages behind ■
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
1.3 Replication
Replication is Kafka's mechanism for fault tolerance and high availability. It ensures data durability by
maintaining multiple copies of each partition across different brokers.

Replication Architecture:

■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■ REPLICATION EXAMPLE ■
■ Topic: orders, Partition: 0, RF: 3 ■
■ ■
■ ■■■■■■■■■■■■■■■■ ■■■■■■■■■■■■■■■■ ■■■■■■■■■■■■■■■■ ■
■ ■ Broker 1 ■ ■ Broker 2 ■ ■ Broker 3 ■ ■
■ ■ ■ ■ ■ ■ ■ ■
■ ■ Partition 0 ■ ■ Partition 0 ■ ■ Partition 0 ■ ■
■ ■ [LEADER] ■■■■■■ [FOLLOWER] ■ ■ [FOLLOWER] ■ ■
■ ■ ■ ■ (ISR) ■ ■ (Out of sync)■ ■
■ ■ Offsets: ■ ■ ■ ■ ■ ■
■ ■ 0-100 ■ ■ Offsets: ■ ■ Offsets: ■ ■
■ ■ ■ ■ 0-100 ■ ■ 0-85 ■ ■
■ ■ ■■■■■■■■■■■■ ■ ■ ■ ■ ■ ■
■ ■ ■Producer ■ ■ ■ ■ ■ ■ ■
■ ■ ■writes ■ ■ ■ ■ ■ ■ ■
■ ■ ■here ■ ■ ■ ■ ■ ■ ■
■ ■ ■■■■■■■■■■■■ ■ ■ ■ ■ ■ ■
■ ■ ↓ ■ ■ ↑ ■ ■ ↑ ■ ■
■ ■ [Write] ■■■■■■ [Replicate] ■ ■ [Replicate] ■ ■
■ ■ ■ ■ (fast) ■ ■ (lagging) ■ ■
■ ■■■■■■■■■■■■■■■■ ■■■■■■■■■■■■■■■■ ■■■■■■■■■■■■■■■■ ■
■ ■
■ ISR (In-Sync Replicas): {Broker 1, Broker 2} ■
■ Replicas: {Broker 1, Broker 2, Broker 3} ■
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■

Replication Factor
Definition: The number of copies of each partition maintained across the cluster.

RF Use Case Fault Tolerance Notes

1 Development, non-critical None Single point of failure

2 Low-risk production 1 broker failure Not recommended (split-brain risk)

3 Standard production 2 broker failures Industry standard

4+ Critical data 3+ broker failures Higher storage/network cost

Calculation:
Total storage = Data size × Replication factor
Example: 1 TB data, RF=3 → 3 TB total storage needed
Leader & Followers
Leader:

• One per partition: Each partition has exactly one leader at a time
• Handles all I/O: All reads and writes go through the leader
• Maintains offset: Tracks the high watermark (HW) and log end offset (LEO)
• Coordinates followers: Manages follower replication

Followers:

• Passive replicas: Don't serve client requests (except with Kafka 2.4+ read-from-follower)
• Continuously fetch: Pull data from the leader
• Maintain copies: Keep up-to-date replicas of leader's log
• Ready to lead: Can become leader if current leader fails

ISR (In-Sync Replicas)

Definition: The set of replicas that are fully caught up with the leader. This includes the leader itself.

Criteria for ISR Membership:

• Alive: Broker is reachable and sending heartbeats

• Caught up: Replica has fetched all messages up to the leader's high watermark within
[Link] (default: 10 seconds)

ISR vs. All Replicas:

Replicas (AR): {Broker 1, Broker 2, Broker 3} ← All replicas

ISR: {Broker 1, Broker 2} ← Only in-sync replicas
Out-of-sync: {Broker 3} ← Lagging replica

Producer Acknowledgment Modes (JavaScript/NestJS):

// acks=0: No waiting (fastest, no durability)

await [Link]({
topic: 'orders',
messages: [{ key: 'order-1', value: orderData }],
acks: 0
});
// Leader doesn't wait, follower doesn't replicate yet
// Use case: Logging, metrics (data loss acceptable)

// acks=1: Leader acknowledgment (balanced)

await [Link]({
topic: 'orders',
messages: [{ key: 'order-1', value: orderData }],
acks: 1
 });
// Leader writes to log, returns immediately
// Use case: Most applications (low latency, good durability)

// acks=-1 (or 'all'): ISR acknowledgment (strongest durability)

await [Link]({
topic: 'payments',
messages: [{ key: 'payment-1', value: paymentData }],
acks: -1
});
// Leader waits for all ISR members to replicate
// Use case: Financial transactions, critical data

Acknowledgment Latency Durability

acks=0 ~1ms ★■■■■

acks=1 ~5ms ★★★■■

acks=-1 ~10ms ★★★★★

Why Replication Matters

1. Data Durability

Scenario: Broker crashes

Without Replication (RF=1):

Broker 1 crashes → Partition data lost forever ■

With Replication (RF=3):

Broker 1 crashes → Partition still available on Brokers 2, 3 ✓
Data intact, no downtime

2. High Availability

SLA Calculation:
RF=1: 99% availability (broker uptime)
RF=3: 99.99% availability (requires 2/3 brokers to fail simultaneously)

Downtime per year:

99%: 3.65 days
99.99%: 52 minutes
2. JavaScript/NestJS Implementation Examples
2.1 Producer Examples
Basic Producer (KafkaJS):

import { Kafka } from 'kafkajs';

// Initialize Kafka client

const kafka = new Kafka({
clientId: 'my-app',
brokers: ['localhost:9092', 'localhost:9093', 'localhost:9094']
});

// Create producer
const producer = [Link]();

// Connect and send messages

async function sendMessage() {
await [Link]();

try {
const result = await [Link]({
topic: 'user-events',
messages: [
{
key: 'user-123',
value: [Link]({
userId: '123',
action: 'login',
timestamp: new Date().toISOString()
}),
headers: {
'trace-id': 'abc-123-def-456',
'source': 'web-app'
}
}
]
});

[Link]('Message sent:', result);

} catch (error) {
[Link]('Error sending message:', error);
}
}

// Graceful shutdown
[Link]('SIGTERM', async () => {
await [Link]();
});

Batch Producer (High Throughput):

 const producer = [Link]({
// Performance configurations
allowAutoTopicCreation: false,
transactionTimeout: 30000,

// Batching for throughput

compression: [Link],

// Retry configuration
retry: {
initialRetryTime: 100,
retries: 8
}
});

async function sendBatch(events) {

const messages = [Link](event => ({
key: [Link],
value: [Link](event),
headers: {
'event-type': [Link],
'timestamp': [Link]().toString()
}
}));

await [Link]({
topic: 'user-events',
messages: messages,
acks: -1, // Wait for all ISR replicas
timeout: 30000
});
}

Transactional Producer (Exactly-Once Semantics):

const producer = [Link]({
transactionalId: 'my-transactional-producer',
maxInFlightRequests: 1,
idempotent: true
});

async function sendTransactional(orders) {

await [Link]();
const transaction = await [Link]();

try {
// Send multiple messages in a transaction
for (const order of orders) {
await [Link]({
topic: 'orders',
messages: [{
key: [Link],
value: [Link](order)
}]
});
 await [Link]({
topic: 'inventory',
messages: [{
key: [Link],
value: [Link]({
action: 'reserve',
quantity: [Link]
})
}]
});
}

// Commit transaction
await [Link]();
[Link]('Transaction committed');
} catch (error) {
// Abort on error
await [Link]();
[Link]('Transaction aborted:', error);
}
}
2.2 Consumer Examples
Basic Consumer:

const consumer = [Link]({

groupId: 'my-consumer-group',
sessionTimeout: 30000,
heartbeatInterval: 3000
});

async function consumeMessages() {

await [Link]();

// Subscribe to topic
await [Link]({
topic: 'user-events',
fromBeginning: false // Start from latest
});

// Process messages
await [Link]({
eachMessage: async ({ topic, partition, message }) => {
const key = [Link]?.toString();
const value = [Link]([Link]());
const headers = {};

// Extract headers
for (const [headerKey, headerValue] of [Link]([Link])) {
headers[headerKey] = [Link]();
}

[Link]({
topic,
partition,
offset: [Link],
key,
value,
headers,
timestamp: [Link]
});

// Process message
await processEvent(value);
}
});
}

async function processEvent(event) {

// Your business logic here
[Link]('Processing event:', event);
}

Consumer with Manual Offset Commit:

 const consumer = [Link]({
groupId: 'manual-commit-group',
// Disable auto-commit
autoCommit: false
});

await [Link]({
eachMessage: async ({ topic, partition, message }) => {
try {
// Process message
await processMessage(message);

// Manually commit offset after successful processing

await [Link]([
{
topic,
partition,
offset: (parseInt([Link]) + 1).toString()
}
]);

[Link](`Committed offset ${[Link]}`);

} catch (error) {
[Link]('Processing failed:', error);
// Don't commit on error - message will be reprocessed
}
}
});

Batch Consumer (High Throughput):

const consumer = [Link]({
groupId: 'batch-consumer-group',
maxWaitTimeInMs: 5000,
minBytes: 1024, // Wait for at least 1KB
maxBytes: 10485760 // Max 10MB per fetch
});

await [Link]({
eachBatch: async ({
batch,
resolveOffset,
heartbeat,
isRunning
}) => {
const messages = [Link];
[Link](`Processing batch of ${[Link]} messages`);

for (const message of messages) {

if (!isRunning()) break;

try {
await processMessage(message);

// Resolve offset for each processed message

 resolveOffset([Link]);

// Send heartbeat every few messages

await heartbeat();
} catch (error) {
[Link]('Error processing message:', error);
break; // Stop batch on error
}
}
}
});
2.3 NestJS Integration
Kafka Module Setup:

// [Link]
import { Module } from '@nestjs/common';
import { ClientsModule, Transport } from '@nestjs/microservices';
import { KafkaService } from './[Link]';

@Module({
imports: [
[Link]([
{
name: 'KAFKA_SERVICE',
transport: [Link],
options: {
client: {
clientId: 'nestjs-app',
brokers: ['localhost:9092']
},
consumer: {
groupId: 'nestjs-consumer-group'
}
}
}
])
],
providers: [KafkaService],
exports: [KafkaService]
})
export class KafkaModule {}

Kafka Service:
// [Link]
import { Injectable, Inject, OnModuleInit } from '@nestjs/common';
import { ClientKafka } from '@nestjs/microservices';

@Injectable()
export class KafkaService implements OnModuleInit {
constructor(
@Inject('KAFKA_SERVICE')
private readonly kafkaClient: ClientKafka
) {}

async onModuleInit() {
// Subscribe to topics for response
[Link]('[Link]');
await [Link]();
}

async sendMessage(topic: string, message: any) {

return [Link](topic, {
key: [Link],
 value: [Link](message),
headers: {
'timestamp': [Link]().toString(),
'source': 'nestjs-app'
}
});
}

async sendMessageWithKey(
topic: string,
key: string,
message: any
) {
return [Link](topic, {
key,
value: [Link](message)
});
}
}

Controller with Event Handlers:

// [Link]
import { Controller } from '@nestjs/common';
import {
MessagePattern,
Payload,
Ctx,
KafkaContext
} from '@nestjs/microservices';

@Controller()
export class EventsController {

@MessagePattern('[Link]')
async handleUserEvent(
@Payload() message: any,
@Ctx() context: KafkaContext
) {
const originalMessage = [Link]();
const { headers, key, value, offset, timestamp } = originalMessage;

[Link]('Received message:', {
topic: [Link](),
partition: [Link](),
offset,
key: [Link](),
value: [Link](),
timestamp
});

// Process the event

await [Link]([Link]([Link]()));

// Manual commit (if autoCommit is false)

 const consumer = [Link]();
await [Link]([
{
topic: [Link](),
partition: [Link](),
offset: (parseInt(offset) + 1).toString()
}
]);
}

@MessagePattern('[Link]')
async handleOrderCreated(@Payload() order: any) {
[Link]('New order:', order);
// Business logic here
}

private async processUserEvent(event: any) {

// Your business logic
[Link]('Processing:', event);
}
}
Production-Ready Service Example:

// [Link]
import { Injectable, Logger } from '@nestjs/common';
import { KafkaService } from './kafka/[Link]';

interface Order {
orderId: string;
userId: string;
items: OrderItem[];
total: number;
}

interface OrderItem {
productId: string;
quantity: number;
price: number;
}

@Injectable()
export class OrderService {
private readonly logger = new Logger([Link]);

constructor(private readonly kafkaService: KafkaService) {}

async createOrder(order: Order): Promise<void> {

try {
// Validate order
[Link](order);

// Send to Kafka with proper key for ordering

await [Link](
'[Link]',
[Link], // Use orderId as key for partition assignment
{
...order,
createdAt: new Date().toISOString(),
status: 'pending'
}
);

[Link](`Order ${[Link]} sent to Kafka`);

// Send inventory reservation event

for (const item of [Link]) {
await [Link](
'[Link]',
[Link], // Use productId as key
{
orderId: [Link],
productId: [Link],
quantity: [Link]
}
);
}
 } catch (error) {
[Link](`Failed to create order: ${[Link]}`);
throw error;
}
}

private validateOrder(order: Order): void {

if (![Link] || ![Link]) {
throw new Error('Invalid order: missing required fields');
}
if (![Link] || [Link] === 0) {
throw new Error('Invalid order: no items');
}
}
}

Error Handling & Retry Logic:

// [Link]
import { Injectable, Logger } from '@nestjs/common';
import { Kafka, Consumer, EachMessagePayload } from 'kafkajs';

@Injectable()
export class KafkaConsumerService {
private readonly logger = new Logger([Link]);
private consumer: Consumer;
private readonly maxRetries = 3;

async consume() {
await [Link]({
eachMessage: async (payload: EachMessagePayload) => {
const { topic, partition, message } = payload;
let retries = 0;

while (retries < [Link]) {

try {
await [Link](message);
break; // Success, exit retry loop
} catch (error) {
retries++;
[Link](
`Error processing message (attempt ${retries}/${[Link]}):`,
error
);

if (retries >= [Link]) {

// Send to dead letter queue
await [Link](topic, message, error);
} else {
// Exponential backoff
await [Link]([Link](2, retries) * 1000);
}
}
}
}
 });
}

private async processMessage(message: any) {

// Your processing logic
const value = [Link]([Link]());
[Link]('Processing message:', value);
}

private async sendToDeadLetterQueue(

originalTopic: string,
message: any,
error: Error
) {
const dlqTopic = `${originalTopic}.dlq`;

await [Link]({
topic: dlqTopic,
messages: [{
key: [Link],
value: [Link],
headers: {
...[Link],
'original-topic': originalTopic,
'error-message': [Link],
'failed-at': new Date().toISOString()
}
}]
});

[Link](`Message sent to DLQ: ${dlqTopic}`);

}

private sleep(ms: number): Promise<void> {

return new Promise(resolve => setTimeout(resolve, ms));
}
}
3. Complete Message Lifecycle
This section illustrates the complete journey of a message through the Kafka ecosystem, from
producer to consumer, including all intermediate steps.

■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■ COMPLETE MESSAGE FLOW ■
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■

STEP 1: PRODUCER SENDS MESSAGE

■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■■■■■■■■■■■■■■■■
■ Producer ■
■ (NestJS) ■ Message:
■ ■ - Key: "user-123"
■ key: "u123" ■ - Value: {"action": "purchase"}
■ value: {...}■ - Headers: {trace-id: "abc"}
■■■■■■■■■■■■■■■■ - Timestamp: 1706543210000
■
■ 1. Send via KafkaJS
↓
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■ Partition Selection ■
■ hash("user-123") % 3 = Partition 1■
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■
↓
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■ KAFKA CLUSTER ■
■ ■
■ ■■■■■■■■■■■■■■■■ ■■■■■■■■■■■■■■■■ ■■■■■■■■■■■■■■■■ ■
■ ■ Broker 1 ■ ■ Broker 2 ■ ■ Broker 3 ■ ■
■ ■ ■ ■ ■ ■ ■ ■
■ ■ Partition 0 ■ ■ Partition 1 ■ ■ Partition 2 ■ ■
■ ■ [Follower] ■ ■ [LEADER] ■ ■ [Follower] ■ ■
■ ■■■■■■■■■■■■■■■■ ■■■■■■■■■■■■■■■■ ■■■■■■■■■■■■■■■■ ■
■ ■ 2. Leader ■
■ ■ appends ■
■ ■ offset: 42 ■
■ ↓ ■
■ ■■■■■■■■■■■■■■■■■■■■ ■
■ ■ Partition 1 Log ■ ■
■ ■ ... ■ ■
■ ■ 40: {...} ■ ■
■ ■ 41: {...} ■ ■
■ ■ 42: {NEW MSG} ■ ← Our message ■
■ ■■■■■■■■■■■■■■■■■■■■ ■
■ ■ ■ ■
■ ■ 3. Replicate (if acks=-1) ■
■ ■■■■■▼■■■■■ ■▼■■■■■■■■■■ ■
■ ■ Broker 1■ ■ Broker 3 ■ ■
■ ■ Replica ■ ■ Replica ■ ■
■ ■ (ISR) ■ ■ (ISR) ■ ■
■ ■■■■■■■■■■■ ■■■■■■■■■■■■ ■
■ ■
■ 4. ACK to producer (when ISR replicates, if acks=-1) ■
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■

STEP 2: CONSUMER READS MESSAGE

■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■■■■■■■■■■■■■■■■
■ Consumer ■
■ (NestJS) ■
■ Group: "cg1" ■
■■■■■■■■■■■■■■■■
■ 5. Poll via KafkaJS
↓
■■■■■■■■■■■■■■■■
■ Broker 2 ■
■ (Leader) ■
■ ■
■ Returns: ■
■ - Offset: 42 ■
■ - Key: "..." ■
■ - Value: ... ■
■ - Headers:.. ■
■ - Timestamp ■
■■■■■■■■■■■■■■■■
■ 6. Process in NestJS controller
↓
■■■■■■■■■■■■■■■■
■ Consumer ■
■ Processes ■
■ Message ■
■ @MessagePattern■
■■■■■■■■■■■■■■■■
■ 7. Commit offset
↓ offset: 43
■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■ __consumer_offsets topic ■
■ (group:cg1, partition:1, ■
■ offset:43) ■
■■■■■■■■■■■■■■■■■■■■■■■■■■■■
4. Quick Reference Summary
Core Concepts Overview
Concept Key Points Critical For

Topic Logical channel for messages Message organization

Partition Physical storage unit, ordered log Parallelism, ordering

Broker Server storing data Storage, request handling

Cluster Group of brokers HA, scalability

Key Message routing identifier Partitioning, ordering

Value Message payload Actual data

Headers Metadata key-value pairs Tracing, routing

Timestamp Message time Retention, windowing

Offset Sequential message ID Consumer position

Replication Data copies across brokers Durability, HA

Leader Handles all I/O for partition Read/write operations

Follower Replicates leader data Fault tolerance

ISR Replicas caught up with leader Durability guarantees

JavaScript/NestJS Best Practices

• Always use keys for related messages to maintain ordering within partitions
• Implement proper error handling with retry logic and dead letter queues
• Use compression (GZIP, Snappy) for high-throughput scenarios
• Monitor consumer lag continuously in production environments
• Set appropriate acks level based on durability requirements (0, 1, -1)
• Use transactions when you need exactly-once semantics across multiple topics
• Implement graceful shutdown to properly disconnect producers and consumers
• Plan partition count based on throughput requirements and consumer parallelism
• Use manual offset commits for critical data processing workflows
• Leverage headers for tracing, routing, and metadata without bloating the message value
Common Kafka Patterns
• Event Sourcing: Store state changes as events in Kafka topics
• CQRS: Separate read/write models using Kafka for event propagation
• Change Data Capture (CDC): Capture database changes using Debezium
• Saga Pattern: Distributed transactions using choreography with Kafka events
• Outbox Pattern: Ensure reliable message publishing from databases
Conclusion
This guide has covered the core 20% of Kafka knowledge that will enable you to handle 80% of daily
Kafka usage in production environments. You've learned about:

• Core Architecture: Topics, partitions, brokers, and clusters

• Message Components: Keys, values, headers, timestamps, and offsets
• Replication: Leaders, followers, ISR, and durability guarantees
• JavaScript/NestJS Implementation: Production-ready code examples
• Best Practices: Patterns and strategies for real-world applications

With this foundation, you're well-equipped to build scalable, fault-tolerant event-driven applications
using Apache Kafka. Remember to focus on understanding the fundamentals deeply, as they form
the basis for all advanced Kafka usage patterns.

Next Steps:
• Experiment with the code examples in your development environment
• Study Kafka Streams and Connect for stream processing and integration
• Explore monitoring tools like Kafka Manager, Burrow, or Confluent Control Center
• Practice troubleshooting common issues (consumer lag, rebalancing, etc.)
• Review Kafka's official documentation for advanced topics

Apache Kafka Complete Guide

JavaScript/NestJS Edition
© 2024