Apache Kafka

Complete Fundamentals Guide

Technical Interview Preparation

Section Topics Covered

1. Introduction What is Kafka, Use Cases, Key Features

2. Core Architecture Brokers, Topics, Partitions, Clusters

3. Producers APIs, Serialization, Partitioning, Acks

4. Consumers Consumer Groups, Offsets, Rebalancing

5. Replication Leaders, Followers, ISR, Fault Tolerance

6. Storage Logs, Segments, Retention, Compaction

7. Zookeeper/KRaft Coordination, Metadata, Migration

8. Performance Batching, Compression, Zero-Copy

9. Kafka Streams Stream Processing, KTables, Joins

10. Kafka Connect Sources, Sinks, Connectors

11. Security Authentication, Authorization, Encryption

12. Interview Q&A 50+ Common Interview Questions

1. Introduction to Apache Kafka

What is Apache Kafka?

Apache Kafka is a distributed event streaming platform designed for high-throughput, fault-tolerant, and scalable
real-time data pipelines. Originally developed at LinkedIn and later open-sourced through the Apache Software
Foundation, Kafka has become the de facto standard for building event-driven architectures.

Key Characteristics
• Distributed: Runs as a cluster across multiple servers (brokers)
• Scalable: Horizontally scalable by adding more brokers and partitions
• Fault-Tolerant: Data is replicated across multiple brokers
• High Throughput: Can handle millions of messages per second
• Durable: Messages are persisted to disk with configurable retention
• Real-time: Sub-second latency for message delivery

Common Use Cases

• Messaging: Replacement for traditional message brokers (RabbitMQ, ActiveMQ)
• Activity Tracking: User activity, page views, searches, metrics
• Log Aggregation: Collecting logs from multiple services
• Stream Processing: Real-time data transformations and analytics
• Event Sourcing: Storing state changes as a sequence of events
• Commit Log: External commit log for distributed systems
• CDC (Change Data Capture): Capturing database changes in real-time

Kafka vs Traditional Message Queues

Feature Kafka Traditional MQ (e.g., RabbitMQ)

Message Retention Time/size-based retention Deleted after consumption

Consumption Model Pull-based Push-based

Consumer Groups Built-in parallel consumption Limited/Manual

Message Replay Yes (offset-based) No (once consumed, gone)

Ordering Per-partition guarantee Per-queue guarantee

Throughput Millions/sec Thousands/sec

Use Case Event streaming, logs Task queues, RPC

2. Core Architecture
Kafka Cluster Architecture

Producer 1 Kafka Cluster Consumer 1

Broker 1 Broker 2 Broker 3

Producer 2 Consumer 2
P0 P0 P0

P1 P1 P1

Producer 3 Consumer 3
P2 P2 P2

Key Components

Broker
A Kafka broker is a single Kafka server that receives messages from producers, assigns offsets to them, stores
them on disk, and serves them to consumers. Multiple brokers form a Kafka cluster. Each broker is identified by a
unique ID and can handle hundreds of thousands of reads and writes per second.

Topic
A topic is a category or feed name to which messages are published. Topics are multi-subscriber: a topic can
have zero, one, or many consumers. Topics are split into partitions for parallelism and scalability. Topic names
should be descriptive (e.g., 'user-signups', 'order-events', 'payment-transactions').

Partition
Each topic is divided into partitions - ordered, immutable sequences of messages. Each message within a
partition gets an incremental ID called an offset. Partitions enable parallel processing and are the unit of
parallelism in Kafka. Messages with the same key always go to the same partition, ensuring ordering for related
events.
 Topic with Partitions & Offsets

Topic: orders

P0 Offset 0 Offset 1 Offset 2 Offset 3 Offset 4 Offset 5 Offset 6 Offset 7

P1 Offset 0 Offset 1 Offset 2 Offset 3 Offset 4 Offset 5 Offset 6 Offset 7

P2 Offset 0 Offset 1 Offset 2 Offset 3 Offset 4 Offset 5 Offset 6 Offset 7

Offset
An offset is a unique identifier for each message within a partition. Offsets are sequential, starting from 0.
Consumers track their position using offsets - they can commit offsets to remember where they left off. This
enables message replay by simply resetting the consumer's offset to an earlier position.

Cluster
A Kafka cluster consists of multiple brokers working together. The cluster provides fault tolerance through
replication - if one broker fails, others can serve the data. One broker acts as the controller, responsible for
administrative operations like partition assignment and leader election.

Tip: A production Kafka cluster typically has at least 3 brokers for fault tolerance. The replication factor should be at least
3 to survive the loss of 2 brokers.
3. Producers
Producers are client applications that publish (write) events to Kafka topics. They are responsible for choosing
which partition to send a message to within a topic.

Message Production Flow

Producer Serialize Partition Batch Compress Send

Create msg Key/Value Select P Accumulate lz4/snappy To broker

Producer Configuration
Property Description Default

[Link] List of broker addresses for initial connection Required

[Link] Serializer class for message keys Required

[Link] Serializer class for message values Required

acks Acknowledgment level (0, 1, all) 1

retries Number of retries on failure 2147483647

[Link] Batch size in bytes 16384

[Link] Time to wait for batch to fill 0

[Link] Compression algorithm (none, gzip, snappy, lz4, zstd) none

[Link] Max unacknowledged requests 5

[Link] Enable exactly-once semantics true

Acknowledgments (acks)
acks=0: Fire and forget - no acknowledgment. Fastest but can lose messages.
acks=1: Leader acknowledgment only. Message written to leader's log. Risk if leader fails before replication.
acks=all (-1): All in-sync replicas acknowledge. Strongest durability guarantee. Slowest but safest.

Partitioning Strategies
 • Key-based: Messages with the same key go to the same partition (default when key is present). Uses
murmur2 hash.
• Round-robin: Messages distributed evenly across partitions (default when no key).
• Custom Partitioner: Implement your own logic by extending the Partitioner interface.
• Sticky Partitioner: Batches messages to same partition until batch is full (Kafka 2.4+).

Idempotent Producer
Idempotent producers ensure exactly-once delivery to a single partition. Even if a producer retries sending a
message, it will only be written once. Enabled by setting [Link]=true. Each producer gets a unique
PID (Producer ID) and sequence numbers are used to detect duplicates.

Best Practice: For critical data, use acks=all, [Link]=true, and set [Link]=2 on the topic for
maximum durability.
4. Consumers
Consumers read messages from Kafka topics. They subscribe to one or more topics and process the feed of
published messages. Kafka consumers are pull-based, meaning they request data from brokers rather than
having it pushed to them.

Consumer Groups & Partition Assignment

Topic: events Consumer Group A Consumer Group B

C1 C2
Partition 0
C1 C2
Partition 1 C3 C4

Partition 2

Partition 3

Group A: 2 consumers handle 4 partitions (2 each)Group B: 4 consumers (1 partition each)

Each partition is consumed by only ONE consumer within a group

Consumer Groups
A consumer group is a set of consumers that cooperatively consume data from topics. Each partition is
consumed by exactly one consumer within a group, enabling parallel processing. Different consumer groups can
independently consume from the same topic - this is how Kafka supports both pub/sub and queue semantics.

• Each partition is assigned to only one consumer in a group

• If consumers > partitions, some consumers will be idle
• If partitions > consumers, some consumers handle multiple partitions
• Adding/removing consumers triggers a rebalance
• Each consumer group maintains its own offset per partition

Offset Management
Consumers track their position in each partition using offsets. Kafka stores committed offsets in an internal topic
called __consumer_offsets. Consumers can choose when to commit offsets:

• Auto Commit: Offsets committed automatically at regular intervals ([Link]=true,

[Link]=5000). Simple but can cause duplicates or data loss.
• Manual Sync Commit: Call commitSync() - blocks until offsets are committed. Ensures exactly-once
processing but slower.
• Manual Async Commit: Call commitAsync() - non-blocking. Faster but may lose offsets on failure.
Consumer Configuration
Property Description Default

[Link] Consumer group identifier Required

[Link] Where to start if no offset (earliest, latest, none) latest

[Link] Auto commit offsets true

[Link] Auto commit frequency 5000

[Link] Max records returned per poll 500

[Link] Max time between polls before considered dead 300000

[Link] Consumer heartbeat timeout 45000

[Link] Min data to fetch per request 1

[Link] Max wait for [Link] 500

Rebalancing
A rebalance occurs when partition ownership changes among consumers. Triggers include: consumer
joins/leaves group, topic partition count changes, or consumer fails heartbeat. During rebalance, consumption
stops temporarily.

• Eager (Range/RoundRobin): All partitions revoked, then reassigned. Causes stop-the-world pause.
• Cooperative (Incremental): Only affected partitions revoked. Minimizes disruption. Kafka 2.4+.
• Static Membership: Use [Link] to avoid rebalance on restarts.
5. Replication & Fault Tolerance
Kafka achieves fault tolerance through replication. Each partition can have multiple replicas across different
brokers. Replication ensures data durability and availability even when brokers fail.

Replication: Leaders & Followers (RF=3)

Broker 1 Broker 2 Broker 3

P0 - Leader P0 - Follower P0 - Follower

P1 - Follower P1 - Leader P1 - Follower

P2 - Follower P2 - Follower P2 - Leader

Leaders and Followers

Each partition has one leader and zero or more followers. The leader handles all read and write requests.
Followers replicate the leader's log. If the leader fails, one of the followers becomes the new leader.

In-Sync Replicas (ISR)

ISR is the set of replicas that are fully caught up with the leader. A replica is considered in-sync if it has fetched all
messages up to the leader's log end offset within [Link] (default 30 seconds). Only ISR
members can become leaders.

• ISR shrinks when followers fall behind

• ISR grows when followers catch up
• [Link] sets minimum ISR size for writes to succeed with acks=all
• If ISR drops below [Link], producers with acks=all will fail

Replication Factor
The replication factor (RF) determines how many copies of each partition exist. With RF=3, each partition has 3
replicas across 3 different brokers. Higher RF means better fault tolerance but more storage and network
overhead.

• Development: RF=1 (no fault tolerance)

• Production: RF=3 (survives 2 broker failures)
• Critical data: RF=3 with [Link]=2
Unclean Leader Election
When all ISR replicas are unavailable, Kafka can either wait for an ISR replica to come back online or elect an
out-of-sync replica as leader (unclean leader election). Controlled by [Link] (default:
false). Enabling this risks data loss.

Production Tip: Set [Link]=false and [Link]=2 with RF=3. This ensures no data
loss even during broker failures.
6. Storage & Log Management

Log Structure
Kafka stores messages in log files on disk. Each partition is a directory containing multiple segment files.
Messages are appended to the active segment. Old segments are either deleted or compacted based on
retention policy.

• Each partition = directory on broker's disk

• Segments = individual log files (e.g., [Link])
• Index files map offsets to file positions for fast lookups
• Time index enables time-based message retrieval

Log Segments
A log segment is a single file containing messages. New segments are created when the active segment reaches
[Link] (default 1GB) or [Link] (default 7 days). Only the active segment is written to; older
segments are immutable.

Retention Policies

Policy Property Default Description

Time-based [Link] 168 (7 days) Delete segments older than this

Size-based [Link] -1 (unlimited) Delete oldest when total exceeds

Compaction [Link] delete Set to 'compact' for key-based retention

Log Compaction
Log compaction retains only the latest value for each key. Instead of deleting old segments, Kafka removes
superseded records. Useful for changelog topics, caches, and state restoration. A tombstone (null value)
removes a key entirely.

• Guarantees at least the last update for each key is retained

• Useful for database CDC, state stores, configuration topics
• Compaction runs in background, doesn't block producers
• [Link] controls when compaction starts

Zero-Copy Optimization
Kafka uses the sendfile() system call to transfer data directly from disk to network socket without copying through
application memory. This zero-copy optimization dramatically improves throughput, especially for consumers
reading historical data.
7. Zookeeper & KRaft
Zookeeper / KRaft Responsibilities

Zookeeper (Legacy) KRaft (New)

• Broker registration • No external dependency

• Topic config • Built into Kafka

• Leader election • Raft consensus

• ACLs • Faster failover

Zookeeper (Legacy)
Traditionally, Kafka relied on Apache Zookeeper for cluster coordination and metadata management. Zookeeper
handles broker registration, topic configuration, leader election, and ACL storage. However, this dependency
added operational complexity.

• Broker liveness detection and registration

• Controller election among brokers
• Topic and partition metadata storage
• Consumer group membership (old consumer API)
• Access control lists (ACLs)

KRaft Mode (Kafka Raft)

KRaft removes the Zookeeper dependency by implementing Raft consensus within Kafka itself. Metadata is
stored in an internal topic (__cluster_metadata) and replicated across controller nodes. KRaft is production-ready
since Kafka 3.3+ and is the recommended mode for new deployments.

• Simplified operations - no separate Zookeeper cluster

• Faster controller failover (seconds vs minutes)
• Improved scalability - supports millions of partitions
• Single security model for entire cluster
• Reduced operational complexity and resource usage

Controller in KRaft
In KRaft mode, dedicated controller nodes (or combined controller-broker nodes) manage cluster metadata using
the Raft consensus protocol. One controller is the active leader; others are followers ready to take over if needed.
Migration Note: Existing Zookeeper-based clusters can migrate to KRaft. The process involves running in hybrid mode,
migrating metadata, and then removing Zookeeper. Plan carefully!
8. Performance Tuning

Why Kafka is Fast

• Sequential I/O: Appends data to end of log files, leveraging disk sequential read/write speed
• Zero-Copy: sendfile() transfers data directly from disk to network
• Page Cache: Relies on OS page cache for reads, avoiding JVM heap overhead
• Batching: Groups multiple messages into batches for efficient network and disk I/O
• Compression: Reduces network bandwidth and storage at the batch level
• Partition Parallelism: Multiple partitions enable parallel processing

Producer Performance

Setting Impact Recommendation

[Link] Larger batches = better throughput 16KB-64KB typical

[Link] Wait time for batching 5-100ms for throughput

[Link] Reduces network/storage lz4 or zstd recommended

[Link] Total memory for batching 32MB-64MB

acks Durability vs latency tradeoff acks=all for durability

Consumer Performance

Setting Impact Recommendation

[Link] Min data per fetch 1KB-1MB based on latency needs

[Link] Max wait for min bytes 100-500ms

[Link] Records per poll() 500-1000 typical

Partition count Parallelism limit Match consumer count

Broker Performance
• [Link]: I/O threads for disk operations (default 8)
• [Link]: Threads for network requests (default 3)
• [Link]: Threads for replication (increase for many partitions)
• [Link] / [Link]: Network buffer sizes
• Use SSDs for log directories where possible
Compression Comparison

Algorithm Speed Ratio CPU Usage Best For

none Fastest 1:1 None Low-latency, small messages

lz4 Very Fast ~2:1 Low General purpose (recommended)

snappy Fast ~2:1 Low Google-originated, similar to lz4

gzip Slow ~3:1 High Best compression, batch jobs

zstd Fast ~3:1 Medium Best balance (Kafka 2.1+)

9. Kafka Streams
Kafka Streams is a client library for building real-time stream processing applications. It processes data from
Kafka topics and writes results back to Kafka or external systems. Unlike Spark or Flink, Kafka Streams runs as a
regular Java application - no separate cluster needed.

Key Concepts
• KStream: An unbounded stream of records. Each record is an independent event. Like an append-only log.
• KTable: A changelog stream representing the latest value for each key. Like a database table that evolves
over time.
• GlobalKTable: A KTable fully replicated on each instance. Useful for small lookup data.
• Topology: A DAG (directed acyclic graph) of stream processors that defines the processing logic.
• State Store: Local storage for stateful operations (aggregations, joins). Backed by RocksDB.

Stream Operations

Stateless Operations Description

filter() Keep records matching a predicate

map() / mapValues() Transform records

flatMap() / flatMapValues() One-to-many transformation

branch() Split stream into multiple branches

merge() Combine multiple streams

selectKey() Change the key of records

Stateful Operations Description

groupByKey() / groupBy() Group records for aggregation

count() Count records per key

aggregate() Custom aggregation logic

reduce() Combine records with same key

join() Join KStream/KTable with another KStream/KTable

windowedBy() Window-based aggregations (tumbling, hopping, sliding)

Windowing
• Tumbling Window: Fixed-size, non-overlapping windows (e.g., every 5 minutes)
• Hopping Window: Fixed-size, overlapping windows (e.g., 5-min window every 1 minute)
 • Sliding Window: Triggered by events, windows of activity within a time range
• Session Window: Dynamic windows based on activity gaps (inactivity timeout)

Exactly-Once Semantics (EOS)

Kafka Streams supports exactly-once processing via [Link]=exactly_once_v2. This ensures that
even during failures, each record is processed exactly once. Requires idempotent producers, transactional
writes, and proper consumer configuration.
10. Kafka Connect
Kafka Connect is a framework for streaming data between Kafka and external systems. It provides a scalable,
fault-tolerant way to integrate databases, key-value stores, file systems, and other data sources/sinks with Kafka.

Concepts
• Connector: High-level abstraction that manages data copying. Can be a Source (external → Kafka) or Sink
(Kafka → external).
• Task: A connector is broken into tasks for parallel execution. Tasks are distributed across Connect
workers.
• Worker: A process running connectors and tasks. Can run standalone or in distributed mode.
• Converter: Handles serialization between Kafka format and Connect internal format (JSON, Avro,
Protobuf).
• Transform: Single Message Transforms (SMTs) for lightweight per-message modifications.

Popular Connectors

Type Connector Use Case

Source Debezium CDC from MySQL, PostgreSQL, MongoDB

Source JDBC Source Pull data from relational databases

Source FileStream Read from files

Source Kinesis Source AWS Kinesis to Kafka

Sink JDBC Sink Write to relational databases

Sink Elasticsearch Sink Index data for search

Sink S3 Sink Archive to AWS S3

Sink BigQuery Sink Load to Google BigQuery

Sink HDFS Sink Write to Hadoop

Standalone vs Distributed Mode

Aspect Standalone Distributed

Workers Single process Multiple workers in cluster

Fault Tolerance None Tasks redistributed on failure

Scaling Vertical only Horizontal scaling

Config Storage Local file Kafka topics

 Use Case Development/testing Production

Tip: Use Confluent Hub ([Link]/hub) to discover and download pre-built connectors. For CDC, Debezium is the
gold standard for capturing database changes.
11. Security
Kafka provides comprehensive security features including authentication, authorization, and encryption to protect
data in transit and control access to resources.

Authentication
• SSL/TLS: Certificate-based authentication using keystores
• SASL/PLAIN: Username/password (simple, often used with SSL)
• SASL/SCRAM: Salted Challenge Response - more secure than PLAIN
• SASL/GSSAPI (Kerberos): Enterprise authentication integration
• SASL/OAUTHBEARER: OAuth 2.0 token-based authentication

Authorization (ACLs)
Kafka uses Access Control Lists (ACLs) to control access to resources. ACLs specify which principals
(users/groups) can perform which operations on which resources.

Resource Type Operations

Topic Read, Write, Create, Delete, Describe, DescribeConfigs, AlterConfigs

Group Read, Describe, Delete

Cluster Create, ClusterAction, DescribeConfigs, AlterConfigs, IdempotentWrite

TransactionalId Describe, Write

Encryption
• In-Transit: SSL/TLS encrypts data between clients and brokers, and between brokers
• At-Rest: Kafka doesn't natively encrypt data at rest - use disk encryption or encrypted file systems
• End-to-End: Application-level encryption before producing (e.g., encrypt payload before send)

Security Best Practices

• Enable authentication - never run production Kafka without it
• Use SSL/TLS for all inter-broker and client-broker communication
• Implement least-privilege ACLs - grant only necessary permissions
• Use SASL/SCRAM or Kerberos over SASL/PLAIN for stronger auth
• Regularly rotate credentials and certificates
• Separate listener ports for internal and external traffic
• Monitor authentication failures and unauthorized access attempts
12. Interview Questions & Answers

Basic Concepts
Q1: What is Apache Kafka?
Kafka is a distributed event streaming platform for high-throughput, fault-tolerant, real-time data pipelines. It
acts as a distributed commit log where producers publish messages to topics, and consumers subscribe to
process them.

Q2: What are the main components of Kafka?

The main components are: Brokers (servers storing data), Topics (categories of messages), Partitions (units
of parallelism within topics), Producers (publish messages), Consumers (read messages), and
Zookeeper/KRaft (cluster coordination).

Q3: What is a Kafka topic?

A topic is a category or feed name to which messages are published. Topics are split into partitions for
scalability. Topics are multi-subscriber - the same topic can be consumed by multiple consumer groups
independently.

Q4: Explain Kafka partitions.

Partitions are ordered, immutable sequences of messages within a topic. Each message in a partition has a
unique offset. Partitions enable parallel processing and are distributed across brokers. Messages with the
same key go to the same partition.

Q5: What is an offset in Kafka?

An offset is a unique, sequential identifier for each message within a partition, starting from 0. Consumers
track their position using offsets. Offsets enable replay (re-read from earlier offset) and exactly-once
processing.

Q6: What is a consumer group?

A consumer group is a set of consumers that cooperatively consume a topic. Each partition is consumed by
exactly one consumer in the group. This enables parallel processing while ensuring each message is
processed once per group.

Q7: What happens if consumers > partitions?

Some consumers will be idle as each partition can only be assigned to one consumer within a group. To
maximize parallelism, have partitions >= consumers.

Q8: What is the role of Zookeeper in Kafka?

Zookeeper handles cluster coordination: broker registration, controller election, topic/partition metadata, and
ACLs. In newer versions (3.3+), KRaft mode eliminates Zookeeper dependency using Raft consensus within
Kafka itself.
Producer Questions
Q9: Explain the acks configuration in producers.
acks controls durability: acks=0 (fire and forget, fastest), acks=1 (leader acknowledges, balanced), acks=all
(all ISR replicas acknowledge, safest). Use acks=all for critical data with [Link] for maximum
durability.

Q10: How does Kafka determine which partition to send a message to?
If a key is provided, Kafka uses murmur2 hash of the key to select a partition (ensuring same key always goes
to same partition). Without a key, sticky partitioner batches to same partition, then round-robins. Custom
partitioners can override this.

Q11: What is an idempotent producer?

An idempotent producer ensures exactly-once delivery to a partition even with retries. Each producer gets a
PID and sequence numbers detect duplicates. Enable with [Link]=true. It prevents duplicates
from network issues or retries.

Q12: What is producer batching?

Producers accumulate messages into batches before sending. [Link] sets max batch bytes, [Link]
sets max wait time. Batching improves throughput by reducing network requests and enabling better
compression.

Q13: Explain producer compression.

Compression reduces network bandwidth and storage. Set with [Link] (none, gzip, snappy, lz4,
zstd). Compression happens at batch level. lz4 or zstd offer best balance of speed and ratio for most use
cases.

Consumer Questions
Q14: Explain [Link].
Determines where to start when no committed offset exists: 'earliest' reads from beginning, 'latest' reads only
new messages, 'none' throws exception. Use 'earliest' for processing all historical data, 'latest' for real-time
only.

Q15: What is consumer rebalancing?

Rebalancing redistributes partitions among consumers when group membership changes (consumer
joins/leaves/fails). During rebalance, consumption pauses. Use cooperative rebalancing (Kafka 2.4+) to
minimize disruption.

Q16: Difference between commitSync() and commitAsync()?

commitSync() blocks until offset is committed - reliable but slower. commitAsync() is non-blocking and faster
but may lose offsets on failure. Best practice: use async during processing, sync before shutdown.

Q17: How do you handle a slow consumer?

Options include: increase partitions and consumers for parallelism, optimize processing logic, increase
[Link], reduce [Link], use async processing with manual offset commits, or consider
backpressure mechanisms.
Replication & Fault Tolerance
Q18: What is ISR (In-Sync Replicas)?
ISR is the set of replicas fully caught up with the leader. A replica stays in ISR if it fetches within
[Link]. Only ISR members can become leaders. ISR size affects write availability with
acks=all.

Q19: What is the difference between leader and follower replicas?

The leader handles all produce and consume requests for a partition. Followers replicate the leader's data
passively. If leader fails, an ISR follower becomes the new leader. Consumers and producers only interact
with leaders.

Q20: Explain [Link].

Minimum number of replicas that must acknowledge writes for acks=all to succeed. With RF=3 and
[Link]=2, writes succeed if leader + 1 follower acknowledge. Prevents data loss if broker fails
immediately after write.

Q21: What is unclean leader election?

When all ISR replicas are down, [Link]=true allows an out-of-sync replica to become
leader, causing data loss. Default is false (safer). Enable only if availability is more important than durability.

Q22: How many broker failures can Kafka survive?

With replication factor N and [Link] M: can survive N-M broker failures for writes, N-1 failures for
reads. Example: RF=3, [Link]=2 survives 1 failure for writes, 2 for reads.

Storage & Retention

Q23: How does Kafka store messages?
Messages are stored in log segments on disk. Each partition is a directory containing segment files.
Segments are append-only and immutable once rolled. Index files enable fast offset lookups without scanning
entire logs.

Q24: What is log compaction?

Log compaction retains only the latest value for each key, removing older duplicates. Useful for changelog
topics where only current state matters. Set [Link]=compact. Tombstones (null values) delete
keys.

Q25: Difference between [Link] and [Link]?

[Link] is time-based (delete segments older than X hours). [Link] is size-based
(delete oldest when total exceeds X bytes). Both can be set - whichever threshold is reached first triggers
cleanup.

Q26: What is zero-copy in Kafka?

Zero-copy uses sendfile() system call to transfer data directly from disk to network socket, bypassing
application memory. This reduces CPU usage and memory copies, significantly improving throughput for
reads.
Advanced Topics
Q27: Explain exactly-once semantics (EOS) in Kafka.
EOS ensures each message is processed exactly once, even during failures. Achieved via idempotent
producers (no duplicates), transactions (atomic writes to multiple partitions), and proper consumer offset
management. Enable with [Link]=true and transactions.

Q28: What are Kafka transactions?

Transactions enable atomic writes to multiple partitions. Producer begins transaction, sends messages,
commits/aborts atomically. Consumers with [Link]=read_committed only see committed messages.
Used for exactly-once across topics.

Q29: Difference between KStream and KTable in Kafka Streams?

KStream is an unbounded stream of records - each record is an independent event (INSERT semantics).
KTable is a changelog stream - represents latest value per key (UPSERT semantics). Use KStream for
events, KTable for state.

Q30: What is a GlobalKTable?

A GlobalKTable is fully replicated to all Kafka Streams instances, unlike KTable which is partitioned. Used for
small, slowly-changing reference data that needs to be joined with streams without repartitioning.

Q31: Explain Kafka Streams windowing.

Windows group events by time for aggregations. Tumbling windows are fixed, non-overlapping. Hopping
windows are fixed but overlap. Sliding windows are event-triggered. Session windows are dynamic based on
activity gaps.

Q32: What is Kafka Connect?

Kafka Connect is a framework for streaming data between Kafka and external systems. Source connectors
import data into Kafka; sink connectors export data from Kafka. Runs standalone or distributed for scalability.

Q33: What is Debezium?

Debezium is an open-source CDC (Change Data Capture) platform built on Kafka Connect. It captures
row-level changes from databases (MySQL, PostgreSQL, MongoDB, etc.) and streams them to Kafka topics
in real-time.

Q34: What is Schema Registry?

Schema Registry (Confluent) stores and manages Avro/Protobuf/JSON schemas for Kafka messages.
Ensures producers and consumers agree on message format. Supports schema evolution with compatibility
checks.
Performance & Operations
Q35: Why is Kafka so fast?
Sequential I/O (append-only writes), zero-copy for reads, page cache utilization, batching, compression,
partition parallelism, and efficient binary protocol. These optimizations enable millions of messages per
second.

Q36: How do you choose the number of partitions?

Consider: target throughput / per-partition throughput, number of consumers for parallelism, future scaling
needs, and key cardinality. More partitions = more parallelism but also more overhead. Start conservative,
increase as needed.

Q37: How do you monitor Kafka?

Key metrics: under-replicated partitions, ISR shrink rate, request latency, consumer lag, broker
CPU/memory/disk, network throughput. Tools: JMX metrics, Prometheus/Grafana, Confluent Control Center,
Burrow for lag.

Q38: How do you handle consumer lag?

Identify cause (slow processing, insufficient consumers, broker issues). Solutions: add consumers (up to
partition count), optimize processing, increase partitions, use async processing, check for poison messages.

Q39: What is backpressure in Kafka?

Backpressure occurs when consumers can't keep up with producers. Kafka handles this via buffering
(messages accumulate in partitions). Manage by monitoring lag, scaling consumers, or implementing flow
control in producers.

Q40: How do you perform a rolling restart?

Restart brokers one at a time, waiting for each to rejoin and sync before proceeding. Ensure
[Link]=true for clean leadership transfers. Monitor under-replicated partitions during
process.

Scenario-Based Questions
Q41: Design a real-time analytics pipeline with Kafka.
Sources → Kafka Connect (ingest) → Topics → Kafka Streams or ksqlDB (process/aggregate) → Output
topics → Sink connectors to data warehouse/dashboard. Use partitioning by entity for parallelism, windowing
for time-based aggregations.

Q42: How would you implement event sourcing with Kafka?

Events as immutable facts in compacted topics (one topic per aggregate type). Key = entity ID, value = event.
Rebuild state by replaying events from beginning. Use Kafka Streams for materialized views. Enable log
compaction.

Q43: How do you ensure message ordering?

Ordering is guaranteed within a partition. Use the same key for related messages (they'll go to same partition).
For global ordering, use single partition (sacrifices parallelism). With idempotence, set
[Link]=5.
Q44: How do you handle poison messages (messages that always fail)?
Implement dead letter queue (DLQ): on repeated failures, send message to separate error topic. Use try-catch
around processing, track retry count, move to DLQ after threshold. Monitor DLQ for manual intervention.

Q45: Design a multi-datacenter Kafka deployment.

Options: Active-Passive (MirrorMaker 2 replicates to DR site), Active-Active (bidirectional replication with
conflict resolution), Stretched cluster (single cluster across DCs, requires low latency). Consider RPO/RTO
requirements.
Additional Questions
Q46: What is MirrorMaker 2?
MirrorMaker 2 (MM2) is Kafka's built-in tool for cross-cluster replication. It uses Kafka Connect framework,
supports active-active replication, preserves offsets across clusters, and handles topic/consumer offset sync.

Q47: How does Kafka handle back-pressure?

Producers block when [Link] is exhausted ([Link] timeout). Consumers naturally handle
back-pressure by polling at their own pace. Monitor consumer lag and producer blocked time. Scale
consumers or partitions if needed.

Q48: What is rack awareness in Kafka?

Rack awareness ensures replicas are distributed across different racks/availability zones. Configure
[Link] on each broker. Kafka places replicas on different racks for better fault tolerance against rack/AZ
failures.

Q49: Explain quotas in Kafka.

Quotas limit client resource usage. Producer/consumer quotas limit bytes/second per client ID or user.
Request quotas limit CPU time. Prevents single client from overwhelming cluster. Configure via
[Link] or AdminClient.

Q50: What are some Kafka anti-patterns?

Using Kafka as a database (it's a log, not a DB), too few/many partitions, ignoring consumer lag, not setting
proper replication, using synchronous processing for high throughput, neglecting monitoring, unbounded
retention.

Q51: How do you secure Kafka in production?

Enable TLS for encryption, SASL for authentication (SCRAM or Kerberos), ACLs for authorization. Use
separate listeners for internal/external traffic. Encrypt disks at rest. Audit access. Rotate credentials regularly.

Q52: What's new in KRaft mode?

KRaft removes Zookeeper dependency using Raft consensus. Benefits: simpler operations, faster failover,
better scalability (millions of partitions), unified security model. Production-ready since Kafka 3.3.
Quick Reference: Key Configurations
Category Property Recommended Value

Producer acks all (for durability)

Producer [Link] true

Producer [Link] lz4 or zstd

Producer [Link] 16384-65536

Producer [Link] 5-100

Consumer [Link] earliest or latest

Consumer [Link] false (manual preferred)

Consumer [Link] 500

Consumer [Link] read_committed (for EOS)

Broker [Link] 2 (with RF=3)

Broker [Link] false

Broker [Link] 3

Broker [Link] Match workload needs

Topic [Link] Based on use case

Topic [Link] delete or compact

CLI Quick Reference

Command Description

[Link] --create --topic X --partitions 3 --replication-factor 3 Create topic

[Link] --list List all topics

[Link] --describe --topic X Describe topic details

[Link] --topic X --bootstrap-server localhost:9092 Produce messages

[Link] --topic X --from-beginning --bootstrap-server localhost:9092

Consume messages

[Link] --describe --group G Check consumer group lag

[Link] --alter --entity-type topics --entity-name X --add-config

Alter
[Link]=86400000
topic config
Good luck with your interview! Remember: understand the concepts, not just memorize answers. Be
ready to discuss trade-offs and real-world scenarios.