Kafka Interview Questions and Answers

Apache Kafka interviews test whether you understand a distributed commit log—not a classic queue that deletes messages after one consumer reads them.

Java developer loops add client integration depth on top of broker concepts:

Interviewers often ask you to sketch Spring Boot properties for a producer and consumer in the same service—see Spring Boot interviews for broader microservice context.

Live rounds often stress consumer groups and partitions together with delivery semantics—at-least-once, at-most-once, and exactly-once trade-offs.

Run a local KRaft broker or Docker Compose stack and produce/consume with kafka-console-producer / kafka-console-consumer at least once.

Kafka is a distributed event streaming platform built around an append-only partitioned log. Producers append records; consumers pull and track position via offsets.

Kafka fits event sourcing, audit trails, stream processing, and decoupled microservices where many teams read the same history.

A strong answer is:

Kafka is a durable, replayable log with pub/sub via consumer groups—not a delete-on-read queue—so multiple services can consume the same topic at different speeds and rewind when needed.

Data flow: producer → partition leader → followers replicate → consumers fetch.

A strong answer is:

Brokers host partition leaders and replicas; producers write to leaders; consumer groups divide partitions among members; the controller handles metadata and leader election.

A topic is a logical category of records. Each topic is split into partitions—physical logs on disk.

Partition count is chosen at topic creation—increasing partitions is possible but does not reorder existing keys' history.

Partition assignment by key (simulation):

def partition_for_key(key: str, num_partitions: int) -> int:
    # Kafka uses murmur2; this models stable "same key → same partition"
    return hash(key) % num_partitions

keys = ["user-42", "user-42", "user-99", "user-42"]
parts = [partition_for_key(k, 6) for k in keys]
print(parts[0] == parts[1] == parts[3], parts)

When you click Run, the first value should be True—the same key maps to the same partition index, which is how per-key ordering is preserved.

A strong answer is:

A topic is the stream name; partitions are the ordered shards that enable scale—I choose partition count and keys so ordering and throughput match product rules.

Ordering: Kafka guarantees order per partition. If you need all events for order-123 ordered, use key = order-123 so they land in one partition.

Delivery is not one setting—it is a stack of choices:

There is no global order across partitions—design for partition-local order or use a single partition (limits throughput).

A strong answer is:

Ordering is per partition via key routing; end-to-end delivery semantics come from producer acks, replication, and consumer offset timing together.

KRaft (Kafka Raft) stores cluster metadata in Kafka itself using a Raft quorum—ZooKeeper is no longer required for new Kafka versions (3.3+ production-ready path; 4.x removes ZK).

Interviewers in 2026 expect you to know new clusters use KRaft and migration projects move off ZK.

A strong answer is:

KRaft embeds metadata in Kafka with Raft consensus, simplifying operations— I treat ZooKeeper as legacy and KRaft as the default for greenfield clusters.

Simplified produce path:

1. Serializer turns key/value into bytes
2. Partitioner picks partition (key hash or sticky batching)
3. Producer batches records for throughput
4. Request sent to partition leader broker
5. Leader appends to log; followers replicate
6. Broker responds based on acks setting
7. On failure, producer retries (may duplicate without idempotence)

Batching (linger.ms, batch.size) trades latency for throughput.

A strong answer is:

The producer serializes, partitions, batches, and sends to the leader; replication and ack level determine when the produce call succeeds and whether retries can create duplicates.

Pair acks=all with min.insync.replicas ≥ 2 on the broker and topic replication factor ≥ 3 for meaningful safety—otherwise all can still ack with one replica.

Use acks=0 only for metrics or logs where loss is acceptable.

A strong answer is:

acks=0 is fire-and-forget, acks=1 waits for the leader, acks=all waits for the full ISR—I use all for critical events with proper replication and min.insync.replicas.

The key determines partition (when non-null):

Use cases:

• orderId as key — all lifecycle events for one order stay ordered
• userId — per-user ordering
• Null key — maximum spread when order does not matter

Hot keys can create partition skew—one partition overloaded while others idle.

A strong answer is:

Keys route related events to the same partition for ordering; I avoid hot keys and null keys when I need predictable locality.

Enable with enable.idempotence=true (Kafka producer). Broker assigns Producer ID (PID) and tracks sequence numbers per partition—duplicate retries are deduplicated within that producer session.

Idempotence requires acks=all, retries > 0, and appropriate max.in.flight.requests.per.connection (≤ 5 with idempotence enabled).

A strong answer is:

Idempotent producers dedupe retry batches per partition via sequence numbers—necessary but not sufficient alone for end-to-end exactly-once.

Misconfigured max.in.flight.requests.per.connection with retries and no idempotence can reorder batches—classic interview trap.

A strong answer is:

I tune acks, idempotence, and in-flight requests together, and I batch with linger/batch.size only after measuring latency impact.

A consumer group shares one group.id. Each partition is assigned to exactly one consumer in the group at a time.

Different groups reading the same topic are independent—each maintains its own offsets (fan-out).

Consumer group assignment simulation:

def assign_partitions(partitions, consumers):
    assignments = {c: [] for c in consumers}
    for i, p in enumerate(partitions):
        assignments[consumers[i % len(consumers)]].append(p)
    return assignments

parts = [f"P{i}" for i in range(6)]
print(assign_partitions(parts, ["C0", "C1", "C2"]))

When you click Run, you should see each consumer name mapped to two partition labels—round-robin style assignment similar in spirit to range/round-robin assignors.

A strong answer is:

A consumer group divides partitions among members so each partition is processed once per group; extra consumers stay idle unless I add partitions.

An offset is the consumer's position in a partition log. Committed offsets are stored in the internal topic __consumer_offsets.

At-least-once pattern: process record, then commit offset.

At-most-once pattern: commit offset, then process (may lose on crash).

A strong answer is:

Offsets are the consumer's bookmark—I commit after successful processing for at-least-once, and I disable careless auto-commit on critical pipelines.

Rebalance redistributes partitions when group membership changes:

• Consumer joins or leaves
• Consumer exceeds session.timeout.ms / max.poll.interval.ms
• Topic partition count changes
• Cooperative sticky assignor revokes subset incrementally

Mitigations:

• Right-size session.timeout.ms, heartbeat.interval.ms, max.poll.interval.ms
• Avoid long processing in poll loop—pause and resume or decouple processing
• Static membership (group.instance.id) for rolling restarts

A strong answer is:

Rebalances happen on membership or timeout changes—I use cooperative rebalancing, tune poll intervals, and keep processing off the poll thread for long work.

Lag = difference between log end offset and consumer committed/current offset—how far behind a consumer is.

Tools: kafka-consumer-groups.sh --describe, Burrow, Datadog, Prometheus exporters.

Fix: scale consumers (up to partition count), optimize handler, increase partitions with key strategy review, fix poison messages.

A strong answer is:

Lag is how far behind consumption is—I find whether the bottleneck is processing time, partition skew, or rebalance churn, then scale or fix handlers with metrics, not guesses.

If your listener processes one message for five minutes without polling, the consumer is kicked out—even if heartbeats still run (depending on version/config).

Pattern: poll often, hand off to worker pool, use pause/resume for backpressure.

A strong answer is:

session.timeout detects dead members; max.poll.interval caps time between polls—I never block the poll loop on long synchronous work.

Consumers default to read_uncommitted—see all messages including those from aborted transactions.

read_committed hides aborted transactional batches—required for exactly-once consume-transform-produce loops so consumers do not read rolled-back data.

Pair with transactional producers and disabled auto-commit.

A strong answer is:

read_committed lets consumers see only committed transactional records—part of exactly-once pipelines with transactional producers.

Each partition has replication factor N—one leader and N−1 followers on different brokers.

Producer acks=all waits for ISR acks—not merely any replica.

A strong answer is:

Replication across brokers plus ISR tracking and safe leader election keeps partitions available after node loss—I pair that with acks=all and min.insync.replicas.

ISR = replicas whose lag is below replica.lag.time.max.ms (or byte lag thresholds in older configs).

Only ISR members can become leader if unclean.leader.election.enable=false.

If ISR shrinks to one replica:

• Cluster still serves traffic
• No redundancy until followers catch up
• Risk if that single broker fails

Monitor ISR shrink events—they predict durability risk.

A strong answer is:

ISR is the set of replicas safe to promote— I alert when ISR size drops and I never enable unclean election on critical topics.

Broker/topic setting min.insync.replicas defines how many replicas must ack a write for acks=all to succeed.

Example: replication.factor=3, min.insync.replicas=2

• Tolerates one broker loss without stopping writes
• Producer with acks=all fails if only one ISR member available—prefer failing writes over silent data loss

Interview trap: acks=all with min.insync.replicas=1 and RF=3 still allows single-replica commits.

A strong answer is:

min.insync.replicas enforces how many replicas must ack before a write counts—I set it to 2 with RF=3 so one broker can die without weakening durability.

Clients refresh metadata and discover new leaders—brief produce/fetch errors during election.

Operations: rack awareness (broker.rack) for cross-AZ placement, monitor under-replicated partitions.

A strong answer is:

Follower loss reduces redundancy; leader loss triggers ISR election; clients retry after metadata refresh—I design RF and rack awareness so single-AZ loss does not take topics offline.

Delivery semantics simulation:

def at_most_once(process, commit, events):
    lost = []
    for e in events:
        commit(e)
        if e == "crash":
            lost.append("unprocessed-after-commit")
            break
        process(e)
    return lost

def at_least_once(process, commit, events):
    dup = []
    processed = set()
    for e in events:
        if e != "crash":
            process(e)
            processed.add(e)
        else:
            for p in processed:
                process(p)
                dup.append(p)
            break
        commit(e)
    return dup

events = ["A", "B", "crash"]
print("at-most-once lost:", at_most_once(lambda x: None, lambda x: None, events))
print("at-least-once dups:", at_least_once(lambda x: None, lambda x: None, events))

When you click Run, you should see at-most-once report a lost-after-commit case and at-least-once list duplicates after a simulated crash—illustrating why idempotent handlers matter.

A strong answer is:

At-most-once commits early; at-least-once commits after work and may retry duplicates; exactly-once needs broker and application cooperation—I default to at-least-once with idempotent consumers unless EOS is justified.

Broker-side pieces:

1. Idempotent producer — dedupe per-partition retries
2. Transactions — initTransactions, beginTransaction, commitTransaction
3. sendOffsetsToTransaction — atomic offset commit with output records
4. Consumer isolation.level=read_committed

Application-side: processing must be deterministic; external sinks need idempotent writes or transactional stores—EOS in Kafka does not magically dedupe your database.

Kafka Streams offers processing.guarantee=exactly_once_v2 packaging the pattern.

A strong answer is:

Exactly-once combines idempotent producers, transactional writes, and committed offset reads—I still make downstream sinks idempotent because EOS stops at Kafka's boundary.

Steps:

1. Producer initTransactions()
2. Consumer polls records
3. beginTransaction()
4. Process and produce output records
5. sendOffsetsToTransaction with consumed offsets
6. commitTransaction() — all visible or none

On failure: abortTransaction() — consumers with read_committed never see partial output.

Requires unique transactional.id per producer instance (fence zombies after failover).

A strong answer is:

I wrap output records and input offset commits in one transaction so a crash cannot double-publish or skip commits visible to downstream readers.

Most teams run at-least-once (simpler than full transactions). Retries and rebalance redelivery mean duplicate delivery is normal.

Consumer strategies:

"Exactly-once" in interviews often means effective exactly-once—at-least-once transport + idempotent processing.

A strong answer is:

At-least-once will redeliver—I design handlers to dedupe on business keys or store processed IDs so duplicates are harmless.

Schema Registry (often Confluent) stores Avro/JSON/Protobuf schemas with versioning.

Producers send schema ID; consumers fetch schema from registry—contract between teams.

Interview talking point: breaking schema change blocked by compatibility mode—coordinate with consumers before deploy.

A strong answer is:

Schema Registry versions contracts between producers and consumers—I use backward-compatible Avro evolution and test compatibility in CI.

Kafka Connect is a framework for source and sink connectors:

Runs as distributed workers with offset tracking in Kafka—fits ETL without custom consumer boilerplate.

Pair with SQL interviews when discussing CDC and warehouse loads.

A strong answer is:

Kafka Connect moves data in and out with connectors—I use CDC sources and managed sinks instead of one-off consumers when the pattern is standard.

Kafka Streams is a Java library for stream processing on top of Kafka:

• Stateful operations (aggregations, joins) with changelog topics
• Exactly-once v2 processing guarantee option
• No separate cluster like Flink—runs as your app

Use when team is JVM-heavy and logic fits Kafka-centric topologies; use Flink/Spark when you need complex event-time windowing at massive scale.

A strong answer is:

Kafka Streams embeds processing in Java apps with state stores and EOS options—I choose it for JVM microservices, Flink when operability needs a dedicated cluster.

Compaction suits changelog topics—config snapshots, __consumer_offsets, compacted state topics in Streams.

Not a substitute for infinite raw event history—compacted topics are key-value latest views.

A strong answer is:

Time retention drops old data by age; compaction keeps the latest value per key for changelog-style topics like config or state stores.

Long retention enables replay for new consumer groups or reprocessing—costs disk.

Tiered storage (vendor/cloud features) moves old segments to object storage.

A strong answer is:

Retention balances replay needs and disk cost—I size retention per topic based on compliance and consumer onboarding, not one global default.

Alert on lag SLO breach and URP > 0 sustained—not only broker up/down.

A strong answer is:

I monitor lag, under-replication, and disk—I tie alerts to consumer SLOs and broker health, not just process running.

Load test with expected message size and compression; watch disk I/O and network.

A strong answer is:

I size partitions for peak consumer parallelism and key ordering needs, then load-test brokers—partition count is hard to reduce later.

Poison message — fails processing every retry, blocks partition or causes infinite loop.

Patterns:

Always log key, offset, partition, stack trace; alert on DLT rate.

A strong answer is:

I cap retries, route failures to a dead-letter topic with metadata, and alert—never spin forever on the same offset without visibility.

Never expose plaintext brokers to the public internet; rotate credentials; least-privilege ACLs per producer/consumer principal.

A strong answer is:

TLS plus SASL authentication and topic-level ACLs—I give each service only produce or consume rights on the topics it needs.

Pick RabbitMQ for RPC-style work queues; Kafka for event streams, audit logs, and multiple independent readers.

A strong answer is:

RabbitMQ for task routing and low-latency queues; Kafka for durable event streams, replay, and high-throughput fan-out—I do not force all messaging through one tool.

Practices:

Align with full stack and Spring Boot service boundaries.

A strong answer is:

Events are contracts—I name them as facts, version schemas, include correlation IDs, and design consumers to tolerate redelivery.

CDC streams database row changes to Kafka—often Debezium on Kafka Connect.

Challenge: ordering per key, schema changes, initial snapshot load.

A strong answer is:

CDC publishes database changes as events—I use Debezium for near-real-time sync instead of brittle dual-write patterns between services and DBs.

Add spring-kafka and configure:

spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer
spring.kafka.producer.acks=all
spring.kafka.producer.properties.enable.idempotence=true

@Service
@RequiredArgsConstructor
public class OrderEventPublisher {
    private final KafkaTemplate<String, OrderPlacedEvent> kafkaTemplate;

    public void publish(OrderPlacedEvent event) {
        kafkaTemplate.send("orders", event.orderId(), event);
    }
}

Use orderId as key for partition locality. Configure retries and delivery timeout explicitly for critical topics.

A strong answer is:

Spring Kafka wraps KafkaTemplate with serializers and producer props—I set acks, idempotence, and keys explicitly for durable ordered per-order events.

@Component
@Slf4j
public class OrderListener {

    @KafkaListener(topics = "orders", groupId = "billing-service", concurrency = "3")
    public void onOrder(OrderPlacedEvent event) {
        log.info("billing {}", event.orderId());
    }
}

Disable enable-auto-commit for manual ack mode when you need at-least-once discipline:

@KafkaListener(...)
public void listen(ConsumerRecord<String, OrderPlacedEvent> record, Acknowledgment ack) {
    process(record.value());
    ack.acknowledge();
}

A strong answer is:

I match listener concurrency to partitions, use explicit ack when needed, and wire error handlers to dead-letter topics instead of infinite retry loops.

Assert with KafkaTestUtils.getSingleRecord or awaitility on side effects.

Mirror Spring Boot testing pyramid—many handler unit tests, few full broker tests.

A strong answer is:

I unit-test handlers without Kafka, use EmbeddedKafka or Testcontainers for wiring tests, and keep broker tests focused on serialization and ack behavior.

A strong answer is:

I isolate the group and partition skew, compare release timing, inspect handler latency and rebalances, then fix code or scale consumers with metrics proving the bottleneck.

Checklist:

• Whiteboard producer → topic partitions → consumer group
• Explain acks=all + min.insync.replicas with numbers
• Contrast at-least-once vs exactly-once with idempotent consumer
• Walk rebalance triggers and cooperative protocol
• Partition key strategy for one real domain (orders, payments)
• Spring Kafka producer + @KafkaListener + DLT pattern
• One lag or broker failure STAR story
• Java concurrency refresh for poll/thread issues
• Spring Boot microservice context

A strong answer is:

I rehearse one event pipeline end to end—keys, acks, consumer group, failure handling, and Java config—so answers sound like production war stories, not glossary recitation.