Apache Kafka Cheatsheet
Apache Kafka is an open-source, distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
1. Core Architecture Diagram
This diagram shows how producers, brokers, partitions, consumer groups, and consumers interact.
graph TD
P1[Producer 1] -->|Publish| B1[Broker 1: Leader P0]
P2[Producer 2] -->|Publish| B2[Broker 2: Leader P1]
subgraph Kafka Cluster
B1 -->|Sync Replica| B2
B2 -->|Sync Replica| B1
end
subgraph Consumer Group A
C1[Consumer 1] <-->|Subscribes P0| B1
C2[Consumer 2] <-->|Subscribes P1| B2
end
2. Core Concepts & Terminology
| Term | Description |
|---|---|
| Producer | Client application that publishes (writes) events to Kafka topics. |
| Consumer | Client application that subscribes to (reads) and processes events. |
| Broker | A single server in the Kafka cluster that stores and serves messages. |
| Topic | A logical stream of messages (similar to a table in a database). |
| Partition | Physical split of a topic on disk. Partitions allow horizontal scaling, parallelism, and ordering guarantees within the partition. |
| Offset | A unique, sequential integer assigned to each message within a partition. |
| Consumer Group | A group of consumers working together to consume a topic’s partitions in parallel. |
3. Essential CLI Commands (Production-Ready)
Topic Management
# 1. Create a Topic with Partitions & Replication Factor
kafka-topics.sh --bootstrap-server localhost:9092 --create --topic order-events --partitions 6 --replication-factor 3
# 2. List All Topics
kafka-topics.sh --bootstrap-server localhost:9092 --list
# 3. Describe a Topic (Check partition leader, replicas, ISR status)
kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic order-events
# 4. Alter Partitions (Only increase, decreasing partitions is not supported!)
kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic order-events --partitions 12
# 5. Delete a Topic
kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic order-events
Console Producer & Consumer
# 1. Start Console Producer (interactive writing)
kafka-console-producer.sh --bootstrap-server localhost:9092 --topic order-events --property parse.key=true --property key.separator=:
# 2. Start Console Consumer (read from beginning)
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic order-events --from-beginning
# 3. Start Console Consumer with Consumer Group
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic order-events --group group-billing --from-beginning
Consumer Group Management
# 1. List All Consumer Groups
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
# 2. Describe Consumer Group (Check lag, offsets, partitions, host clients)
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group group-billing
# 3. Reset Offsets to Earliest for Group
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group group-billing --reset-offsets --to-earliest --execute --topic order-events
4. Producer & Consumer Configurations
Critical Producer Properties
acks=all: Best durability guarantee. Broker response succeeds only when all in-sync replicas (ISR) acknowledge receipt.retries=2147483647: Re-attempts writing on transient network errors.max.in.flight.requests.per.connection=5: Prevents message re-ordering when retrying (safest value is1for absolute ordering, or5with idempotence enabled).enable.idempotence=true: Enables deduplication on broker. Guarantees “exactly-once” delivery per producer session.
Critical Consumer Properties
enable.auto.commit=false: Disables automatic offset commit. Committing manually ensures no data loss before processing completes.session.timeout.ms=45000: Threshold to detect client heartbeats.heartbeat.interval.ms=3000: Heartbeat frequency. Usually set to $1/3$ ofsession.timeout.ms.max.poll.interval.ms=300000: Maximum duration between consumer polls. If processing takes longer than this, the client is considered dead, prompting a rebalance.
5. Performance Tuning & Best Practices
Low-Latency Tuning
- Compression: Use
lz4orzstdcompression on the producer to reduce network traffic. - Batching: Increase
batch.size(e.g.,32KBor64KB) and adjustlinger.ms(e.g.,5-20ms) to allow larger, more efficient batch sizes instead of single message transfers.
High Throughput & Partitions Scaling
- Set partition counts based on physical parallelism requirements ($Partitions = \max(\text{Producer Throughput}, \text{Consumer Throughput})$).
- Keep broker partition counts below 10,000 per broker to avoid performance degradation.
6. Common Mistakes & Troubleshooting
Client rebalances repeatedly
- Issue: A consumer is taking too long to process a single batch, breaching
max.poll.interval.ms. - Troubleshooting: Increase
max.poll.interval.ms, reducemax.poll.records, or optimize downstream processing (e.g. using worker pools).
Topic metadata updates stall
- Issue: ZooKeeper or Kraft metadata coordination fails, or brokers are unreachable.
- Troubleshooting: Inspect JVM garbage collection logs on the brokers, verify network route availability via
telnet localhost 9092, and checkcontrolled.shutdown.enable=trueconfiguration.
7. Interview Q&A
- Q: How does Kafka achieve extremely high throughput?
- A:
- Sequential I/O: Append-only log design leveraging physical sequential disk writes.
- Zero-Copy Optimization: Uses OS-level Page Cache and
sendfileAPI to transfer bytes directly from kernel page cache to network socket without context switching into application memory. - Batching & Compression: Aggregates payloads into batches.
- A:
- Q: Explain the difference between “At-least-once”, “At-most-once”, and “Exactly-once” processing.
- A:
- At-most-once: Offsets are committed before processing message. Messages might be lost if processing fails.
- At-least-once: Offsets committed after successful processing. No data is lost, but duplicates can occur on retries.
- Exactly-once: Accomplished through transactional producers/consumers and idempotent writing.
- A: