Apache Kafka Cluster Sizing: Partitioning Strategies for High Throughput

Derive partitions from measured consumer throughput

Start with the peak production rate you must sustain, not the average. Then measure what one consumer instance actually processes per second under realistic conditions — including deserialization, business logic, and downstream writes, which usually dominate. Dividing peak production by per-consumer throughput gives the minimum partition count for keeping up with a fully scaled consumer group.

Apply a headroom factor for growth and for degraded operation, typically somewhere between 1.5x and 3x depending on how quickly you can repartition. Repartitioning a keyed topic is disruptive because key-to-partition mapping changes, so keyed topics deserve more headroom than unkeyed ones.

Validate against broker-side cost

Every partition costs memory for indexes, file handles, replication threads, and controller metadata. Rebalance duration grows with partition count, and a group with tens of thousands of assigned partitions can take long enough to rebalance that ordinary deployments look like outages. Validate the derived count against per-broker partition budgets and against your acceptable rebalance window.

Replication factor multiplies both storage and inter-broker traffic. A replication factor of three with a minimum in-sync replica setting of two is the common durable default; treat that as a starting point and confirm the resulting network throughput fits the interconnect.

Design keys for even distribution

Uniform partition counts mean nothing if the key distribution is skewed. A key such as tenant identifier concentrates traffic on whichever partitions host the largest tenants, producing hot brokers while the rest idle. Measure per-partition byte rates in production, and where skew is structural, use a composite key or an explicit partitioner that spreads large tenants across several partitions while preserving the ordering guarantee the domain actually needs.

Ordering requirements are usually narrower than teams assume. Ordering per entity is nearly always sufficient; global ordering almost never is, and demanding it forfeits parallelism entirely.

Plan storage and retention together

Retention policy determines disk sizing as much as throughput does. Compute retained bytes as peak ingest multiplied by retention window multiplied by replication factor, then add compaction overhead for log-compacted topics. Tiered storage changes this arithmetic significantly for long-retention topics and should be evaluated before over-provisioning local disk.