Consumer Groups and Partition Assignment

One consumer per partition per group — how VaultCommerce inventory and payment services scale independently.

Intermediate11 min read

Read these first

Idempotent Producer and Sequence Numbers

Why this matters

VaultCommerce inventory-service and email-service are separate groups on the same topic — both receive every order.
Scaling consumers beyond partition count wastes pods — VaultCommerce HPA caps at partition count.
group.id is the offset namespace — resetting it rewinds consumption.
Core concept for every consumer article and ops playbook.

Topic partitions

Group: inventory-service

Consumer Areads P0

Consumer Breads P1

Consumer Creads P2

Each partition is consumed by exactly one consumer in the group. Add consumers up to partition count.

Partition assignment

Group coordinator broker tracks members and triggers rebalance on join/leave. Range assignor (default) can imbalance; VaultCommerce uses CooperativeStickyAssignor for incremental rebalances. 12 partitions, 4 inventory consumers → 3 partitions each.

Key points

group.id — logical consumer set name; offset commits are per group-topic-partition
Group coordinator — broker managing membership and offset commits
Partition assignment — one consumer per partition within a group
Static membership — group.instance.id reduces rebalance on rolling deploys
Multiple groups — fan-out without duplicate assignment within a group

Independent groups on one topic

Each group maintains its own committed offsets. Analytics lag does not affect inventory lag. VaultCommerce monitors consumer_group tag per team in Grafana.

VaultCommerce rollout checklist

Before promoting changes that touch inventory-service and payment-service consumer groups, run the staging KRaft cluster (Kafka 3.7+, Schema Registry 7.x) through a 10k events/min soak test. Compare producer request latency p99 and consumer lag per group against the pre-deploy baseline. One consumer per partition per group. Document the change in the internal topic registry, attach Grafana screenshots to the change ticket, and keep an engineer on lag dashboards for 30 minutes after production rollout — roll back the service release before altering broker-level settings if lag or under-replicated partitions spike.

Java

// VaultCommerce inventory consumer — dedicated group
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrap);
props.put(ConsumerConfig.GROUP_ID_CONFIG, "inventory-service");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaAvroDeserializer.class);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false); // manual commit after processing

KafkaConsumer<String, OrderPlaced> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("vaultcommerce.orders.placed.v1"));

Quick recall

Everything you need if you only revisit this box.

One consumer per partition per group.
Separate groups = separate offset progress.
Max useful consumers = partition count.

Test yourself

Answer these before moving on — recall is what makes it stick.