Batching, linger.ms, and Compression

Trade a few milliseconds of latency for 10x throughput — batching and compression on the VaultCommerce producer.

Intermediate10 min read

Read these first

acks, Retries, and Durability Tuning

Why this matters

VaultCommerce cut producer CPU network utilization 60% with compression.type=lz4 and linger.ms=5.
Batching is why Kafka achieves millions of msgs/sec — not single-record sends.
batch.size and buffer.memory bound memory per producer instance.
Tuning batching is the first throughput optimization after correctness settings.

ApplicationVaultCommerce OrderService

Serializer

Record batch

Broker leader

Serializer → partitioner → record batch → broker leader. Callback fires with RecordMetadata or error.

linger.ms and batch.size

Producer waits up to linger.ms to fill a batch up to batch.size bytes before sending. VaultCommerce order events (~400 byte Avro) batch ~50 records per partition per 5ms window. Lower linger for latency-sensitive payment capture topic (linger.ms=0).

Key points

batch.size — target batch bytes per partition before flush
linger.ms — max wait to fill a batch
buffer.memory — total memory for unsent batches (default 32MB)
compression.type — lz4, zstd, snappy, gzip, or none
RecordAccumulator — internal queue organizing batches by partition

Compression codecs

lz4 — fast, moderate ratio; VaultCommerce default. zstd — better ratio, more CPU; used on CDC topics with wide JSON payloads. gzip — avoid in hot paths. Compression happens on the producer thread before send.

VaultCommerce rollout checklist

Before promoting changes that touch the VaultCommerce order and payment event backbone, run the staging KRaft cluster (Kafka 3.7+, Schema Registry 7.x) through a 10k events/min soak test. Compare producer request latency p99 and consumer lag per group against the pre-deploy baseline. Batches amortize network overhead. Document the change in the internal topic registry, attach Grafana screenshots to the change ticket, and keep an engineer on lag dashboards for 30 minutes after production rollout — roll back the service release before altering broker-level settings if lag or under-replicated partitions spike.

Java

// VaultCommerce high-throughput analytics producer
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 64 * 1024);      // 64 KB batches
props.put(ProducerConfig.LINGER_MS_CONFIG, 5);               // wait 5ms to coalesce
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 64 * 1024 * 1024L);

// Latency-sensitive override for vaultcommerce.payments.authorized.v1
props.put(ProducerConfig.LINGER_MS_CONFIG, 0);
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 16 * 1024);

Quick recall

Everything you need if you only revisit this box.

Batches amortize network overhead.
linger.ms trades latency for throughput.
lz4 is VaultCommerce's default compression.

Test yourself

Answer these before moving on — recall is what makes it stick.