Why this matters
- JSON cost VaultCommerce 3× storage and 40% higher network vs Avro on order topics.
- Protobuf excels for gRPC services already on protobuf; Avro fits Kafka ecosystem better.
- Schema Registry requires Confluent wire format — plan deserializer config accordingly.
- Format choice is hard to change — pick once per topic family.
Format comparison
JSON: human-readable, no schema enforcement, large payloads. Avro: compact, excellent Schema Registry integration, requires schema at read time. Protobuf: compact, strong typing, good for polyglot. VaultCommerce uses Avro for domain events, JSON for debug/admin topics only.
Key points
- Avro — schema-in-payload-header via registry ID; compact binary
- Protobuf — Google format; schema via proto file + optional registry
- JSON Serializer — no schema enforcement; use only for low-volume debug
- Confluent wire format — magic byte + schema ID prefix
- SpecificRecord vs GenericRecord — generated Java classes vs dynamic maps
Wire format and headers
Confluent Avro: magic byte (0) + 4-byte schema ID + Avro bytes. Consumers fetch schema by ID from registry. VaultCommerce adds content-type and traceId in headers without touching payload format.
VaultCommerce rollout checklist
Before promoting changes that touch the VaultCommerce order and payment event backbone, run the staging KRaft cluster (Kafka 3.7+, Schema Registry 7.x) through a 10k events/min soak test. Compare producer request latency p99 and consumer lag per group against the pre-deploy baseline. Avro + Schema Registry for production events. Document the change in the internal topic registry, attach Grafana screenshots to the change ticket, and keep an engineer on lag dashboards for 30 minutes after production rollout — roll back the service release before altering broker-level settings if lag or under-replicated partitions spike.
// VaultCommerce consumer — Avro specific reader
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaAvroDeserializer.class);
props.put("specific.avro.reader", true);
props.put("schema.registry.url", "https://schema-registry:8081");
// JSON only for internal debug topic
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
props.put(JsonDeserializer.VALUE_DEFAULT_TYPE, "com.vaultcommerce.debug.PayloadDump");
Quick recall
Everything you need if you only revisit this box.
- Avro + Schema Registry for production events.
- JSON for debug/low-volume only.
- Wire format includes schema ID, not full schema.
Test yourself
Answer these before moving on — recall is what makes it stick.