Schema Registry and Compatibility Modes

Avro schemas, BACKWARD compatibility, and evolving OrderPlaced v1 to v2 without breaking consumers.

Advanced12 min read

Read these first

Retention, Compaction, and Tombstones

Why this matters

VaultCommerce evolved OrderPlaced v1→v2 adding optional giftWrap field with BACKWARD compatibility.
Incompatible schema registration fails CI — catches breaking changes before production.
Subject naming: vaultcommerce.orders.placed.v1-value per topic-value convention.
Schema evolution is the contract between 15 producer/consumer teams.

OrderPlaced v1Avro schema

Schema Registry

OrderPlaced v2Add optional field

Schema Registry enforces compatibility. BACKWARD allows new consumers to read old data.

Compatibility modes

BACKWARD (default): new schema can read old data — add optional fields with defaults. FORWARD: old consumers read new data. FULL: both directions. VaultCommerce uses BACKWARD for consumers-up-first deploys; FULL for shared library schemas.

Key points

Schema Registry — Confluent component storing versioned schemas
Subject — {topic}-key or {topic}-value schema namespace
Schema ID — embedded in Confluent wire format (magic byte + id)
BACKWARD compatibility — new consumer reads old producer data
DEFAULT value in Avro — required for safe field additions

Safe evolution patterns

Safe: add optional field with default, add union branch. Unsafe: remove required field, change field type, rename without alias. VaultCommerce runs mvn schema-registry:validate in CI against registered subjects.

VaultCommerce rollout checklist

Before promoting changes that touch the VaultCommerce order and payment event backbone, run the staging KRaft cluster (Kafka 3.7+, Schema Registry 7.x) through a 10k events/min soak test. Compare producer request latency p99 and consumer lag per group against the pre-deploy baseline. BACKWARD: add optional fields with defaults. Document the change in the internal topic registry, attach Grafana screenshots to the change ticket, and keep an engineer on lag dashboards for 30 minutes after production rollout — roll back the service release before altering broker-level settings if lag or under-replicated partitions spike.

Java

// VaultCommerce OrderPlaced Avro v2 — backward compatible addition
@AvroSchema("""
  {"type":"record","name":"OrderPlaced","fields":[
    {"name":"orderId","type":"string"},
    {"name":"customerId","type":"string"},
    {"name":"totalCents","type":"long"},
    {"name":"giftWrap","type":"boolean","default":false}
  ]}
""")
public record OrderPlaced(String orderId, String customerId, long totalCents, boolean giftWrap) {}

// Producer uses KafkaAvroSerializer; old consumers ignore giftWrap via default

Quick recall

Everything you need if you only revisit this box.

BACKWARD: add optional fields with defaults.
Validate schemas in CI before deploy.
Subject per topic-value convention.

Test yourself

Answer these before moving on — recall is what makes it stick.