Event Sourcing on a Kafka Log

Event store patterns on Kafka

Compacted topic keyed by orderId holds latest snapshot OR full event stream on separate topics. VaultCommerce uses non-compacted event stream + Postgres projection as source of truth for queries. Kafka retention = 90 days; archive to S3 for compliance.

Snapshots and replay

Snapshot service consumes until offset N, writes OrderSnapshot to compacted topic. New service instances load snapshot then replay delta. VaultCommerce replays staging topics in Testcontainers for integration tests.

VaultCommerce rollout checklist

Before promoting changes that touch the VaultCommerce order and payment event backbone, run the staging KRaft cluster (Kafka 3.7+, Schema Registry 7.x) through a 10k events/min soak test. Compare producer request latency p99 and consumer lag per group against the pre-deploy baseline. Events are facts; projections are derived state. Document the change in the internal topic registry, attach Grafana screenshots to the change ticket, and keep an engineer on lag dashboards for 30 minutes after production rollout — roll back the service release before altering broker-level settings if lag or under-replicated partitions spike.

// VaultCommerce order projection — rebuild from event stream
public OrderView rebuild(String orderId, long upToOffset) {
    OrderView view = OrderView.empty(orderId);
    try (KafkaConsumer<String, DomainEvent> consumer = createReplayConsumer()) {
        consumer.assign(List.of(new TopicPartition("vaultcommerce.orders.events.v1", 0)));
        consumer.seek(new TopicPartition("vaultcommerce.orders.events.v1", 0), 0);
        while (true) {
            var records = consumer.poll(Duration.ofSeconds(1));
            for (var r : records) {
                if (!r.key().equals(orderId)) continue;
                view = view.apply(r.value());
                if (r.offset() >= upToOffset) return view;
            }
        }
    }
}