Why this matters
- VaultCommerce uses JDBC source for legacy MySQL reports and S3 sink for compliance archives.
- Distributed Connect workers share connector tasks across the cluster for fault tolerance.
- SMTs (Single Message Transforms) filter and reshape records in the pipeline.
- Connect handles offset management for sources — different from consumer groups.
Postgres / S3Source
Kafka topics
Elasticsearch / S3Sink
Source and sink connectors
Source: JDBC polls updated_at column → Kafka topic. Sink: S3 connector batches by size/time → Parquet files. VaultCommerce runs 3 Connect workers with group.id=vaultcommerce-connect for HA.
Key points
- Connector — plugin configuration (Debezium, JDBC, S3, Elasticsearch)
- Task — parallel unit of work; partition of source table or topic
- Worker — JVM running connector tasks in distributed mode
- SMT — in-flight record transform without custom code
- Converter — JSON, Avro, or ByteArray between Connect and Kafka format
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 2 sections
- SMTs and error handling
- VaultCommerce rollout checklist
Test yourself
Answer these before moving on — recall is what makes it stick.