Skip to main content
PreparationZone Logo
PrepZone

Kafka Connect: Sources, Sinks, and SMTs

Move data in and out of Kafka without custom consumers — JDBC source, S3 sink, and single message transforms.

Pass checkpoint below

Why this matters

  • VaultCommerce uses JDBC source for legacy MySQL reports and S3 sink for compliance archives.
  • Distributed Connect workers share connector tasks across the cluster for fault tolerance.
  • SMTs (Single Message Transforms) filter and reshape records in the pipeline.
  • Connect handles offset management for sources — different from consumer groups.
Postgres / S3Source
Kafka topics
Elasticsearch / S3Sink
Source connectors ingest external data. Sink connectors export to warehouses, indexes, or object storage.

Source and sink connectors

Source: JDBC polls updated_at column → Kafka topic. Sink: S3 connector batches by size/time → Parquet files. VaultCommerce runs 3 Connect workers with group.id=vaultcommerce-connect for HA.

Key points

  • Connector — plugin configuration (Debezium, JDBC, S3, Elasticsearch)
  • Task — parallel unit of work; partition of source table or topic
  • Worker — JVM running connector tasks in distributed mode
  • SMT — in-flight record transform without custom code
  • Converter — JSON, Avro, or ByteArray between Connect and Kafka format

The rest of this guide is part of Pro

Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.

Still to come · 2 sections

  • SMTs and error handling
  • VaultCommerce rollout checklist

Test yourself

Answer these before moving on — recall is what makes it stick.