Kafka Connect for Data Pipelines

Moving data between Kafka and external systems is such a common need that Kafka Connect exists purely for it. You configure connectors with JSON instead of writing and operating custom producer code.

A source connector

This example streams change events from Postgres into Kafka using Debezium — no application code required.

{
  "name": "orders-source",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "database.dbname": "shop",
    "table.include.list": "public.orders",
    "topic.prefix": "cdc"
  }
}

Deploy via the REST API

Connect workers expose a REST API for registering and monitoring connectors.

curl -X POST http://localhost:8083/connectors \
  -H 'Content-Type: application/json' -d @pg-source.json

Takeaways

• Source connectors pull data in; sink connectors push it out to stores like S3 or Elasticsearch.
• Connect handles offset tracking, restarts, and scaling for you.
• Use Single Message Transforms for lightweight reshaping without custom code.

Run Connect in distributed mode for production so tasks rebalance across workers when one fails.