Supermetal now outputs to Kafka with two message formats: a Debezium-compatible envelope for existing pipelines, and a compact native format for new builds.
Under the hood, both are powered by source connectors written from scratch using Apache Arrow in Rust. No JVM. No Kafka Connect. No GC pauses. No snapshot locks, no connector state in Kafka topics.
Initial syncs that take hours or days with Debezium complete in a fraction of the time.
Capabilities:
- JSON and Avro serialization with Confluent Schema Registry support
- Support for Debezium configuration options including tombstones, headers, decimal/binary/time precision handling, and transaction metadata
- Optional transactional writes aligned to source database transaction boundaries
Debezium Compatible Format
{
"payload": {
"before": null,
"after": {
"id": 1,
"first_name": "Anne",
"last_name": "Kretchmar",
"email": "annek@noanswer.org"
},
"source": {
"version": "3.4.1.Final",
"connector": "postgresql",
"name": "PostgreSQL_server",
"ts_ms": 1559033904863,
"ts_us": 1559033904863123,
"ts_ns": 1559033904863123000,
"snapshot": true,
"db": "postgres",
"sequence": "[\"24023119\",\"24023128\"]",
"schema": "public",
"table": "customers",
"txId": 555,
"lsn": 24023128,
"xmin": null
},
"op": "c",
"ts_ms": 1559033904863,
"ts_us": 1559033904863841,
"ts_ns": 1559033904863841257
}
}Supermetal Format
{
"id": 1,
"first_name": "Anne",
"last_name": "Kretchmar",
"email": "annek@noanswer.org",
"_sm_version": 1559033904863841257,
"_sm_deleted": false
}Correctness
Supermetal prioritizes correctness above all. Our testing includes end-to-end type fuzzing from source to target database and sqllogictest style test suites, more details published soon.
We run source databases through Supermetal into a target database, then assert the replicated data matches exactly. For Kafka, we run Debezium alongside Supermetal and compare the output. The results are often byte-for-byte identical and always semantically equivalent, accounting for minor serialization differences.
Performance
Supermetal separates compute from IO. Per message serialization to JSON or Avro is CPU intensive. By moving serialization batches to a work-stealing pool, we use all available CPU cores for compute while IO proceeds independently.
Benchmark: Postgres to Kafka
Sub-second latency up to 15K rows/sec. Above 20K/sec, single-threaded Postgres logical decoding saturates. The Breakdown view shows Read Latency (the time delta between source commit and arrival at Supermetal) accounts for 98% of the end to end latency.
Get started in minutes
curl -fsSL https://trial.supermetal.io/install.sh | shiwr -useb https://trial.supermetal.io/install.ps1 | iex