Debezium PostgreSQL Connector — Failure Modes and Slot Management
Debezium’s reliability comes from PostgreSQL’s replication slots — they guarantee that no WAL is discarded before the connector has consumed it, so a crashed connector can catch up without data loss. But that same guarantee creates the biggest operational risk: an abandoned slot tells PostgreSQL to hold WAL indefinitely, and on a busy database the WAL can fill the disk. This post covers connector failure and recovery, leaked slots, and how Debezium handles schema evolution.
Connector Failure and WAL Accumulation
A replication slot is a server-side bookmark. While the connector is running, it continuously reads from the WAL and confirms its position back to PostgreSQL, which allows PostgreSQL to recycle old WAL segments. When the connector goes down, the slot stays — PostgreSQL holds all WAL from the slot’s last confirmed position forward.
Seeing It
Check the current slot state while the connector is running:
1
2
SELECT slot_name, active, restart_lsn, confirmed_flush_lsn
FROM pg_replication_slots;
The active column is true and the lag is small — Debezium is keeping up. Now delete the connector:
1
curl -X DELETE http://localhost:8083/connectors/learn-cdc-connector
The connector is gone, but the slot is still there:
1
SELECT slot_name, active, restart_lsn FROM pg_replication_slots;
active is now false. PostgreSQL will keep all WAL from this point forward because the slot indicates its consumer has not read past here.
WAL Growth While Down
Make some changes while the connector is offline:
1
2
3
4
5
6
7
INSERT INTO orders (customer, product, amount)
VALUES ('eve', 'Widget E', 59.99);
INSERT INTO orders (customer, product, amount)
VALUES ('frank', 'Widget F', 14.99);
UPDATE customers SET tier = 'premium' WHERE id = 2;
Check the lag:
1
2
3
4
SELECT slot_name,
pg_size_pretty(pg_current_wal_lsn() - restart_lsn) AS wal_retained
FROM pg_replication_slots
WHERE slot_name = 'learn_cdc_slot';
The retained WAL is growing. On a production database with high write throughput, this can fill the disk within hours.
Recovery
Re-register the connector with the same slot.name. It reconnects to the existing slot, reads all the WAL that accumulated while it was down, and produces the missed events to Kafka. No data is lost.
This is the key guarantee: as long as the replication slot exists, PostgreSQL will not discard WAL that has not been consumed. The connector can go down and come back, and it picks up exactly where it left off.
Leaked Slots
A leaked slot is a replication slot whose connector has been permanently removed but the slot was never dropped. It is the single biggest operational risk when running Debezium.
How It Happens
Deleting a Debezium connector through the Kafka Connect REST API removes the connector process, but it does not drop the replication slot in PostgreSQL. The slot stays, active = false, telling PostgreSQL to hold WAL indefinitely.
In a lab environment this is harmless — a few extra kilobytes of WAL. In production with continuous writes, the WAL grows without bound because PostgreSQL will not recycle any segment past the slot’s LSN:
- WAL grows unbounded
- Disk fills up
- PostgreSQL stops accepting writes
- Outage
Detecting Leaked Slots
1
2
3
4
SELECT slot_name, active,
pg_size_pretty(pg_current_wal_lsn() - restart_lsn) AS wal_retained,
pg_size_pretty(pg_current_wal_lsn() - confirmed_flush_lsn) AS consumer_lag
FROM pg_replication_slots;
Alert if:
- Any slot has
active = falsefor more than a few minutes wal_retainedexceeds a threshold (e.g., 1 GB)consumer_lagis growing steadily (connector falling behind)
Cleaning Up
1
2
3
4
SELECT pg_drop_replication_slot('learn_cdc_slot');
-- Verify
SELECT slot_name, active FROM pg_replication_slots;
Name slots explicitly (e.g., orders_cdc_slot, analytics_cdc_slot) so they are easy to identify when monitoring. The default Debezium-generated name (debezium) is ambiguous if multiple connectors run against the same database.
Schema Evolution
What happens when a table’s schema changes while Debezium is streaming?
Adding a Column
1
2
3
4
5
6
7
ALTER TABLE orders ADD COLUMN notes TEXT;
INSERT INTO orders (customer, product, amount, notes)
VALUES ('grace', 'Widget G', 24.99, 'Rush delivery');
INSERT INTO orders (customer, product, amount)
VALUES ('henry', 'Widget H', 34.99);
Both events include the notes field — one with the value, one with null. Debezium picks up the schema change automatically. No connector restart is needed.
Renaming a Column
1
2
3
4
ALTER TABLE orders RENAME COLUMN notes TO remarks;
INSERT INTO orders (customer, product, amount, remarks)
VALUES ('iris', 'Widget I', 44.99, 'After rename');
The event has remarks instead of notes. Debezium reflects the current schema.
Dropping a Column
1
2
3
4
ALTER TABLE orders DROP COLUMN remarks;
INSERT INTO orders (customer, product, amount)
VALUES ('jack', 'Widget J', 54.99);
The event no longer has the remarks field.
What This Means
Debezium does not care about schema evolution. It forwards whatever the WAL gives it as JSON. Column added, renamed, dropped — Debezium serializes the current row state and produces it to Kafka. No restart, no config change.
The schema evolution problem moves entirely downstream — the consumer of the Kafka events must handle the changing JSON shape. Common approaches:
- Store events as JSON blobs — use a schemaless column type (e.g.,
JSONBin PostgreSQL,VARIANTin Snowflake,STRINGin BigQuery) so the varying shapes do not cause failures. The JSON blob just has different keys over time. - Schema registry — use Avro or Protobuf with a schema registry that tracks schema versions and handles compatibility. Consumers deserialize using the schema version embedded in each message.
- Flatten on ingest — use ExtractNewRecordState to flatten the envelope, then let the sink connector handle schema mapping. Some sink connectors (e.g., JDBC sink) can auto-create and alter target table columns.
The before/after fields in the event envelope contain the raw row data. If you store those as schemaless JSON, adding or dropping columns in the source table has no effect on the landing table — the JSON blob absorbs the change. Schema evolution on the event envelope itself (e.g., a Debezium version upgrade adding a new metadata field to source) is a separate concern handled by the sink connector or schema registry.
Monitoring Checklist
A minimal monitoring setup for Debezium in production:
PostgreSQL Side
1
2
3
4
5
6
7
8
9
10
11
12
13
14
-- Replication slot health
SELECT slot_name, active,
pg_size_pretty(pg_current_wal_lsn() - restart_lsn) AS wal_retained,
pg_size_pretty(pg_current_wal_lsn() - confirmed_flush_lsn) AS consumer_lag
FROM pg_replication_slots;
-- Publication tables
SELECT * FROM pg_publication_tables;
-- Replica identity per table
SELECT relname, relreplident
FROM pg_class
WHERE relname IN ('orders', 'customers');
-- 'f' = FULL, 'd' = DEFAULT, 'n' = NOTHING, 'i' = INDEX
Kafka Connect Side
1
2
3
4
5
6
7
8
# Connector status
curl http://localhost:8083/connectors/orders-cdc-connector/status | jq
# List all connectors
curl http://localhost:8083/connectors | jq
# Task-level status (a connector can have multiple tasks)
curl http://localhost:8083/connectors/orders-cdc-connector/tasks/0/status | jq
What to Alert On
| Signal | Meaning | Action |
|---|---|---|
Slot active = false for > 5 min |
Connector is down | Investigate connector status, restart if needed |
wal_retained > threshold |
WAL accumulating | Either the connector is behind or the slot is leaked |
Connector state FAILED |
Connector crashed | Check task error in status endpoint, fix config or data issue, restart |
| Consumer lag growing | Connector falling behind | Check for large transactions, schema changes, or resource constraints |
| No new messages on CDC topic | Either no writes or connector stalled | Cross-reference with database write activity |
