Schema Changes

ETL automatically applies the simple column changes listed below when the destination can represent them safely. For unsupported destination changes, ETL either skips the destination DDL with a warning or rejects the change before applying it. These compatibility checks are separate from the source schema-capture behavior described below.

This page describes structural DDL compatibility. Row values are not validated against destination-specific value domains; see Row value handling.

Supported changes

Iceberg is deprecated and rejects every newer schema instead of applying schema-change DDL.

Initial-copy schema protection

An MVCC snapshot determines which rows a copy can see. It does not freeze the relation metadata PostgreSQL uses to interpret them. For each copy unit:

1. The parent transaction locks the table and its existing descendants in ACCESS SHARE mode with NOWAIT, before creating the slot snapshot.
2. ETL verifies that every source relation in the snapshot was already locked, then reads the initial schema.
3. Workers import the snapshot and acquire their own ACCESS SHARE locks with NOWAIT before copying a relation or deparsing its row filter. They retain these locks across chunks.
4. The parent retains the whole copy unit's locks until all source copy workers finish. Protection ends before CDC catch-up.

For partitioned tables, publication discovery determines the copy unit:

Ordinary tables are separate copy units in either mode. Each unit has its own snapshot and retry lifecycle. Discovery does not return a published root alongside the descendants it covers. A foreign-table partition causes a root copy to fail because ETL's CTID copying requires local storage.

This continuous coverage prevents table DDL requiring ACCESS EXCLUSIVE, including incompatible column-layout changes and heap rewrites, from crossing the snapshot boundary or running between chunks. It uses the same parent-lock and worker-NOWAIT pattern as pg_dump parallel dumps.

On a primary, ACCESS SHARE is the same table lock as an ordinary SELECT. It is compatible with normal reads, inserts, updates, deletes, and ordinary vacuum. The snapshot can still prevent vacuum cleanup, and queued ACCESS EXCLUSIVE DDL can delay later application queries.

On a read replica, these locks are local: they do not block primary DDL. Conflicting WAL replay can wait or cancel the copy's queries or sessions under PostgreSQL's standby conflict rules. See Planning long initial copies for source impact and monitoring.

Lock conflicts and retries

Consider a parent holding ACCESS SHARE, a DDL session waiting for ACCESS EXCLUSIVE, and a new copy worker requesting ACCESS SHARE. The worker can queue behind the DDL even though its lock is compatible with the parent's lock. The parent is waiting for the worker in ETL, outside PostgreSQL's lock manager, so PostgreSQL cannot detect this application-level deadlock.

A worker's NOWAIT request fails instead of joining that queue. A conflict fails the attempt with SourceTableCopyLockConflict; the configured retry policy discards the incomplete destination copy and starts from a fresh snapshot. Already-held locks are reused: ETL does not continuously check for later DDL waiters or automatically abort a copy to unblock applications.

Copy connections use a 30-second lock_timeout for other lock acquisitions, including waits for old writers during copy-slot creation. This breaks cycles where those writers depend on DDL blocked by the copy. PostgreSQL-detected deadlocks during copy-slot creation are also retryable. The timeout applies to each acquisition, not the total copy or slot-creation duration. Apply-slot creation temporarily disables it because that transaction holds no copy locks.

Retries belong to the running table-sync worker. If the pipeline stops while a table is persisted as Errored, restarting alone does not resume it. Resolve the cause and reset the table for resynchronization. Exhausted retries also require manual intervention.

What the locks do not protect

ACCESS SHARE does not freeze all metadata. Publication column lists and partition membership are read from the copy's MVCC snapshot; row-filter expressions also come from that snapshot, but pg_get_expr deparses them using internal catalog lookups. The helper limitations below still apply.

The initial schema and identity metadata are read together, and the publication column query uses the same slot snapshot. ETL builds the replication mask from those column names and intersects the identity columns with that mask. COPY chooses columns, filters, and physical partitions through direct catalog SQL instead of pg_publication_tables or pg_partition_tree. Concurrent publication or partition changes do not replace that snapshot-visible selection.

Supported publication column-list changes can proceed and are handled after copying. A partition attached before the snapshot without a pre-existing lock causes a retry; a partition attached after it is excluded from the running copy plan. Neither rule makes live partition-topology changes supported.

Coordinate a stop and resynchronization for changes to namespaces, user-defined types or functions, publication membership, row filters, or partition topology. For example, an enum rename does not acquire ACCESS EXCLUSIVE on every table using the enum. Initial-copy locks also do not fix later event-trigger capture or the other schema-capture cautions.

How schema changes reach a destination

ETL installs a PostgreSQL event trigger for published permanent tables. After a supported ALTER TABLE or publication column-list change:

1. ETL's ddl_command_end trigger describes the table after the command and emits a transactional schema snapshot into WAL, before the transaction commits. A rollback discards the message with the DDL.
2. ETL stores the new versioned table schema.
3. ETL combines the snapshot with column masks from PostgreSQL's RELATION metadata. If no-op DDL produces no fresh relation, ETL reuses the previous masks before the next insert, update, or delete.
4. ETL sends the destination an ordered Event::Relation containing the new ReplicatedTableSchema before the affected data event. PostgreSQL supplies a protocol relation before truncate.
5. Destinations that support the transition apply any required DDL and record the new snapshot before writing following rows.

Relation is an ordered notification, not a batch boundary. One write_events() call can contain several relation and row events, which must be processed in order. Reconnects and maintenance can resend an unchanged snapshot and mask; that identical pair needs no schema transition. Supported publication column-list changes use a newer snapshot even when the full table schema is unchanged.

ETL captures an unprojected table-schema description instead of replaying raw PostgreSQL SQL. It matches columns by PostgreSQL attnum, so a rename keeps its logical identity while a drop and re-add becomes a drop plus an add. The description is intended to represent the post-command schema; the visibility limitations below explain why it is not always a faithful description of the live catalog state.

Why snapshots and replication masks are separate

A table schema and a pipeline's view of that schema are different things. One publication might expose id, email, while another exposes id, status from the same table. The source trigger captures the table's columns without projecting them separately for every pipeline. Each pipeline selects the stored schema version at its position in the stream, then uses its RELATION column metadata to build a replication mask: which columns from that version are actually replicated. An identity mask separately describes the columns PostgreSQL uses to identify rows.

This keeps pipeline-specific projection out of ordinary table-DDL capture. The trigger reports that a table changed and supplies its schema snapshot; each interested pipeline derives its own ReplicatedTableSchema. Snapshot selection follows WAL ordering and recovery state; ETL does not select an arbitrary historical version merely because its column names match.

The same mechanism makes a publication column-list change a logical schema change for a destination, even when the physical table has not changed. Supported column-list changes emit a new snapshot version, and the pipeline's new mask produces the projected add/drop transition using the same ordered schema-planning path as table DDL.

The trigger still checks whether a table is published. Ordinary ALTER TABLE snapshots are not scoped to one publication; ALTER PUBLICATION snapshots include a publication scope so unrelated pipelines can ignore them. That scope does not replace the per-pipeline mask or mean that live publication membership changes are supported.

Ordered schema plans

ETL compares the previously applied logical snapshot with the new one and produces one destination-aware plan. Operations are ordered as follows:

1. Drop columns that are no longer present.
2. Apply renames, using a temporary name when needed to break a rename cycle.
3. Add new columns.
4. Apply type, nullability, and default alterations in order.

Every alteration includes adjacent before and after column schemas. A default addition, removal, or replacement is represented by one Default alteration. For a replacement, destinations that support existing-column defaults first remove the old default idempotently, then set the new default only if it is supported by the current destination renderer.

Mapped destination-name collisions fail before DDL. BigQuery and DuckLake use ASCII-lowercase destination names; ClickHouse, Snowflake, and Iceberg preserve source spelling. Snowflake requires QUOTED_IDENTIFIERS_IGNORE_CASE = FALSE so quoted names remain stable.

Publication column changes

Changing the column list of a table already tracked by the running pipeline is supported. Other live ALTER PUBLICATION changes are not.

When a publication starts exposing an existing column, ETL does not know that column's historical values in the destination. It therefore follows this policy:

• Add the destination column as nullable, even when PostgreSQL says NOT NULL.
• Keep existing destination rows NULL.
• Write the real value from future PostgreSQL row events.
• Install source default metadata only when the destination guarantees that it affects future writes without changing existing rows. BigQuery provides this guarantee; ClickHouse, DuckLake, and Snowflake do not.
• Reject the transition before DDL when the destination type cannot represent an unknown nullable value. BigQuery therefore rejects publication-added arrays and requires a table resynchronization. ClickHouse stores a NULL array as an empty array, so existing rows read an empty array instead.

Stopping publication of a column drops the destination column and its stored history. Publishing it again creates a fresh nullable column; it does not restore the removed values. Resynchronize the table when exact history is required.

Only column-list changes for already tracked tables are supported while a pipeline is running. Use a controlled stop, restart, table resynchronization, or new pipeline for changes to:

• Publication table membership or TABLES IN SCHEMA.
• Row filters or published operation settings.
• Publication names or ownership.
• publish_via_partition_root and root/leaf identity.

Destination data for a table removed from a publication is not automatically deleted.

Defaults and historical rows

Destination defaults are best-effort metadata translations. PostgreSQL still sends evaluated values in future row events, so skipping destination default metadata does not lose replicated row values.

ETL supports deterministic literal defaults such as strings, numbers, booleans, dates, times, timestamps, JSON, and UUIDs when the destination can render them safely. Runtime or session-dependent expressions—including now(), sequences, random(), current_user, and generated UUID functions—are skipped with a warning.

ETL does not issue table-wide UPDATE, MERGE, CTAS/swap, or materialization operations to backfill historical destination rows. The result depends on why the column appeared:

BigQuery is the publication-reveal exception for default metadata: ETL adds the nullable column first and then uses SET DEFAULT, which affects only future inserts.

Destination behavior

See Destinations for maturity, setup requirements, and broader destination limitations.

Recovery and safety

Stored source schemas can be ahead of durable replication progress. For tables already in steady replication, restart selects the schema committed strictly before the selected restart boundary, then applies schema changes in replay order. A transaction whose commit record starts exactly at that boundary remains replayable. This keeps future schemas out of earlier rows without rewinding destination DDL. Tables still handing over from initial sync use their durable handoff state, which can be ahead of the apply worker's restart position.

The initial table-copy schema uses SnapshotId::initial() (0:0), a sentinel without a schema-message commit position. It remains eligible even when the restart frontier is zero.

A corrected restart boundary cannot recreate missing or previously pruned schema history. If a required historical schema is unavailable, resynchronize the table. The current source schema cannot substitute for the schema needed to decode earlier rows.

Destination metadata records the target snapshot and replication mask. While a change is Applying, it also records the previous snapshot and previous replication mask as one logical recovery endpoint.

Initial table setup uses a separate Creating state. If setup is interrupted, ETL retries the destination's idempotent write-path setup for that exact target schema and records Applied only after the destination is ready for writes. Iceberg also uses Creating to bracket its ETL-owned drop-and-recreate implementation of source truncation, so an interrupted recreation can finish before replayed writes continue. Ancillary objects that are not required for writes, such as BigQuery's customer-facing view, have their own idempotent reconciliation. This keeps creation recovery separate from partially applied schema changes, which may require operator recovery.

Applied metadata is authoritative. It records that ETL completed setup for the destination table and logical schema, so a process restart or empty local cache does not trigger creation DDL or structural repair. Destinations may make read-only metadata calls needed to rebuild indispensable process-local state, reopen ephemeral write channels, reconcile fully derived non-data-bearing objects such as views, or clean up ETL-owned temporary artifacts left by a completed operation. Those operations do not authorize table repair. If the data-bearing table was deleted or modified outside ETL, ordinary destination operations fail; resynchronize the table to establish a new, linear lifecycle.

ETL does not infer missing recovery state:

• Previous snapshot and mask values must be stored together.
• Incomplete or stale recovery metadata fails during loading.
• An equal snapshot with a different replication mask is rejected because it has no valid schema ordering.

Older relations never rewind destination DDL. ClickHouse, BigQuery, Snowflake, and Iceberg reject a replayed relation older than their applied destination snapshot. DuckLake skips older relations and uses durable per-table progress to discard already-applied rows and truncates before validating the remaining events against the applied schema. DuckLake commits that progress in the same transaction as the data changes; an older schema on an event outside that durable progress still fails validation.

Recovery then follows each destination's DDL guarantees:

• DuckLake applies a plan transactionally and can retry unambiguous old or target states. Rename-cycle interruptions still require operator recovery.
• ClickHouse resumes only when the physical columns match a complete old or target endpoint. A partially applied DDL sequence requires operator recovery.
• BigQuery and Snowflake leave interrupted nontransactional DDL in Applying for operator recovery rather than guessing what completed.
• Iceberg rejects schema evolution.

Resynchronize the affected table when recovery cannot prove a safe endpoint.

Schema-capture cautions

Initial-copy locks protect the copied table's layout before the snapshot is established and throughout copying. Schema, identity, publication columns, and partition selection use the copy snapshot. These protections do not freeze all referenced objects or extend to later DDL event-trigger capture.

Follow Planning schema changes to reduce schema-capture risks. A successful source DDL command or schema validation does not guarantee that every captured field describes the same schema state.

Catalog rows and PostgreSQL helper functions can see different metadata

SQL reads of pg_catalog follow an MVCC snapshot, but helpers such as format_type and pg_get_expr can use newer internal catalog metadata. A concurrent change to an enum, type, function, or namespace can therefore produce a name or expression that disagrees with the captured catalog rows. The table's copy lock does not protect all of these shared objects.

COPY reads publication columns, row-filter definitions, and partition membership through snapshot-visible catalog SQL. Row-filter deparsing still uses pg_get_expr, and the DDL event trigger still discovers publication members through pg_publication_tables. Use a maintenance window for changes to shared objects and unsupported publication settings.

Concurrent DDL can outlive the capturing transaction's snapshot

The DDL event trigger runs in the source transaction that executes the DDL. With REPEATABLE READ or SERIALIZABLE, a snapshot established before another transaction's committed schema change can omit that change from the trigger's catalog queries. PostgreSQL's internal relation cache can recognize the newer table definition without refreshing the SQL snapshot. Acquiring a table lock or committing the transaction does not repair an incomplete captured schema.

Run migrations in short, fresh READ COMMITTED transactions on the source. Changing ETL's connection isolation does not change the trigger's visibility. SERIALIZABLE is not a workaround: it does not guarantee that every affected DDL transaction will abort. See PostgreSQL's transaction isolation rules.

Separate trigger queries can mix committed states

The trigger reads table names, columns, and replica identity in separate SQL queries. Under READ COMMITTED, another transaction can commit between those queries. Each STABLE helper uses its calling query's snapshot, but separate calls from the VOLATILE trigger need not share one snapshot. See PostgreSQL's function snapshot rules.

A table alteration holding ACCESS EXCLUSIVE prevents another transaction from changing that table during capture. A publication-level command might not hold equivalent protection on every table it captures. Its message can then combine an earlier column list with a newer replica identity. Serialize table and publication migrations affecting the same tables; READ COMMITTED alone does not prevent this race.

Relation metadata cannot distinguish every schema mismatch from filtering

ETL combines stored schema snapshots with PostgreSQL RELATION messages to identify replicated columns. A column missing from a relation message can mean publication filtering or inconsistent schema history. Unknown columns can be rejected, but a valid-looking subset does not prove that the selected schema is correct.

If schema history is inconsistent, resolve the source changes and resynchronize the affected table. Restarting the pipeline alone cannot repair that history. During initial-sync handover, ETL fails when it cannot establish a complete decoding schema; retry after schema activity settles or resynchronize the table.

Planning schema changes

The supported operating model permits ordinary reads and writes while keeping schema changes controlled. Apply these rules to replicated tables and the objects they depend on:

The initial-copy implementation can preserve snapshot-visible publication column lists across concurrent changes. Scheduling those changes after initial sync remains the recommended operating rule. ETL does not automatically reject or detect every operation outside these rules.

Coordinate migrations across every pipeline using the affected tables or shared objects. Stopping a pipeline does not disable the source event trigger, so keep maintenance transactions short and use READ COMMITTED too. A restart alone does not rebuild destination tables or repair inconsistent schema history; plan any required table resynchronization before resuming replication.

• Table creation, removal, and rename are pipeline lifecycle operations, not live schema changes.
• General constraints, identity changes, replica-identity changes, and indexes are not portable destination DDL operations. Destinations may reject key changes required by their write model.
• Destination default and add-column behavior is not a historical-data migration contract. Resynchronize when exact historical values matter.

Custom destinations

Custom destinations receive schema notifications through Event::Relation. They should:

1. Process the batch in order and flush writes using the previous schema.
2. Compare the incoming snapshot and replication mask with applied metadata; an identical pair needs no transition. For a newer snapshot, compare the projected logical schemas.
3. Validate the complete plan before recording Applying or issuing DDL.
4. Apply supported operations in the supplied order.
5. Record Applied only after following rows can safely use the new schema.

See Custom Implementations for the consumer contract and Events for relation-event details.