# Supabase ETL Documentation
> A high-performance Postgres replication engine written in Rust. Embed it in your Rust application or run it as a standalone binary.
- Project status: Supabase ETL is under active development. APIs and setup steps may change before the first stable release.
- Canonical documentation: https://supabase.github.io/etl/
- Repository: https://github.com/supabase/etl
- Managed product: https://supabase.com/docs/guides/database/replication/pipelines
Terminology: the two replication phases are **initial sync** and **ongoing replication**. Supabase ETL initially syncs existing table rows, then replicates subsequent changes and delivers them to the destination. Streaming is a transfer mode, not a replication phase.
---
# Home
> A high-performance Postgres replication engine written in Rust. Embed it in your Rust application or run it as a standalone binary.
- Canonical HTML: https://supabase.github.io/etl/
- Agent-readable Markdown: https://supabase.github.io/etl/index.md
- Source: https://github.com/supabase/etl/blob/main/site/content/docs/index.mdx
A high-performance Postgres replication engine written in Rust. Embed it in your Rust application or run it as a standalone binary. For each published table, it performs an initial sync of existing rows, then replicates changes to a destination. Supabase ETL is under active development. APIs and setup steps may change before the first stable release.
For the managed Supabase product, use [Supabase Pipelines](https://supabase.com/docs/guides/database/replication/pipelines).
## Documentation map
### Get started
- [First Pipeline](https://supabase.github.io/etl/guides/first-pipeline.md): Learn Supabase ETL by building a working Postgres replication pipeline.
- [Standalone Replicator](https://supabase.github.io/etl/guides/standalone-replicator.md): Build, configure, and run Supabase ETL as a standalone process.
### Guides
- [Configure Postgres](https://supabase.github.io/etl/guides/configure-postgres.md): Set up Postgres with the permissions and settings required by Supabase ETL.
- [Custom Implementations](https://supabase.github.io/etl/guides/custom-implementations.md): Implement your own stores and destinations.
### Concepts
- [Logical Replication](https://supabase.github.io/etl/explanation/concepts.md): Essential Postgres logical replication concepts for working with Supabase ETL.
- [Architecture](https://supabase.github.io/etl/explanation/architecture.md): How Supabase ETL initially syncs tables and replicates changes to destinations.
- [Schema Changes](https://supabase.github.io/etl/explanation/schema-changes.md): How Supabase ETL handles DDL and evolving table schemas.
### Reference
- [Destinations](https://supabase.github.io/etl/reference/destinations.md): Official built-in destinations, maturity, requirements, and limitations.
- [Events](https://supabase.github.io/etl/explanation/events.md): Understand the events Supabase ETL delivers during ongoing replication.
- [Extension Points](https://supabase.github.io/etl/explanation/traits.md): Traits you implement to customize Supabase ETL behavior.
## Replication phases
1. **Initial sync:** Copy the existing rows selected by the publication.
2. **Ongoing replication:** Capture subsequent inserts, updates, deletes, and truncates, then deliver those changes as ordered events.
Streaming describes a transfer mode that may be used within either phase; it is not a separate replication phase. Across both phases, ETL persists checkpoints and table state so replication can recover safely after a restart.
---
# Architecture
> How Supabase ETL initially syncs tables and replicates changes to destinations.
- Canonical HTML: https://supabase.github.io/etl/explanation/architecture/
- Agent-readable Markdown: https://supabase.github.io/etl/explanation/architecture.md
- Source: https://github.com/supabase/etl/blob/main/site/content/docs/explanation/architecture.mdx
Supabase ETL uses **Postgres logical replication** to initially sync published
tables, then replicate subsequent changes to a destination in near real time.
## Overview [#overview]
## How It Works [#how-it-works]
ETL operates in **two phases**:
### Phase 1: Initial Sync [#phase-1-initial-sync]
When a pipeline starts, it copies all existing data from each table in the publication. Multiple **Table Sync Workers** run in parallel to copy tables concurrently. Each worker:
1. Creates a replication slot to capture a consistent snapshot
2. Copies all rows using Postgres `COPY`
3. Sends rows to the destination via `write_table_rows()`
> **Note**
>
> During this phase, `Begin` and `Commit` events may be delivered multiple times
> because workers consume slots in parallel. This is expected: destinations
> should rely on **per-table event ordering** and **idempotent writes** rather
> than treating transaction markers as a complete global transaction boundary.
### Phase 2: Ongoing Replication [#phase-2-ongoing-replication]
Once tables are copied, the **Apply Worker** handles ongoing replication from
the Postgres WAL. It:
1. Receives change events (inserts, updates, deletes)
2. Batches events for efficiency
3. Sends batches to the destination via `write_events()`
### Schema Changes [#schema-changes]
ETL supports simple column evolution from `ALTER TABLE` and supported
`ALTER PUBLICATION` changes. A source-side event trigger emits internal schema
messages for published permanent tables, ETL stores a new schema snapshot, and
destinations observe the change through a fresh `Relation` event before
following row events. See [Schema Changes](https://supabase.github.io/etl/explanation/schema-changes.md) for
the supported operations and limitations.
## Core Components [#core-components]
### Pipeline [#pipeline]
The **central orchestrator** that manages the entire replication process. It spawns workers, coordinates state transitions, and handles shutdown.
### Destination [#destination]
The `Destination` trait receives bulk rows during initial sync and event batches
during catch-up and ongoing replication. Result handles let a destination distinguish
accepted work from durable work without blocking dispatch. See [Extension
Points](https://supabase.github.io/etl/explanation/traits.md#destination) for the method contract and [Custom
Stores and Destinations](https://supabase.github.io/etl/guides/custom-implementations.md) for an implementation.
### Store [#store]
Three store traits persist the state needed to resume after restarts:
* **StateStore**: Tracks table state, persisted replication checkpoints, and destination table metadata
* **SchemaStore**: Stores versioned table schema information (columns, types, primary keys, snapshot IDs) and prunes obsolete schema versions behind persisted checkpoints while preserving the retained boundary schema and newer versions
* **TableStateLifecycleStore**: Prepares table-copy state, resets table states for resync, and deletes all ETL-owned state when a table leaves the publication
See [Extension Points](https://supabase.github.io/etl/explanation/traits.md#schemastore) for the cache and
durability contracts.
## Delivery Guarantees [#delivery-guarantees]
ETL provides **at-least-once delivery**. If restarts occur, some events may be delivered more than once. This is a deliberate design choice.
### Why Not Exactly-Once? [#why-not-exactly-once]
Exactly-once delivery requires distributed transactions between Postgres and the destination, adding complexity and latency. Instead, ETL optimizes for throughput and simplicity while minimizing duplicates through:
* **Controlled shutdown**: The pipeline attempts to finish in-flight work before its shutdown deadline; interrupted work can be replayed after restart
* **Frequent status updates**: Progress is reported to Postgres regularly, reducing the replay window after restarts
### Handling Duplicates [#handling-duplicates]
Destinations should make writes **idempotent** using the source table's replica
identity or primary key plus ETL's event ordering metadata. For append-style CDC
tables, persist a sequence key derived from `commit_lsn` and `tx_ordinal`. For
current-state tables, upsert by the destination's chosen row key so replayed
events converge to the same state.
The `commit_lsn` and `tx_ordinal` fields on sequenced events provide stable
**ordering and checkpointing**. For example, BigQuery destinations use the pair
to maintain correct event order in destination tables. See [Event
Types](https://supabase.github.io/etl/explanation/events.md#understanding-event-sequence-keys) for details.
## Table States [#table-states]
Each table progresses through these states:
| State | Set By | Description |
| ---------------- | ----------------- | -------------------------------------------------------------------------------- |
| **Init** | Pipeline | Table discovered, ready for initial sync |
| **DataSync** | Table Sync Worker | Initial table copy in progress |
| **FinishedCopy** | Table Sync Worker | Initial sync complete |
| **SyncWait** | Table Sync Worker | Waiting for Apply Worker to pause (in-memory only) |
| **Catchup** | Apply Worker | Apply Worker paused; Table Sync Worker catching up to its LSN (in-memory only) |
| **SyncDone** | Table Sync Worker | Catch-up complete; durable decoder retained until Apply materializes local state |
| **Ready** | Apply Worker | Apply Worker now handles this table exclusively |
| **Errored** | Either | Error occurred; contains reason, solution hint, and retry policy |
---
# Logical Replication
> Essential Postgres logical replication concepts for working with Supabase ETL.
- Canonical HTML: https://supabase.github.io/etl/explanation/concepts/
- Agent-readable Markdown: https://supabase.github.io/etl/explanation/concepts.md
- Source: https://github.com/supabase/etl/blob/main/site/content/docs/explanation/concepts.md
Read this first if Postgres logical replication is new to you.
## What is Logical Replication? [#what-is-logical-replication]
Postgres supports two types of replication:
| Type | What it copies | Use case |
| ------------ | -------------------------------------------- | -------------------------------- |
| **Physical** | Exact byte-for-byte copy of data files | Disaster recovery, read replicas |
| **Logical** | Decoded row changes (INSERT, UPDATE, DELETE) | Data integration, ETL, CDC |
ETL uses logical replication because decoded row changes can be sent to systems
other than Postgres.
## The Write-Ahead Log (WAL) [#the-write-ahead-log-wal]
Before Postgres modifies data on disk, it first writes the change to the **Write-Ahead Log (WAL)**. This guarantees durability: if Postgres crashes, it can replay the WAL to recover.
For logical replication, Postgres decodes the WAL back into **logical changes**:
ETL receives these decoded events and forwards them to downstream consumers.
### WAL Level [#wal-level]
Postgres must be configured to record enough information for logical decoding:
```ini
# In postgresql.conf
wal_level = logical
```
With `wal_level = logical`, Postgres records additional metadata needed to reconstruct row changes. Lower levels (`replica`, `minimal`) **do not capture enough detail**.
## Publications [#publications]
A **publication** defines which tables to replicate. Think of it as a filter that says "replicate changes from these tables."
```sql
-- Replicate specific tables
CREATE PUBLICATION my_publication FOR TABLE users, orders;
-- Replicate all tables (use with caution)
CREATE PUBLICATION my_publication FOR ALL TABLES;
```
When you create an ETL pipeline, you specify which publication to consume.
**Only tables and operations selected by that publication are replicated.**
### What Publications Control [#what-publications-control]
* **Which tables**: Only tables in the publication are replicated
* **Which operations**: You can filter to only INSERT, UPDATE, or DELETE
* **Which columns** (Postgres 15+): Replicate only specific columns
* **Which rows** (Postgres 15+): Filter rows with a WHERE clause
## Replication Slots [#replication-slots]
A **replication slot** is a bookmark that tracks how far a consumer has read in the WAL.
### Why Slots Exist [#why-slots-exist]
Without slots, Postgres would delete old WAL files when it no longer needs them for crash recovery. If ETL disconnects temporarily, it needs those WAL files to catch up when it reconnects.
Replication slots tell Postgres: **"Don't delete WAL files until this consumer has processed them."**
```sql
-- View existing slots
SELECT slot_name, confirmed_flush_lsn, active
FROM pg_replication_slots;
```
### How ETL Uses Slots [#how-etl-uses-slots]
ETL creates replication slots automatically:
| Slot | Purpose |
| -------------------------------------------------- | --------------------------------- |
| `supabase_etl_apply_{pipeline_id}` | Main slot for ongoing replication |
| `supabase_etl_table_sync_{pipeline_id}_{table_id}` | Temporary slots for initial sync |
The Apply Worker uses one persistent slot. Table Sync Workers create temporary slots during initial sync, then delete them.
### Slot Risks [#slot-risks]
Slots prevent WAL cleanup. If ETL stops consuming because of crashes, network issues, or a slow consumer, WAL files accumulate on disk. **This can fill your disk.**
To mitigate this risk:
* Monitor slot lag with `pg_replication_slots`
* Set `max_slot_wal_keep_size` to limit WAL retention
* Alert when slots fall behind
See [Configure Postgres](https://supabase.github.io/etl/guides/configure-postgres.md#wal-buildup-and-disk-usage) for details.
## The pgoutput Decoder [#the-pgoutput-decoder]
When Postgres decodes WAL for logical replication, it uses a **decoder plugin**. ETL uses `pgoutput`, Postgres's built-in decoder.
The decoder transforms binary WAL records into structured messages:
| Message | Meaning |
| ---------- | ----------------------------- |
| `BEGIN` | Transaction started |
| `RELATION` | Table schema (columns, types) |
| `INSERT` | Row added |
| `UPDATE` | Row modified |
| `DELETE` | Row removed |
| `TRUNCATE` | Table cleared |
| `COMMIT` | Transaction completed |
ETL receives these messages and converts them to events.
## Why Two Phases? [#why-two-phases]
ETL replicates data in two phases: **initial sync** and **ongoing
replication**.
### Phase 1: Initial Sync [#phase-1-initial-sync]
Logical replication only captures **changes**. It does not know about data that existed before replication started.
So ETL first copies all existing rows using Postgres's `COPY` command:
1. Create replication slot (captures consistent snapshot point)
2. COPY all rows from the table
3. Begin ongoing replication from the snapshot point
The slot ensures **no changes are lost** between the snapshot and ongoing
replication.
### Phase 2: Ongoing Replication [#phase-2-ongoing-replication]
After initial sync, ETL begins ongoing replication. It captures subsequent
changes from the WAL and delivers them to the destination:
Each change is delivered as an `Event` through `write_events()`.
Large tables can spend significant time in initial sync. ETL exposes separate
`write_table_rows()` and `write_events()` methods so destinations can optimize
initial-copy rows and change events independently.
## Replica Identity [#replica-identity]
**REPLICA IDENTITY** controls what data Postgres includes in UPDATE and DELETE events.
### The Problem [#the-problem]
When a row is updated or deleted, downstream systems need enough old-row information to identify which source row changed.
The important nuance is that PostgreSQL does **not** always send an old-side tuple for `UPDATE`.
Under key-based replica identity, it only sends a key image when it is needed.
For `DELETE`, PostgreSQL sends an old-side tuple whenever the delete is publishable.
This means replica identity is both a **PostgreSQL logging rule** and a **consumer contract** for
downstream consumers. It determines whether an event contains enough old-row
data to match an existing row, detect key changes, or compare before-and-after
values.
### Settings [#settings]
```sql
-- See current setting (d=default, f=full, n=nothing, i=index)
SELECT relname, relreplident FROM pg_class WHERE relname = 'your_table';
-- Change setting
ALTER TABLE your_table REPLICA IDENTITY FULL;
```
| Setting | Published `UPDATE` payload | Published `DELETE` payload | Notes |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------- |
| `DEFAULT` with a primary key | Old primary-key columns only when PostgreSQL determines the old key must be logged; otherwise no old tuple | Old primary-key columns | Most tables with a primary key |
| `DEFAULT` without a primary key | Source `UPDATE` is rejected when the table publishes updates | Source `DELETE` is rejected when the table publishes deletes | Equivalent to having no usable replica identity for update/delete |
| `FULL` | Full old row | Full old row | Use when consumers need full old-row images |
| `NOTHING` | Source `UPDATE` is rejected when the table publishes updates | Source `DELETE` is rejected when the table publishes deletes | Suitable only when updates/deletes are not published |
| `USING INDEX` | Old replica-identity index columns only when PostgreSQL determines the old key must be logged; otherwise no old tuple | Old replica-identity index columns | Tables whose replication identity differs from the primary key |
### Impact on ETL [#impact-on-etl]
ETL preserves PostgreSQL's old-row semantics in update and delete events:
```rust
pub old_table_row: Option
```
* `Some(OldTableRow::Key(row))` means PostgreSQL sent only the replica-identity columns, normalized into replicated table-column order.
* `Some(OldTableRow::Full(row))` means PostgreSQL sent the full old row.
* `None` means PostgreSQL did not send an old-side tuple for that update. This
is normal under `DEFAULT` or `USING INDEX` when PostgreSQL determines no
old-side image is required.
For `FULL`, PostgreSQL sends a full old row for every published update and delete.
For `DELETE`, valid pgoutput messages always include either a full old row or a
key image. `REPLICA IDENTITY NOTHING`, and `DEFAULT` on a table without a primary
key, do not produce update/delete events when those actions are published; the
source statement is rejected instead.
The Rust event API keeps the old-row fields optional at the boundary, but those
`None` cases are broader than the PostgreSQL pgoutput shapes described here.
**TOAST adds one more wrinkle.** PostgreSQL can mark unchanged toasted update values
as `UnchangedToast` instead of resending the value. ETL can reconstruct those
values only if the old-side row image contains them, so tables with toasted
columns can produce partial update rows unless they use `REPLICA IDENTITY FULL`
or the missing values are present in a logged key image.
If you need **old values** for auditing, comparison, complete replacement rows,
or reliable reconstruction of unchanged toasted columns, set
`REPLICA IDENTITY FULL` on those tables. If a consumer only needs stable key
values, `DEFAULT` with a primary key or `USING INDEX` is usually enough, but
update events will not always include `old_table_row`.
## LSN (Log Sequence Number) [#lsn-log-sequence-number]
Every position in the WAL has a unique **LSN** - a monotonically increasing pointer.
```text
Format: 0/16B3748 (segment/offset)
```
### LSNs in Events [#lsns-in-events]
Sequenced ETL events include a commit LSN and transaction-local ordinal:
| Field | Meaning |
| ------------ | --------------------------------------------- |
| `commit_lsn` | LSN of the commit message in the WAL |
| `tx_ordinal` | Zero-based event order within the transaction |
Multiple events in the same transaction share the same `commit_lsn`; their
`tx_ordinal` values distinguish their order. Relation events are connection-local
metadata and do not have an event sequence key.
## Persisted State [#persisted-state]
ETL persists the state needed to resume safely after a restart:
ETL stores:
| State | Purpose |
| -------------------------------- | ---------------------------------------------------------------------------- |
| Table state | Track each table from initial sync through ongoing replication |
| Persisted replication checkpoint | Resume workers from a safe replay frontier |
| Table schemas | Decode events against the correct versioned schema |
| Destination table metadata | Track destination table IDs, applied schema snapshots, and replication masks |
The built-in `PostgresStore` persists to your Postgres database and runs its
state-store migrations when it is created. If the pipeline reads from a
read-only replica, configure `store_pg_connection` to point at a writable
Postgres endpoint for this state. `MemoryStore` is for testing only - state is
lost on restart. `Pipeline::start()` runs the ETL source migrations that install
schema helpers and the DDL event trigger before replication begins. See
[Architecture](https://supabase.github.io/etl/explanation/architecture.md) for the worker lifecycle and
[Configure Postgres](https://supabase.github.io/etl/guides/configure-postgres.md) for production settings.
---
# Events
> Understand the events Supabase ETL delivers during ongoing replication.
- Canonical HTML: https://supabase.github.io/etl/explanation/events/
- Agent-readable Markdown: https://supabase.github.io/etl/explanation/events.md
- Source: https://github.com/supabase/etl/blob/main/site/content/docs/explanation/events.mdx
During ongoing replication, Supabase ETL delivers events decoded from Postgres
logical replication via `write_events()`. This page documents their Rust shapes
and PostgreSQL semantics.
## Event Overview [#event-overview]
| Event | Description | Has Table ID |
| ------------- | ----------------- | ------------ |
| `Begin` | Transaction start | No |
| `Commit` | Transaction end | No |
| `Insert` | New row added | Yes |
| `Update` | Row modified | Yes |
| `Delete` | Row removed | Yes |
| `Truncate` | Table cleared | Yes |
| `Relation` | Table schema | Yes |
| `Unsupported` | Unknown event | No |
## Data Modification Events [#data-modification-events]
These events carry **row data** and are associated with specific tables.
### Row Images [#row-images]
Data modification events use **row-image helper types**:
```rust
pub enum UpdatedTableRow {
Full(TableRow),
Partial(PartialTableRow),
}
pub enum OldTableRow {
Full(TableRow),
Key(TableRow),
}
```
`TableRow` is a **complete dense row**. Its values are ordered to match the
replicated table-column order.
`PartialTableRow` is used when an update row is **not complete**. It exposes:
* `total_columns()`: the number of replicated columns in the table schema;
* `values()`: present values in replicated table-column order, excluding missing columns;
* `missing_column_indexes()`: zero-based replicated-column indexes for values ETL could not reconstruct.
`OldTableRow::Full(row)` contains a **complete old row**. `OldTableRow::Key(row)`
contains only replica-identity columns, densely packed in replicated
table-column order.
### Insert [#insert]
A new row was added to a table.
```rust
pub struct InsertEvent {
pub commit_lsn: PgLsn,
pub tx_ordinal: u64,
pub replicated_table_schema: ReplicatedTableSchema,
pub table_row: TableRow,
}
```
### Update [#update]
An existing row was modified.
```rust
pub struct UpdateEvent {
pub commit_lsn: PgLsn,
pub tx_ordinal: u64,
pub replicated_table_schema: ReplicatedTableSchema,
pub updated_table_row: UpdatedTableRow,
pub old_table_row: Option,
}
```
`updated_table_row` is the **authoritative post-update payload**:
* `UpdatedTableRow::Full` when ETL knows every replicated column value after decoding the update.
* `UpdatedTableRow::Partial` when PostgreSQL emitted `UnchangedToast` fields that ETL could not reconstruct safely.
`old_table_row` provides optional context for row matching, key changes, and
reconstruction.
#### Unchanged Toast [#unchanged-toast]
PostgreSQL may encode an unchanged toasted value as `UnchangedToast` in the
update's new tuple instead of resending the full column value. ETL can turn the
update into `UpdatedTableRow::Full` only when that value can be recovered from
the old-side row image:
* a `FULL` old row can recover unchanged toasted values for any replicated column;
* a key-only old row can recover unchanged toasted values only for
replica-identity columns included in that key image;
* no old row means unchanged toasted values cannot be recovered from the event.
When any `UnchangedToast` field cannot be recovered, ETL emits
`UpdatedTableRow::Partial` with known values and missing column indexes instead
of pretending the unknown value is `NULL` or a replacement value.
Destinations that need complete replacement rows, full before/after comparison,
or complete audit records should require `REPLICA IDENTITY FULL` for tables with
toasted columns, or keep enough prior state to fill missing values themselves.
#### Old Row Mapping [#old-row-mapping]
ETL maps PostgreSQL pgoutput update tuple markers directly:
| pgoutput marker | ETL field |
| ----------------------- | ------------------------------ |
| `O` old tuple | `Some(OldTableRow::Full(row))` |
| `K` old key | `Some(OldTableRow::Key(row))` |
| no old tuple/key marker | `None` |
| `N` new tuple | `updated_table_row` |
PostgreSQL chooses which old-side marker to emit from the table's replica
identity:
| REPLICA IDENTITY | `old_table_row` contains for published updates |
| ------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `FULL` | `Some(OldTableRow::Full(row))` |
| `DEFAULT` with a primary key | `Some(OldTableRow::Key(row))` when PostgreSQL determines the old key must be logged, otherwise `None` |
| `DEFAULT` without a primary key | Source `UPDATE` is rejected when the table publishes updates |
| `USING INDEX` | `Some(OldTableRow::Key(row))` when PostgreSQL determines the old key must be logged, otherwise `None` |
| `NOTHING` | Source `UPDATE` is rejected when the table publishes updates |
For `UPDATE`, PostgreSQL only sends an old key image under `DEFAULT` or
`USING INDEX` when it determines the old key must be logged. In practice, that
happens when:
* any replica-identity column changed, or
* any replica-identity column contains external data, such as a toasted value
that must be available from the old tuple.
That means non-identity updates under `DEFAULT` or `USING INDEX` often arrive with `old_table_row = None`.
Under `FULL`, PostgreSQL sends a full old row for every published update.
A `FULL` update with `old_table_row = None` is not a pgoutput shape.
When handling updates:
1. Treat `updated_table_row` as the post-update payload and handle
`UpdatedTableRow::Partial` explicitly.
2. Treat `OldTableRow::Key` as replica-identity values, not necessarily the
primary key. Use `OldTableRow::Full` for complete before-images.
3. If `old_table_row` is `None`, match or upsert from the new row according to
the destination's keying model.
### Delete [#delete]
A row was removed from a table.
```rust
pub struct DeleteEvent {
pub commit_lsn: PgLsn,
pub tx_ordinal: u64,
pub replicated_table_schema: ReplicatedTableSchema,
pub old_table_row: Option,
}
```
For `DELETE`, valid PostgreSQL publications send an old-side image:
| REPLICA IDENTITY | `old_table_row` contains for published deletes |
| ------------------------------- | ------------------------------------------------------------ |
| `FULL` | `Some(OldTableRow::Full(row))` |
| `DEFAULT` with a primary key | `Some(OldTableRow::Key(row))` |
| `DEFAULT` without a primary key | Source `DELETE` is rejected when the table publishes deletes |
| `USING INDEX` | `Some(OldTableRow::Key(row))` |
| `NOTHING` | Source `DELETE` is rejected when the table publishes deletes |
**Important implications:**
* Deletes do not carry a new row image. `old_table_row` is the delete payload.
* `OldTableRow::Key(row)` again means replica-identity columns only, not necessarily the table's primary key.
* Consumers that require full old rows for deletes need `REPLICA IDENTITY FULL`.
* The Rust field is optional at the event API boundary, but PostgreSQL pgoutput
populates it for every published delete. Tables without usable replica
identity fail at the source when they publish deletes.
* Destination implementations should decide up front whether key-only deletes
are enough. If they are not, require `REPLICA IDENTITY FULL` for those tables.
### Truncate [#truncate]
One or more tables were truncated (all rows deleted).
```rust
pub struct TruncateEvent {
pub commit_lsn: PgLsn,
pub tx_ordinal: u64,
pub options: i8,
pub truncated_tables: Vec,
}
```
> **Note**
>
> A single `Truncate` event can affect multiple tables when using
> `TRUNCATE ... CASCADE`.
## Transaction Events [#transaction-events]
These events mark transaction boundaries.
### Begin [#begin]
Marks the start of a transaction.
```rust
pub struct BeginEvent {
pub commit_lsn: PgLsn,
pub tx_ordinal: u64,
pub timestamp: i64,
pub xid: u32,
}
```
### Commit [#commit]
Marks successful transaction completion.
```rust
pub struct CommitEvent {
pub commit_lsn: PgLsn,
pub tx_ordinal: u64,
pub flags: i8,
pub end_lsn: PgLsn,
pub timestamp: i64,
}
```
## Schema Events [#schema-events]
### Relation [#relation]
Provides **table schema information**. Sent before data events for a table and
again after supported schema changes.
```rust
pub struct RelationEvent {
pub replicated_table_schema: ReplicatedTableSchema,
}
```
Relation messages are connection-local protocol metadata. PostgreSQL can emit
the same relation again after a reconnect even when the source schema did not
change. A `RelationEvent` therefore has no durable event sequence key and does
not consume a transaction ordinal.
PostgreSQL pgoutput builds relation messages by walking the table descriptor in
`pg_attribute.attnum` order and skipping columns that are not published. It
sends tuple data in the same order as the relation message.
ETL builds replication and identity masks from relation-message column names, so
the order is not needed to decide which columns are included. That name-based
matching is sound because PostgreSQL live column names are unique within a table
schema version. The order matters only after the masks are applied: stored table
schemas are also ordered by `attnum`, so
`ReplicatedTableSchema::column_schemas()` becomes a positional view that matches
the tuple payloads exactly, even when a publication filters columns.
For DDL behavior, including add/drop/rename semantics, default and nullability
handling, and current limitations, see
[Schema Changes](https://supabase.github.io/etl/explanation/schema-changes.md).
## Begin/Commit Behavior [#begincommit-behavior]
During initial sync, parallel Table Sync Workers may deliver `Begin` and
`Commit` markers more than once. Ignore them if you do not need transaction
markers; otherwise deduplicate them by `commit_lsn` plus `tx_ordinal`. Row
writes must still be idempotent because ETL provides at-least-once delivery.
## Understanding Event Sequence Keys [#understanding-event-sequence-keys]
Transaction and data events include a commit **LSN (Log Sequence Number)** plus
a transaction-local ordinal. Together, these fields provide stable ordering for
idempotent destination writes. Relation events are not sequenced because they
are connection-local protocol metadata.
### `commit_lsn` and `tx_ordinal` [#commit_lsn-and-tx_ordinal]
| Field | Meaning | Use Case |
| ------------ | ---------------------------------------------------- | --------------------------------------------- |
| `commit_lsn` | Position where the transaction will commit | Transaction grouping, recovery checkpoints |
| `tx_ordinal` | Zero-based order of the event within its transaction | Ordering and idempotency within a transaction |
Events in one transaction share a `commit_lsn`; `tx_ordinal` distinguishes
their order. Use the pair as a sequence key for ordered writes and replay
detection. See [Logical Replication](https://supabase.github.io/etl/explanation/concepts.md#lsn-log-sequence-number)
for LSN background.
## Event Batching [#event-batching]
ETL batches events before calling `write_events()`. A batch may contain events from **multiple tables, multiple transactions, and mixed event types**.
Schema changes do not end a batch: because batching is driven by size and time,
one batch may contain multiple `Relation` events and multiple schema changes.
**Ordering requirement:** Process events for the same destination row in order.
Events for independent rows may be processed concurrently.
---
# Schema Changes
> How Supabase ETL handles DDL and evolving table schemas.
- Canonical HTML: https://supabase.github.io/etl/explanation/schema-changes/
- Agent-readable Markdown: https://supabase.github.io/etl/explanation/schema-changes.md
- Source: https://github.com/supabase/etl/blob/main/site/content/docs/explanation/schema-changes.mdx
Schema-change support is in public beta and is being expanded incrementally. The
current implementation is intentionally conservative: the source-side event
trigger captures a rich PostgreSQL-shaped snapshot, while ETL currently models
well-understood column changes: **adds, drops, renames, and column default
and nullability changes**, plus publication column-list changes for tables the
running pipeline already tracks. A few known edge cases remain.
Built-in destination support varies by destination DDL capabilities.
**BigQuery, ClickHouse, DuckLake, and Snowflake** apply supported schema changes
automatically; Iceberg is deprecated for new deployments and does not support
schema-change DDL.
## Short Version [#short-version]
For published permanent tables, ETL currently models these `ALTER TABLE`
changes:
| Source change | ETL interpretation |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| Add a replicated column | Add column |
| Drop a replicated column | Drop column |
| Rename a replicated column | Rename column |
| Change a replicated column default | Column default modification |
| Drop a replicated column default | Column default removal |
| Drop `NOT NULL` from a replicated column | BigQuery relaxes an existing `REQUIRED` column to `NULLABLE`; other built-in destinations currently leave nullability unchanged |
| Set `NOT NULL` on a replicated column | Detected in the schema snapshot, but not applied to built-in destinations |
| Several of the above in one statement | One schema snapshot, diffed into column additions, removals, and grouped column modifications |
When several attributes of the same logical column change at once, ETL groups
them into one column change with multiple modifications. For example, renaming a
column and changing its default in one `ALTER TABLE` statement is treated as a
single logical column change.
## How It Works [#how-it-works]
ETL installs a PostgreSQL `ddl_command_end` event trigger named
`supabase_etl_ddl_message_trigger`. When an `ALTER TABLE` statement or supported
`ALTER PUBLICATION` change affects a published permanent table, the trigger
emits a transactional logical message with prefix `supabase_etl_ddl`.
PostgreSQL does not pass user-defined arguments to an event-trigger function.
The function reads `TG_TAG` and the object addresses returned by
`pg_event_trigger_ddl_commands()`, then resolves the post-DDL catalogs:
* For `ALTER TABLE`, the object address identifies the table. The message has
no publication name because a physical table change applies to every
publication containing that table.
* For per-table `ALTER PUBLICATION` changes, the object address identifies a
surviving `pg_publication_rel` row, which links the explicitly named table to
its publication. This path does not expand a named partition root into its
effective leaves.
* For publication-level `ALTER PUBLICATION` changes, the object address
identifies a `pg_publication` row. The trigger does not parse which parameter
changed; it expands the publication's complete post-command effective table
set through `pg_publication_tables`.
The publication name is therefore optional in the payload for table DDL but
required for publication DDL. Logical decoding exposes custom messages to a
slot independently of pgoutput's table filtering, so ETL accepts an
`ALTER PUBLICATION` message only when its resolved name exactly matches the
pipeline's configured publication. A missing name fails closed.
That message is **internal plumbing**. Destinations do not receive it directly.
Instead, ETL:
1. Parses the schema-change message.
2. Stores a new versioned table schema using a composite snapshot ID ordered by
commit LSN and then message LSN.
3. Invalidates the in-memory relation state for that table.
4. Waits for PostgreSQL pgoutput to emit a fresh `RELATION` message before the
next row event for that table.
5. Sends destinations a public `Event::Relation` with the new
`ReplicatedTableSchema`.
Snapshot IDs display both LSNs as decimal unsigned 64-bit values in
`commit_lsn:message_lsn` form. Their ordering is the numeric tuple ordering of
those two components, not the lexical ordering of the displayed string.
`ALTER PUBLICATION ... DROP TABLE` intentionally emits no schema snapshot for
the removed table because it is no longer part of the publication. PostgreSQL
14 may still send an empty `BEGIN`/`COMMIT` pair for that transaction;
PostgreSQL 15 and later suppress empty logical-replication transactions. The
empty pair contains no schema or row event and does not affect snapshot
ordering. See the [PostgreSQL 15 release
notes](https://www.postgresql.org/docs/15/release-15.html) and the [upstream
change](https://github.com/postgres/postgres/commit/d5a9d86d8ffcadc52ff3729cd00fbd83bc38643c).
For a table already known to the pipeline, a supported publication change
creates a new snapshot ID even when the full physical table schema is unchanged.
The following `RELATION` message supplies the current publication and identity
masks. This gives successive mask changes distinct snapshot IDs and preserves
their ordering. Multiple schema messages sharing one commit LSN are ordered by
their message LSN. A newly published table that the running pipeline does not
know is ignored until startup publication reconciliation creates its table
state and initial sync.
### `ALTER PUBLICATION` support boundary [#alter-publication-support-boundary]
The trigger's ability to observe an `ALTER PUBLICATION` command is not the same
as runtime support for that publication change. The trigger emits schema
snapshots; it does not add or remove ETL table state, start an initial sync, or
change which relation OIDs the running apply worker owns.
| Publication change | Trigger behavior | Runtime behavior |
| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| Change the column list of an already tracked table | Emits a snapshot scoped to that table and publication | Supported. The next `RELATION` message installs the new replication mask before following row events. |
| `ADD TABLE` or a membership-changing `SET TABLE` | Emits snapshots for surviving, explicitly identified `pg_publication_rel` rows. A named partition root is not expanded into leaves on this path. | Newly effective relation OIDs are ignored until startup reconciliation initializes and copies them. |
| `DROP TABLE` | Emits no snapshot because the `pg_publication_rel` row is gone at `ddl_command_end`. | Removed table state is purged during startup reconciliation. Destination data is not automatically removed. |
| Add, set, or drop `TABLES IN SCHEMA` | The current trigger does not resolve `pg_publication_namespace` events into table snapshots. | Effective membership is loaded during startup reconciliation. |
| Change a row filter | May emit a per-table snapshot, but row-filter mutation is not part of the supported live-update contract. | Do not rely on changing row filters while the pipeline is running. |
| Change a publication-level setting such as `publish` | A `pg_publication` event expands to one snapshot for every post-command effective table, even if physical schemas did not change. | Publication settings are not a supported live schema-update interface. |
| Change `publish_via_partition_root` | Emits snapshots for the complete post-change effective set: the root when enabled, or the leaves when disabled. | Unsupported while running. The new relation OIDs are not dynamically initialized, and the previous destination relation layout is not migrated. |
| Rename the publication or change its owner | These are publication-level events and may expand across the effective table set. A rename scopes messages to the new name. | Not supported as live schema changes. A configured pipeline does not automatically adopt a new publication name. |
At startup, ETL calls `pg_get_publication_tables()` to load the same effective
relation identities PostgreSQL uses for pgoutput. New identities enter initial
sync and identities no longer returned by PostgreSQL have their ETL state
purged. This startup process does not rename, merge, or delete destination
tables created for an earlier root or leaf identity.
> **Warning**
>
> Only publication column-list changes for already tracked tables are supported
> while a pipeline is running. Other `ALTER PUBLICATION` changes can produce
> skipped events, stale destination tables, or missing or duplicated data. Plan
> a controlled change with source writes paused and the pipeline stopped,
> followed by an explicit restart, table resynchronization, or new pipeline
> instead of relying on live behavior.
The important public boundary is:
```text
... -> internal DDL message -> Relation(new schema) -> Insert/Update/Delete ...
```
Destinations should treat `Event::Relation` as the point where the **active schema changes** for following row events.
`Relation` is an ordered event, not a batch boundary. ETL batches calls to
`write_events()` based on size and time, so a single destination batch may
contain zero, one, or many schema changes, including multiple relation events
for the same table.
## Destination-Specific DDL Behavior [#destination-specific-ddl-behavior]
ETL has one shared schema-change signal, but **DDL behavior is implemented per destination**. A destination may choose to apply DDL automatically, reject a schema change, or require operator handling.
For every built-in destination, a relation whose snapshot ID and replication
mask exactly match the applied destination metadata is idempotent. BigQuery,
ClickHouse, DuckLake, and Snowflake currently reject an older snapshot, or the
same snapshot with a different replication mask, instead of attempting to infer
schema ordering from later row events. A relation has no DML sequence key of its
own, so destination row-replay deduplication does not prove that reverse or
ambiguously ordered DDL is safe. Recovering from either rejection currently
requires resynchronizing the table. This comparison applies only to metadata
already marked `Applied`; it neither defines nor initiates recovery from an
interrupted `Applying` state.
| Destination | Current DDL behavior |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| BigQuery | Supports add, drop, rename, `REQUIRED` to `NULLABLE` relaxation, and supported literal default metadata. BigQuery requires added columns to be nullable and does not backfill existing rows for `ADD COLUMN ... DEFAULT`. PostgreSQL remains responsible for enforcing later `SET NOT NULL` changes because BigQuery cannot tighten an existing column in place. |
| ClickHouse | Supports add, drop, rename, and supported literal defaults. `ReplacingMergeTree` rejects primary-key drops or renames because the ordering expression cannot be rewritten safely. ClickHouse default expressions are metadata-only unless explicitly materialized; ETL does not issue `MATERIALIZE COLUMN`. |
| DuckLake | Supports add, drop, rename, and supported literal defaults. DuckLake records supported add-time defaults as metadata without rewriting existing data files. |
| Snowflake | Supports add, drop, rename, create-table literal defaults, and literal add-column defaults. Literal defaults are included in `ADD COLUMN` so Snowflake can expose add-time default values for existing rows; non-literal defaults and later default changes are skipped with a warning. |
| Iceberg | Deprecated. An identical relation is idempotent, but schema-change DDL is not supported and any newer schema is rejected. Older or ambiguously masked relations are rejected before row writes. |
| Custom destinations | Destination authors decide which `Event::Relation` changes to apply, reject, or handle manually. |
See [Destinations](https://supabase.github.io/etl/reference/destinations.md) for the canonical maturity status
and broader limitations of every built-in implementation.
## Default Backfills [#default-backfills]
When a source table adds a replicated column with a default, PostgreSQL can make
pre-existing source rows read as though they already contain that default. ETL
deliberately avoids **physical destination backfills** for those existing rows.
Built-in destinations avoid operations such as `UPDATE`, `MERGE`, CTAS/swap
rewrites, or ClickHouse `MATERIALIZE COLUMN` because those operations can
rewrite large tables, block replication progress, and create
destination-specific cost spikes.
Instead, destinations apply the schema change in the cheapest safe form they
support:
* BigQuery adds the column as nullable, then sets supported literal default
metadata for future writes.
* ClickHouse may expose default values for pre-existing rows through default
metadata, but ETL does not issue `MATERIALIZE COLUMN`.
* DuckLake uses add-time initial default metadata for supported defaults; its
schema evolution does not rewrite data files.
* Snowflake uses `ADD COLUMN ... DEFAULT` for supported literal defaults.
Snowflake exposes default values for existing rows and does not document this
as a physical row rewrite, but defaults created this way cannot later be
dropped.
As a result, pre-existing destination rows might not match PostgreSQL's
historical `ADD COLUMN ... DEFAULT` view unless the destination has a
metadata-only initial-default mechanism and the source default is supported.
This does **not** mean future replicated tuples lose their default values:
PostgreSQL sends evaluated column values in row data after the relation change,
and ETL writes those values normally. The limitation is only that unsupported
defaults are not installed as destination schema default metadata, and existing
destination rows are not rewritten by ETL.
## Supported Column Defaults [#supported-column-defaults]
Column defaults are **best-effort metadata translations**, not a PostgreSQL
expression evaluator. ETL reads the source default from PostgreSQL's
`pg_get_expr` output, parses only deterministic literal defaults, and asks each
destination whether that parsed default can be rendered safely in that
destination's SQL dialect.
If a default is not in the supported subset, ETL skips the destination default
metadata with a warning. Replication does not fail, and future tuples still
carry evaluated values from PostgreSQL. This is intentional: runtime-generated
defaults can evaluate at different times or under different session settings in
different systems, so installing a similar-looking destination default can create
silent mismatches. ETL can add support for specific additional defaults later
when their semantics can be preserved for the destination.
The shared parser currently recognizes only these source default shapes:
| Source default shape | Examples |
| ---------------------------- | ---------------------------------------------------------------------------- |
| String literals | `'pending'::text`, `('don''t'::text)` |
| Numeric literals | `42`, `-1`, `'42.10'::numeric(10,2)` |
| Boolean literals | `true`, `false`, `'true'::boolean` |
| Date/time/timestamp literals | `'2026-01-01'::date`, `'12:30:00'::time`, `'2026-01-01 12:30:00'::timestamp` |
| JSON literals | `'{}'::jsonb`, `'{"enabled": true}'::json` |
| UUID literals | `'a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11'::uuid` |
The parser is intentionally conservative. These PostgreSQL defaults are
examples of **unsupported** expressions:
```sql
default nextval('users_id_seq')
default 'a' || 'b'
default lower('USER' || '_ID')
default concat('a', 'b')
default md5('x')
default random()
default clock_timestamp()
default now()
default current_timestamp
default current_user
default gen_random_uuid()
default uuid_generate_v4()
default timezone('UTC', now())
default array['a', 'b']
default (select 'x')
default current_setting('app.tenant_id')
default 1e6
default interval '1 day 2 hours'
```
Unsupported defaults are skipped because translating arbitrary PostgreSQL
expressions would require both a PostgreSQL parser and a destination-specific
expression translator. Most destinations do not accept arbitrary PostgreSQL
expressions as column defaults, and even similar-looking SQL can have different
volatility, time zone, type coercion, or evaluation semantics. Skipping the
destination schema default does not drop actual row values emitted by
PostgreSQL.
Destination support may be narrower than parser support:
| Destination | Supported default behavior |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| BigQuery | Supports compatible string, numeric, boolean, date, time, timestamp, JSON, and UUID literals. Added columns are created nullable and supported defaults are set afterward for future writes. |
| ClickHouse | Supports compatible string, numeric, boolean, date, time, timestamp, JSON, and UUID literals. Defaults are metadata only unless separately materialized. |
| DuckLake | Supports compatible string, numeric, date, time, timestamp, JSON, and UUID literals. Boolean defaults are currently skipped by the DuckLake destination. |
| Snowflake | `CREATE TABLE` supports compatible string, numeric, boolean, date, time, timestamp, JSON, and UUID literals. `ADD COLUMN` only receives the literal subset Snowflake allows for add-column defaults: string, numeric, and boolean literals. Later default changes on existing columns are skipped. |
When changing a default from one supported expression to an unsupported
expression, destinations that can safely remove defaults drop the old supported
default to avoid leaving stale destination behavior behind. Snowflake is the
exception for defaults introduced by `ADD COLUMN ... DEFAULT`, because Snowflake
does not allow those defaults to be dropped safely.
## Diff Semantics [#diff-semantics]
ETL stores schemas in **PostgreSQL column ordinal order** (`pg_attribute.attnum`)
and computes destination schema diffs over replicated columns only.
The current diff rules are:
* Same ordinal position, different name: column rename.
* Same ordinal position, different default expression: column default change.
* Same ordinal position, different nullability: detected schema metadata change.
* Old ordinal position missing from the new schema: column drop.
* New ordinal position missing from the old schema: column add.
This matches PostgreSQL's normal behavior for simple column operations:
renames keep the same `attnum`, dropped columns disappear from the visible
schema, and newly added columns receive new ordinal positions.
## Destination Handling [#destination-handling]
Custom destinations should handle schema changes in `write_events()` by
watching for `Event::Relation`.
A practical flow is:
1. Iterate through the batch in order, treating each relation event as a
possible schema transition for that table.
2. Flush any buffered rows/events for the old schema before processing the
relation event.
3. Compare the old destination schema with the relation event's new
`ReplicatedTableSchema`.
4. Mark destination metadata as `Applying` if the destination needs recovery
bookkeeping for the DDL transition.
5. Apply supported destination DDL for adds, drops, renames, and default changes.
6. Mark destination metadata as `Applied` only after the destination schema is
actually ready for following row events.
7. Process following row events with the new schema.
The built-in BigQuery, ClickHouse, DuckLake, and Snowflake destinations follow
this shape: they mark destination schema metadata as `Applying`, apply the
supported DDL operations, then mark the schema as `Applied`. Because destination
DDL is not always transactional, recovery is destination-specific. ClickHouse
and DuckLake can retry an interrupted `Applying` operation. When an arriving
relation drives that retry, it must exactly match the snapshot ID and
replication mask recorded as the target; DuckLake startup recovery can instead
reconstruct that exact target from durable schema state. BigQuery does not
automatically repair `Applying` schema-change metadata, and Snowflake only
automatically retries interrupted initial setup, not an interrupted schema
change. Other interrupted states require resynchronization.
Other destination modules may support a narrower schema-change surface. Treat
`Event::Relation` as the stable ETL contract, then check the destination's
status and implementation before relying on automatic destination DDL.
## Supported Scope [#supported-scope]
The source event trigger intentionally observes a broad schema snapshot, but
ETL currently supports the simplest safe cases:
* `ALTER TABLE ... ADD COLUMN` for replicated columns.
* `ALTER TABLE ... DROP COLUMN` for replicated columns.
* `ALTER TABLE ... RENAME COLUMN` for replicated columns.
* `ALTER TABLE ... ALTER COLUMN ... SET DEFAULT` where the destination supports
setting compatible default metadata.
* `ALTER TABLE ... ALTER COLUMN ... DROP DEFAULT` where the destination supports
removing default metadata safely.
* Multi-subcommand `ALTER TABLE` statements composed of those simple changes.
* Changes to published permanent tables only.
The trigger ignores temporary tables, unpublished tables, generated columns,
dropped-column catalog tombstones, extension-owned DDL, and non-logical-WAL
databases.
## Known Beta Limitations [#known-beta-limitations]
These behaviors are **not full destination DDL semantics** yet:
* Only `ALTER TABLE` and supported `ALTER PUBLICATION` changes are captured by
the ETL DDL trigger today.
* Type changes, constraint changes, identity changes, and replica-identity
changes may be visible in the emitted snapshot, but they are not yet
interpreted as destination DDL operations.
* Table create/drop/rename operations are outside the current schema-change
contract. Publication membership and table cleanup remain separate pipeline
lifecycle concerns.
* Live publication membership, row-filter, operation-setting, publication-name,
and `publish_via_partition_root` changes are unsupported. Although the source
trigger may emit schema snapshots for some of these commands, it does not
dynamically reconcile table ownership or migrate destination table identity.
* If a table-sync worker decodes a DDL or publication-column change during
catch-up but receives no following relation before its handover boundary, it
cannot construct the complete decoder required by `SyncDone`. The DDL
message supplies the physical schema, but only the relation supplies the
exact publication and replica-identity masks for that WAL position. ETL
cannot safely reuse older masks or read newer catalog state, so this
correctness edge case fails the table sync closed. Retry or resynchronize
the table after schema activity has settled.
* A drop and re-add is not treated as a rename. It becomes a drop plus an add
because PostgreSQL assigns a new ordinal position to the new column.
* Destination defaults are best-effort metadata translations. Unsupported
defaults are skipped with a warning instead of failing replication. This does
not remove values PostgreSQL emits in future row events; it only means the
destination schema default metadata is not set. If a previously supported
default becomes unsupported, ETL removes the old destination default where the
destination supports that operation so stale behavior is not left behind.
Snowflake default changes on existing columns are skipped with a warning
because `ALTER COLUMN SET DEFAULT` is documented only for existing sequence
defaults, and defaults introduced by `ALTER TABLE ADD COLUMN ... DEFAULT`
cannot be dropped safely.
* Runtime-generated defaults are intentionally unsupported as destination schema
defaults. Examples include `now()`, `clock_timestamp()`, `gen_random_uuid()`,
`random()`, sequence defaults, and session-dependent expressions such as
`current_user`. Those expressions can evaluate at different times or under
different session settings in each destination. PostgreSQL still sends
evaluated values for future row events, but existing destination rows are not
physically backfilled by ETL. Avoid runtime-generated defaults for replicated
schema changes when exact historical values matter.
* `ADD COLUMN ... DEFAULT` semantics differ by destination. ETL intentionally
avoids destination DDL that rewrites all existing rows. BigQuery leaves
pre-existing destination rows null for newly added defaulted columns, while
ClickHouse, DuckLake, and Snowflake can expose supported add-time defaults
without ETL issuing a materialization rewrite. Snowflake only receives
add-column defaults for source defaults that can be rendered as Snowflake
literals.
* BigQuery applies ongoing-replication `DROP NOT NULL` changes so future source
NULL values remain writable. BigQuery cannot change an existing `NULLABLE` column to
`REQUIRED`, so PostgreSQL enforces later `SET NOT NULL` changes while the
destination column remains nullable. Other built-in destinations currently
leave ongoing-replication nullability changes unchanged. Newly added columns
remain nullable where the destination requires that for historical rows.
* The trigger payload includes `current_query` for debugging only. It can
contain literals and multiple statements, so it must not be treated as
replayable DDL.
* BigQuery, ClickHouse, DuckLake, and Snowflake reject stale or ambiguously
ordered relation schemas instead of rewinding. An older snapshot could drive
reverse DDL that drops newer columns and their data. An equal snapshot with a
different replication mask is also rejected: supported publication
column-list changes always receive a new composite snapshot ID, so the
conflicting masks have no ordering with which to choose a safe winner. The
pipeline fails with a schema-rewind error and the affected table must be
resynchronized.
* Sessions can set `supabase_etl.skip_ddl_log = 'true'` as an emergency
opt-out while recovering a system. DDL executed with that setting enabled is
not logged for ETL.
## Event Ordering [#event-ordering]
Schema changes are transactional logical messages, so they appear in WAL order
relative to row changes. ETL updates its stored schema when it decodes the DDL
message, then waits for the next `RELATION` message to rebuild the runtime
replication and identity masks for row decoding.
This matters for custom destinations: row events after a relation event should
be decoded and written using that relation event's schema. Row events before it
belong to the previous schema.
---
# Extension Points
> Traits you implement to customize Supabase ETL behavior.
- Canonical HTML: https://supabase.github.io/etl/explanation/traits/
- Agent-readable Markdown: https://supabase.github.io/etl/explanation/traits.md
- Source: https://github.com/supabase/etl/blob/main/site/content/docs/explanation/traits.md
Implement these traits to control where replicated data goes and how ETL state
is stored.
## Destination [#destination]
Receives replicated data. This is the **primary extension point** for sending data to custom systems. ETL is **at least once**, so destinations must tolerate duplicate writes and concurrent calls.
```rust
pub trait Destination {
fn name() -> &'static str;
fn shutdown(&self) -> impl Future