Destinations
Official built-in destinations, release status, requirements, and limitations.
Supabase ETL ships official destination implementations in the
etl-destinations crate. Enable only the destination feature you need when
embedding ETL or building the standalone replicator. ClickHouse is the
easiest destination to start with locally: cargo x init starts it, and
cargo x setup replicator configures it by default. BigQuery is the most
mature cloud destination.
| Feature | Destination | Status | Guidance |
|---|---|---|---|
clickhouse | ClickHouse | Public alpha | Recommended local getting-started path |
bigquery | Google BigQuery | Public alpha | Most mature cloud destination |
ducklake | DuckLake | Public alpha | Evaluate for your workload |
snowflake | Snowflake | Public alpha | Evaluate for your workload |
iceberg | Apache Iceberg | Deprecated | Do not use for new deployments |
Status definitions
- Public alpha: The managed service is in public alpha. Features and behavior may change. Local and self-hosted use of the open-source destinations is unrestricted. Review limitations and test recovery and schema changes against your workload before production use.
- Deprecated: Retained for compatibility but no longer recommended for new deployments. Plan to move away from it.
Row value handling
ETL validates structural prerequisites such as primary keys, replica identity, column mappings, and schema transitions. It does not pre-validate each row value against destination-specific domains. When ETL can represent a value in the destination request or wire format, it sends that value and relies on the destination's native coercion, rounding, canonicalization, and rejection behavior. These semantics apply consistently to initial copy, inserts, and updates.
For row values, ETL returns a local encoding error when it cannot construct a faithful wire representation. Known destination differences do not add per-cell runtime validation. Test destination-specific value semantics that matter to your workload.
Publication column additions
An existing column newly exposed by a publication is added as nullable so old
destination rows remain NULL. Defaults and array handling differ by
destination. See Publication column
changes for the
complete policy.
Destination ownership
ETL assumes exclusive ownership of every data-bearing table it creates.
Destination metadata is the authority for whether initial setup or a schema
change completed; process-local caches do not replace that state. Once metadata
is Applied, ETL does not recreate a missing table or repair external schema
changes during ordinary writes or restart. Such changes are unsupported and
typically cause the destination operation to fail. Use the table
resynchronization lifecycle when a destination table must be rebuilt.
ETL may rebuild ephemeral client state and derived non-data-bearing objects, such as Snowflake streaming channels or BigQuery views. Direct destination truncation is also outside ETL's lifecycle: later behavior depends on the destination's offset and write semantics, and previously replicated rows are not restored automatically.
BigQuery
Public alphaThis status applies to the managed service, not local or self-hosted use.
BigQuery uses the BigQuery Storage Write API and BigQuery change data capture to maintain destination tables. Use it when you already have a Google Cloud project. For a local first pipeline, prefer ClickHouse.
Physical table versions and upgrades
Query the stable logical view for each replicated table. Initial copy, recopy,
and source TRUNCATE operations create physical tables with fresh random
version IDs. ETL does not deliberately reuse a deleted physical name:
streaming writes after same-name recreation can be acknowledged without
reaching the replacement table. The active logical view and its physical target
must remain available; older physical tables are no longer needed.
Cleanup is optimistic: after truncate switches the view and persists the new target, ETL attempts background deletion of previously discovered versions. Explicit drop/reset also attempts physical cleanup after removing the view. Listing or physical deletion failures are logged and can leave unused tables behind without blocking replication. A later truncate or reset can try again; there is no cleanup queue or periodic collector. Operators can use the table IDs in warnings to remove unused tables manually, after checking that they are not the current view target or the target recorded in destination metadata. Interrupted creation leaves its abandoned table for these same cleanup paths. Creation, view changes, and metadata persistence still propagate failures.
The suffix is an underscore followed by 25 lowercase letters and digits, encoding a full UUID without relying on the clock. It adds 26 ASCII bytes to the encoded source name. BigQuery permits table names up to 1,024 UTF-8 bytes.
Existing numeric versions remain readable and active after an upgrade. The next truncate or recopy allocates a random version while preserving the logical view name. No metadata database migration or immediate recopy is required. Once random IDs have been persisted, downgrading requires a release that understands this format; older releases that only parse numeric suffixes cannot resume those tables. Do not edit stored IDs to enable a downgrade. If rollback to an older deployment must remain possible, deploy support for reading both formats in that release before deploying random-ID allocation.
Query completion and recovery
ETL waits for BigQuery DDL to finish before advancing destination metadata. Query submission, completion polling, result pages, and retries share a ten-minute deadline. A pending response keeps the original job ID and location. Each poll requests a ten-second server wait; another incomplete response is followed by a one-second pause. The server wait is a hint: HTTP latency can exceed it, and it does not limit the remote job's lifetime.
Transient HTTP failures allow up to 20 retries of the same request. Backoff starts at one second, doubles to a 32-second cap, and adds up to one second of jitter within that cap, following BigQuery's reliability guidance. Only BigQuery's explicit retryable failed-job responses permit resubmission with a new request ID, with at most four replacement jobs. The shared deadline can stop either retry loop sooner. Permanent SQL, permission, and configuration errors require correction.
A local timeout does not cancel the remote job. ETL reports an unconfirmed outcome and requires manual recovery instead of automatically replaying DDL. Use the reported request ID, job ID, and location to investigate the outcome before retrying or resetting the affected table. View changes and metadata writes are separate operations; after an interrupted transition, inspect both before deleting any physical table. Storage Write API propagation retries are separate from these query retries.
Table partitioning and clustering
Use table_options to select the physical layout for individual source
tables. table_options, along with max_staleness_mins, is applied only
when a table is created or recreated, never to a table that already exists.
Restarting a pipeline does not modify an existing table, and changing either
setting has no effect until the table is created again. A table is created,
and so picks up the current settings, when it is replicated for the first
time, when its replication state is reset, or when a source truncate makes
ETL create a new physical table.
destination:
big_query:
# Project, dataset, and authentication fields omitted.
table_options:
tables:
- table_id: 16384
partition_by:
kind: time_column
column: created_at
granularity: day
cluster_by:
- tenant_id
- event_typetable_id is the source PostgreSQL table OID, matching the identifiers used by
table_sync_copy. Find it in your database using the qualified table name:
SELECT 'public.events'::regclass::oid AS table_id;Replace public.events with the table you want to configure. Because the
configuration follows the source table ID rather than
its name, renaming the source table does not detach its physical-layout
settings, and destination metadata can continue pointing it at a different
physical table. If a table is dropped and recreated, PostgreSQL assigns it a
new OID, so its configuration must use the new table_id.
partition_by and cluster_by are independent. Set either one or both in a
table entry: omit cluster_by for a partitioned-only table, or omit
partition_by for a clustered-only table. To use BigQuery's default
unpartitioned and unclustered layout, omit that table from tables. An entry
with neither option is rejected because it has no effect.
Time-column partitioning supports replicated PostgreSQL date, timestamp,
and timestamptz columns with hour, day, month, or year granularity;
date columns do not support hour. Use kind: integer_range with column,
start, end, and interval for an integer column, or
kind: ingestion_time with granularity for ingestion-time partitioning.
Clustering accepts one to four ordered, distinct replicated column names.
BigQuery validates whether clustering column types are supported.
partition_by.column and each cluster_by entry use the source PostgreSQL
column name; ETL applies the same flexible column
names mapping to them as it does to the column
itself, so a partitioning or clustering column does not need to be quoted or
otherwise adjusted for the destination.
See the BigQuery documentation for partition expressions and clustering column requirements.
Flexible column names
BigQuery's flexible column names accept spaces, most punctuation, and non-ASCII Unicode characters. ETL passes any replicated PostgreSQL column name through to BigQuery for table creation, schema changes, and row writes; a column name is no longer required to be an ASCII identifier.
BigQuery documents column names as case-insensitive but does not specify the
Unicode case-folding algorithm it uses for identifier resolution. ETL folds
only ASCII letters when mapping a source column name to its destination name,
matching the destination-name
collision behavior used
elsewhere for BigQuery. Name and name therefore collide locally and are
treated as one destination column, while Ä and ä remain distinct. If
BigQuery's own case-insensitivity considers such a non-ASCII pair equivalent,
the table DDL is rejected instead of ETL silently merging the two columns.
Literal defaults
ETL translates supported PostgreSQL literal defaults when it creates a table
or applies a source default change. Text and JSON defaults preserve emoji and
other supplementary Unicode characters through GoogleSQL's eight-digit Unicode
escapes.
PostgreSQL numeric defaults cast decimal text to BIGNUMERIC, supporting
integers beyond the INT64 range and BigQuery's native rounding of excess
fractional digits. PostgreSQL real defaults retain the source's
single-precision rounding when rendered for BigQuery's FLOAT64 columns.
PostgreSQL evaluates source defaults before replication. Explicit source NULL
values remain NULL in scalar columns even when the destination has a default.
Existing destination default metadata is updated through source schema changes
or table resynchronization; deploying a new replicator alone does not rewrite it.
Limitations
- Every replicated source table needs a primary key, and all primary-key columns must be included in the publication.
- The source primary key must not be
DEFERRABLE. A deferrable key lets one statement move a row into a key before the key's previous row moves out, which BigQuery's per-key change application cannot represent. ETL fails the table during its initial copy. A schema change that makes an existing key deferrable fails instead, and replication of every table stops until you resynchronize this one. - PostgreSQL arrays containing
NULLelements are not supported. - BigQuery stores a top-level PostgreSQL
NULLarray as an empty array, so the two source values are indistinguishable in the destination. - Numeric and JSON values follow BigQuery's native domain, rounding, and
canonicalization behavior. BigQuery may transform or reject values that do
not map exactly to its types.
BIGNUMERICsupports a maximum scale of 38; JSON numbers use BigQuery's integer andFLOAT64domains, so wide PostgreSQL JSON numbers can lose precision. See BigQuery data types. - BigQuery dates and timestamps are limited to years 0001 through 9999, and
TIMEends at23:59:59.999999. PostgreSQL BC dates, temporal infinities, and24:00:00do not have equivalent values in those native BigQuery types. Preserving such values would require a different column mapping and a table resynchronization. - Supported schema changes are applied automatically, but PostgreSQL default expressions and backfill behavior are only supported where they map safely to BigQuery. See Schema Changes.
- Source type changes log a warning and skip destination type DDL. Subsequent
writes use the new source schema, so BigQuery may reject them or interpret
their values differently if the destination column is incompatible.
PostgreSQL data rewrites from casts or
USINGexpressions are not replicated. Resynchronize when a type change needs a different destination column type or transforms existing values. - BigQuery column renames are subject to BigQuery's row-access-policy and metadata limitations. Review BigQuery schema change behavior before renaming columns with tags, aspects, or policies.
ClickHouse
Public alphaThis status applies to the managed service, not local or self-hosted use.
ClickHouse is the local default destination. cargo x init starts ClickHouse
on http://localhost:8123 (etl / etl), and cargo x setup replicator
writes config for that service. It needs no cloud account.
See the Standalone Replicator guide for setup and the ClickHouse destination guide for queries, example flags, and upgrades.
ClickHouse supports a current-state layout with ReplacingMergeTree and an
append-only event log with MergeTree.
engine applies only when ETL creates or recreates a table. ClickHouse cannot
change a table's engine, so if the configured engine differs from an existing
table, ETL fails the write rather than silently keeping the old layout. Drop
and resynchronize the table, or revert engine.
When ETL has no destination metadata for a source table, it creates the
ClickHouse table and, for ReplacingMergeTree, its __current view. If either
name already exists, ETL fails the table instead of reusing an object it cannot
prove it owns. This happens when a table is re-added to the publication, when
ETL state is reset against the same database, when two pipelines share a
database, or when a resync after a rename would copy into another table's
destination. Drop the conflicting object or use another database.
Limitations
- The default
ReplacingMergeTreelayout requires a source primary key and ClickHouse 23.5 or newer. - In either engine, publish every primary-key column if the source has a key.
- In either engine, the source primary key must not be
DEFERRABLE. A deferrable key lets one statement move a row into a key before the key's previous row moves out, which per-key current state cannot represent. ETL fails the table during its initial copy. A schema change that makes an existing key deferrable fails instead, and replication of every table stops until you resynchronize this one. - For updates and deletes on tables with a primary key, replica identity must
be
DEFAULT,FULL, orUSING INDEXwith the same identity columns as the primary key. ETL rejects other index identities. WithNOTHING, Postgres rejects published updates and deletes before they reach ETL. A primary-key-changing update writes a tombstone for the old key followed by the new row. MergeTreekeeps an append-only event log; current state needs alimit 1 byquery overcdc_lsnandcdc_tx_ordinal.- A replicated
TRUNCATEclears the destination table in either engine, includingMergeTreeevent history. - A source type change that changes the mapped ClickHouse type, such as
integertobigint, fails the schema change before ETL alters anything, as doestimestamptotimestamptz, which shares a ClickHouse type but converts existing values through the session time zone. Replication stops until you resynchronize the table. Changes that keep the ClickHouse type, such as a longervarchar, continue without DDL. Every insert names its columns and types, so ClickHouse also rejects writes to a column whose type changed outside ETL. - PostgreSQL
timestampandtimestamptzcolumns, and their arrays, map toDateTime64(6, 'UTC'). Atimestampvalue reads back as its source wall-clock value in any server or session time zone. Tables created by earlier ETL versions useDateTime64(6)fortimestampand fail with a column type mismatch; follow the upgrade steps in the ClickHouse destination guide. - ClickHouse
Date32andDateTime64columns support 1900-01-01 through 2299-12-31. ETL writes dates and timestamps outside that range exactly, and ClickHouse stores them, but reads them back clamped to the nearest limit, without an error. For example,1800-06-15reads back as1900-01-01. Filters match neither the source value nor the clamped one. Keep replicated dates and timestamps inside this range. - ClickHouse
Arraycolumns are non-nullable. As with BigQuery, ETL stores a top-level PostgreSQLNULLarray as an empty array, so the two source values are indistinguishable in the destination.NULLarray elements are preserved. - ETL sends any value RowBinary can encode; ClickHouse applies its own conversion and rejection rules.
- You may add
MATERIALIZED,ALIAS, orEPHEMERALcolumns to a table ETL created; ETL ignores them when writing. A resynchronization recreates the table without them. - Dotted nested subcolumns can be renamed only within the same parent, matching
ClickHouse's documented
RENAME COLUMNlimitation. - Tombstone cleanup and
OPTIMIZE ... FINAL CLEANUPare operator-managed. Follow ClickHouse's cleanup requirements to avoid resurrecting deleted rows.
DuckLake
Public alphaThis status applies to the managed service, not local or self-hosted use.
DuckLake writes through DuckDB to a file or PostgreSQL catalog and local or object storage.
Limitations
- Data storage URLs currently support
file,s3, andgsschemes. - Deployments must account for the required DuckDB extensions and the catalog, storage, and maintenance services they configure.
- Primary-key sorting requires source tables to have a primary key; other sorting modes do not add that requirement.
- DuckLake adds streaming columns as nullable. Supported literal defaults are
stored as metadata; ETL does not emit
DEFAULT(NULL)when no source default exists, although a catalog client may display an omitted default asNULL. - On
REPLICA IDENTITY FULLtables without a primary key, or with aDEFERRABLEone, each source update or delete changes one matching DuckLake row, so duplicate rows stay in sync. ETL runs one statement per changed row, so bulk deletes on these tables are slower. If such a table has a column namedrowid(in any case), the column hides DuckDB's row id, and an update or delete changes every matching row.
Snowflake
Public alphaThis status applies to the managed service, not local or self-hosted use.
Snowflake uses direct Snowpipe Streaming and key-pair authentication. ETL stores inserts, updates and deletes as CDC records. For tables meeting the requirements below, ETL creates a companion view exposing current state. The raw table remains available as event history.
Partial updates and materialization
Snowflake supports complete rows and partial updates. With a supported primary
key and REPLICA IDENTITY DEFAULT, Postgres can omit unchanged large values
stored using TOAST. ETL records which columns were omitted so a query can carry
forward their last supplied values. With REPLICA IDENTITY FULL, ETL writes
complete update rows.
Choosing replica identity and a query
Choose based on how much data changes and how often you query current state:
| Source setting | Stored updates | Query choice and trade-off |
|---|---|---|
REPLICA IDENTITY DEFAULT with a supported primary key | Complete rows and partial updates when Postgres omits unchanged TOAST values. | Use the materialization query. Omitting large unchanged values reduces the payload sent to Snowflake, while resolving individual columns requires more query work. |
REPLICA IDENTITY FULL with a stable primary key | Complete update rows, including unchanged values. | For a history of complete insert/update records, use a latest-event-per-key query. It avoids per-column value resolution, at the cost of sending more source data and repeatedly writing unchanged values. The materialization query also works. |
For tables with large values that change infrequently, start with DEFAULT and the materialization query, subject to the requirements below. If rows are small or current state is queried frequently, evaluate FULL with latest-event selection. Measure ingestion and query costs together using representative history; reduced payload size alone does not establish a lower total bill. Query cost also depends on warehouse size and running time.
A latest-event-per-key query selects the event with the greatest
_cdc_sequence_number for each primary key, then excludes keys whose selected
event is a delete. Use it only when all relevant insert/update history is
complete and the key and history requirements below hold. For mixed or partial
history, use the materialization query: taking just the latest event would treat omitted
values as NULL. Changing replica identity to FULL does not fill in earlier
partial records.
Missing values and materialization
Each CDC record includes _cdc_missing_columns ARRAY. An omitted value is stored
as a NULL placeholder with its PostgreSQL attribute number (pg_attribute.attnum)
in this array. These IDs remain stable across renames; dropping a column does
not make its ID available for reuse within the same table. They are not row
indexes or Snowflake column positions. An explicitly supplied
SQL NULL has no entry; it replaces the previous value. Complete records have an
empty array. Changed large values and initial copy rows must fit the compressed
request limit.
Query the automatically created <physical_table>_CURRENT view:
select * from PUBLIC_ORDERS_CURRENT;This ordinary view runs the materialization query when read. It does not store results or run a background refresh, and it requires a Snowflake warehouse. ETL updates its definition after supported schema changes. No PostgreSQL access, ETL binary, or manual query generation is needed.
For custom SQL, read the column-ID and primary-key mapping entirely in Snowflake:
select column_name,
try_parse_json(comment):column_id::integer as column_id,
try_parse_json(comment):primary_key_position::integer as primary_key_position
from ETL_DB.information_schema.columns
where table_schema = 'PUBLIC'
and table_name = 'PUBLIC_ORDERS_CURRENT'
and try_parse_json(comment):column_id is not null
order by ordinal_position;Replace the example database, schema, and view identifiers. Source-column comments
contain JSON with column_id and nullable primary_key_position. Diagnostic
columns have descriptive comments instead. Use GET_DDL('VIEW', 'ETL_DB.PUBLIC.PUBLIC_ORDERS_CURRENT') to inspect the generated SQL. Refresh
customer-owned SQL after DDL; Snowflake column positions cannot replace these IDs.
The Rust etl_destinations::snowflake::materialization_query API is also available
as a query-generation helper.
The query carries forward omitted values, preserves explicit NULLs and respects
delete/reinsert boundaries. It returns _cdc_anchored,
_cdc_unresolved_columns, and _cdc_last_sequence alongside source columns.
An unanchored row or unresolved columns indicates incomplete history; these
flags cannot detect arbitrary publication gaps or removed intermediate events.
The materialization query scans retained history for insert/delete boundaries, then resolves values from each key's latest complete record and later partial updates. This limits per-column work when complete records are available, but the initial history scan remains. Retain the stored events; this query does not make history pruning safe.
Requirements for materializing CDC history
- A stable, nondeferrable primary key with every component replicated. Supported
key types are
smallint,integer,bigint,oid,boolean,text,varchar, PostgreSQL"char",char(n),name,uuid,bytea,numeric,date,timestamp, andtimestamptz. Other key types need separate validation. - Stable key values and representation. Column renames and drops are supported;
ETL refreshes the companion view after applying DDL. Regenerate customer-owned
materialization queries against the updated mapping.
The materialization query reserves names beginning with
_cdc_. The writer reserves its three physical CDC column names. - Complete, uninterrupted publication coverage with unchanged actions, column lists and membership, and no row filters, including while stopped.
- A coherent initial snapshot and all accepted history from that generation. Do not prune, manually edit or combine generations.
Partial updates enforce the supported key structure and reject a delivered key change. Complete-row ingestion accepts a broader set of source schemas and schema changes. Successfully storing an event does not establish that its history meets the requirements for materializing CDC history. Publication changes are an operator responsibility; ETL cannot detect events Postgres never emits.
pgoutput announces a changed relation with the next row event; DDL alone does not update the destination or its view.
Omission metadata keeps its meaning after a rename or drop, including renaming
a key without changing its value. _cdc_unresolved_columns reports current
column names. Dropping and re-adding a name creates a new column with a new ID.
Added columns retain the destination's existing default-value limitations: only
supported literal defaults populate historical records. This feature does not
add support for arbitrary defaults or type changes. See
Schema changes for the existing DDL behavior.
After incompatible key/schema/publication changes or lost history, request a full resync through the supported reset operation: recreate the owned destination history from a new coherent snapshot. Recreate the pipeline if per-table resync is unavailable. Standalone deployments without reset support should use fresh ETL state and a fresh destination schema. A normal restart does not request a resync; manual truncation is not a substitute.
Companion-view permissions and lifecycle
The ETL role needs CREATE VIEW on the destination schema and ownership of its
raw tables and views. For example:
grant create view on schema ETL_DB.PUBLIC to role ETL_ROLE;Grant this before starting or upgrading the replicator. ETL does not change role
grants automatically. Missing permissions or conflicting objects fail setup or
startup reconciliation. Consumers need SELECT on the view and the usual
usage privileges for its database, schema, and query warehouse.
ETL owns the _CURRENT and _CURRENT_PENDING companion names and their comments.
For physical names longer than 239 characters, the view name is
_ETL_<uppercase SHA-256 of the physical name>_CURRENT; its recovery name adds
_PENDING.
The second name is an internal recovery object, not a supported query interface.
Before a drop or rename, the public view becomes temporarily unavailable; it is
restored after the raw schema and new definition are ready. Add-only changes
replace the definition without taking the public name offline. Startup
reconciles views for tables that completed initial synchronization, including
idle tables. Tables awaiting a fresh copy and errored tables are skipped.
Interrupted raw DDL follows the existing schema recovery/resync requirements.
Explicit view grants survive definition replacement and same-name resync. Do not attach custom policies, tags, or streams to ETL-owned views or change their definitions/comments. Create a separate customer-owned view for custom behavior. Replacing an ETL view makes streams on it stale. During initial copy or resync, the view may expose a snapshot that is still being copied.
Complete-row ingestion for keyless tables, unsupported identities/key types, or
source names beginning with _cdc_ remains raw-only and emits a warning. Losing
eligibility after a view has been created makes it unavailable and fails
reconciliation. Fresh pipeline state requires an unused destination namespace
or deliberate cleanup of old tables and both reserved companion names.
Resynchronizing after a source table rename
A source rename preserves the recorded destination table name during streaming. Resynchronization drops that recorded table, then creates a fresh destination from the current source name. If that name is already occupied, ETL fails the copy instead of writing into another table's destination.
For example, suppose public.orders is renamed to public.ordersold, and
public.ordersnew is renamed to public.orders. The first table still owns
PUBLIC_ORDERS, so resynchronizing the second table fails when it tries to
create PUBLIC_ORDERS.
To recover, resynchronize public.ordersold first and wait for it to finish.
Provided PUBLIC_ORDERSOLD is available, this rebuilds the first table there
and frees PUBLIC_ORDERS. Then retry resynchronizing public.orders.
Resynchronization deletes previous destination data and CDC history. Existing
source rows are copied again only when initial copying is enabled. Update any
queries that depend on the first table's previous destination name. ETL removes
the old companion view when resync moves to another name. Set up consumer grants
and references for the new view.
If the existing destination must remain unchanged, use a new pipeline targeting an unused destination schema. Do not manually delete or rename a destination that another replicated table still uses.
Limitations
- Setup requires a Snowflake user and role with the documented warehouse, database, and schema privileges.
- The effective
QUOTED_IDENTIFIERS_IGNORE_CASEparameter must beFALSE. Connectivity validation rejects other settings because ETL preserves quoted source column names exactly. See Snowflake identifier resolution. - Schema evolution support is still in progress. Only defaults that ETL can translate safely at table creation or column addition are applied. Later existing-column default changes are skipped because Snowflake default provenance cannot be inferred safely.
jsonandjsonbuse nativeVARIANTstorage and preserve SQLNULLseparately from JSONnull. ETL omits destination defaults for these columns and removes existing defaults before opening or refreshing their streaming channels. PostgreSQL still applies source defaults before replication. If a destination default cannot be removed, channel setup fails before writing. Existing destination columns mapped fromjsonorjsonbmust have physical typeVARIANT; incompatible types are rejected during channel setup.- PostgreSQL
json[]andjsonb[]arrays containing SQLNULLelements are rejected during encoding because the ingestion format cannot distinguish them from JSONnullelements. JSONnullelements, empty arrays, and a whole SQLNULLarray are supported. - Earlier versions could merge SQL
NULLand JSONnull. The encoding fix does not repair existing rows; restoring that distinction requires the source data or a fresh copy. - Validate channel recovery, committed offsets, and account-specific resource limits before production use.
Existing destination tables
For ETL-owned tables using the supported previous CDC format, ETL validates the
column set and CDC metadata, adds the nullable _cdc_missing_columns ARRAY
column, and validates the result before opening the write channel. Existing
insert/update records have NULL metadata, interpreted as complete within their
replicated schema at event time. Historical deletes remain deletion boundaries.
The upgrade preserves stored history and replication offsets and requires no
backfill.
The configured Snowflake role must allow the ALTER. Unexpected schemas and
metadata types remain errors. Customer-owned queries must support partial
records before consuming them; automatic table upgrades do not update those
queries. The materialization query requires _cdc_missing_columns to exist.
Iceberg
DeprecatedThe Apache Iceberg implementation is retained for compatibility and is not recommended for new deployments.
Limitations
- Schema-change DDL is not supported. A newer relation schema is rejected even when only a default, nullability, mapped-equivalent type, or publication mask changed. Existing tables must match the exact generated Iceberg schema.
- In
OnePerSchemamode, source schema names are normalized into Iceberg namespace names. The destination rejects different source schemas that map to the same namespace within a pipeline, including during restart and recovery. This check does not detect collisions with other pipelines or orphaned tables. - Apache Iceberg supports native schema evolution, but this deprecated ETL adapter does not implement it and does not translate PostgreSQL defaults.
- The destination is deprecated and is not receiving the same product investment as the active destination implementations.
Custom destinations and upstream support
Adding an official destination is a long-term maintenance commitment. It requires durable and idempotent writes, schema-evolution behavior, restart and recovery coverage, integration infrastructure, credential handling, ongoing dependency updates, and operational support. For that reason, maintainers are careful about accepting new destination implementations upstream. Open an issue or discussion before investing in an upstream implementation; acceptance is not guaranteed.
You do not need to contribute a destination upstream to use ETL. Implement the
Destination trait in your own Rust
project, wire it into Pipeline::new, and maintain it alongside your
application. See Custom Implementations for
a complete example.