Destinations

Supabase ETL ships official destination implementations in the etl-destinations crate. Enable only the destination feature you need when embedding ETL or building the standalone replicator. ClickHouse is the easiest destination to start with locally: cargo x init starts it, and cargo x setup replicator configures it by default. BigQuery is the most mature cloud destination.

Status definitions

• Public alpha: The managed service is in public alpha. Features and behavior may change. Local and self-hosted use of the open-source destinations is unrestricted. Review limitations and test recovery and schema changes against your workload before production use.
• Deprecated: Retained for compatibility but no longer recommended for new deployments. Plan to move away from it.

Row value handling

ETL validates structural prerequisites such as primary keys, replica identity, column mappings, and schema transitions. It does not pre-validate each row value against destination-specific domains. When ETL can represent a value in the destination request or wire format, it sends that value and relies on the destination's native coercion, rounding, canonicalization, and rejection behavior. These semantics apply consistently to initial copy, inserts, and updates.

For row values, ETL returns a local encoding error when it cannot construct a faithful wire representation. Known destination differences do not add per-cell runtime validation. Test destination-specific value semantics that matter to your workload.

Publication column additions

An existing column newly exposed by a publication is added as nullable so old destination rows remain NULL. Defaults and array handling differ by destination. See Publication column changes for the complete policy.

Destination ownership

ETL assumes exclusive ownership of every data-bearing table it creates. Destination metadata is the authority for whether initial setup or a schema change completed; process-local caches do not replace that state. Once metadata is Applied, ETL does not recreate a missing table or repair external schema changes during ordinary writes or restart. Such changes are unsupported and typically cause the destination operation to fail. Use the table resynchronization lifecycle when a destination table must be rebuilt.

ETL may rebuild ephemeral client state and derived non-data-bearing objects, such as Snowflake streaming channels or BigQuery views. Direct destination truncation is also outside ETL's lifecycle: later behavior depends on the destination's offset and write semantics, and previously replicated rows are not restored automatically.

BigQuery

This status applies to the managed service, not local or self-hosted use.

BigQuery uses the BigQuery Storage Write API and BigQuery change data capture to maintain destination tables. Use it when you already have a Google Cloud project. For a local first pipeline, prefer ClickHouse.

Physical table versions and upgrades

Query the stable logical view for each replicated table. Initial copy, recopy, and source TRUNCATE operations create physical tables with fresh random version IDs. ETL does not deliberately reuse a deleted physical name: streaming writes after same-name recreation can be acknowledged without reaching the replacement table. The active logical view and its physical target must remain available; older physical tables are no longer needed.

Cleanup is optimistic: after truncate switches the view and persists the new target, ETL attempts background deletion of previously discovered versions. Explicit drop/reset also attempts physical cleanup after removing the view. Listing or physical deletion failures are logged and can leave unused tables behind without blocking replication. A later truncate or reset can try again; there is no cleanup queue or periodic collector. Operators can use the table IDs in warnings to remove unused tables manually, after checking that they are not the current view target or the target recorded in destination metadata. Interrupted creation leaves its abandoned table for these same cleanup paths. Creation, view changes, and metadata persistence still propagate failures.

The suffix is an underscore followed by 25 lowercase letters and digits, encoding a full UUID without relying on the clock. It adds 26 ASCII bytes to the encoded source name. BigQuery permits table names up to 1,024 UTF-8 bytes.

Existing numeric versions remain readable and active after an upgrade. The next truncate or recopy allocates a random version while preserving the logical view name. No metadata database migration or immediate recopy is required. Once random IDs have been persisted, downgrading requires a release that understands this format; older releases that only parse numeric suffixes cannot resume those tables. Do not edit stored IDs to enable a downgrade. If rollback to an older deployment must remain possible, deploy support for reading both formats in that release before deploying random-ID allocation.

Query completion and recovery

ETL waits for BigQuery DDL to finish before advancing destination metadata. Query submission, completion polling, result pages, and retries share a ten-minute deadline. A pending response keeps the original job ID and location. Each poll requests a ten-second server wait; another incomplete response is followed by a one-second pause. The server wait is a hint: HTTP latency can exceed it, and it does not limit the remote job's lifetime.

Transient HTTP failures allow up to 20 retries of the same request. Backoff starts at one second, doubles to a 32-second cap, and adds up to one second of jitter within that cap, following BigQuery's reliability guidance. Only BigQuery's explicit retryable failed-job responses permit resubmission with a new request ID, with at most four replacement jobs. The shared deadline can stop either retry loop sooner. Permanent SQL, permission, and configuration errors require correction.

A local timeout does not cancel the remote job. ETL reports an unconfirmed outcome and requires manual recovery instead of automatically replaying DDL. Use the reported request ID, job ID, and location to investigate the outcome before retrying or resetting the affected table. View changes and metadata writes are separate operations; after an interrupted transition, inspect both before deleting any physical table. Storage Write API propagation retries are separate from these query retries.

Table partitioning and clustering

Use table_options to select the physical layout for individual source tables. table_options, along with max_staleness_mins, is applied only when a table is created or recreated, never to a table that already exists. Restarting a pipeline does not modify an existing table, and changing either setting has no effect until the table is created again. A table is created, and so picks up the current settings, when it is replicated for the first time, when its replication state is reset, or when a source truncate makes ETL create a new physical table.

destination:
  big_query:
    # Project, dataset, and authentication fields omitted.
    table_options:
      tables:
        - table_id: 16384
          partition_by:
            kind: time_column
            column: created_at
            granularity: day
          cluster_by:
            - tenant_id
            - event_type

table_id is the source PostgreSQL table OID, matching the identifiers used by table_sync_copy. Find it in your database using the qualified table name:

SELECT 'public.events'::regclass::oid AS table_id;

Replace public.events with the table you want to configure. Because the configuration follows the source table ID rather than its name, renaming the source table does not detach its physical-layout settings, and destination metadata can continue pointing it at a different physical table. If a table is dropped and recreated, PostgreSQL assigns it a new OID, so its configuration must use the new table_id.

partition_by and cluster_by are independent. Set either one or both in a table entry: omit cluster_by for a partitioned-only table, or omit partition_by for a clustered-only table. To use BigQuery's default unpartitioned and unclustered layout, omit that table from tables. An entry with neither option is rejected because it has no effect.

Time-column partitioning supports replicated PostgreSQL date, timestamp, and timestamptz columns with hour, day, month, or year granularity; date columns do not support hour. Use kind: integer_range with column, start, end, and interval for an integer column, or kind: ingestion_time with granularity for ingestion-time partitioning. Clustering accepts one to four ordered, distinct replicated column names. BigQuery validates whether clustering column types are supported.

partition_by.column and each cluster_by entry use the source PostgreSQL column name; ETL applies the same flexible column names mapping to them as it does to the column itself, so a partitioning or clustering column does not need to be quoted or otherwise adjusted for the destination.

See the BigQuery documentation for partition expressions and clustering column requirements.

Flexible column names

BigQuery's flexible column names accept spaces, most punctuation, and non-ASCII Unicode characters. ETL passes any replicated PostgreSQL column name through to BigQuery for table creation, schema changes, and row writes; a column name is no longer required to be an ASCII identifier.

BigQuery documents column names as case-insensitive but does not specify the Unicode case-folding algorithm it uses for identifier resolution. ETL folds only ASCII letters when mapping a source column name to its destination name, matching the destination-name collision behavior used elsewhere for BigQuery. Name and name therefore collide locally and are treated as one destination column, while Ä and ä remain distinct. If BigQuery's own case-insensitivity considers such a non-ASCII pair equivalent, the table DDL is rejected instead of ETL silently merging the two columns.

Limitations

• Every replicated source table needs a primary key, and all primary-key columns must be included in the publication.
• PostgreSQL arrays containing NULL elements are not supported.
• BigQuery stores a top-level PostgreSQL NULL array as an empty array, so the two source values are indistinguishable in the destination.
• Numeric and JSON values follow BigQuery's native domain, rounding, and canonicalization behavior. BigQuery may transform or reject values that do not map exactly to its types.
• Supported schema changes are applied automatically, but PostgreSQL default expressions and backfill behavior are only supported where they map safely to BigQuery. See Schema Changes.
• BigQuery column renames are subject to BigQuery's row-access-policy and metadata limitations. Review BigQuery schema change behavior before renaming columns with tags, aspects, or policies.

ClickHouse

This status applies to the managed service, not local or self-hosted use.

ClickHouse is the local default destination. cargo x init starts ClickHouse on http://localhost:8123 (etl / etl), and cargo x setup replicator writes config for that service. It needs no cloud account.

See the Standalone Replicator guide for setup and the ClickHouse destination guide for queries, example flags, and upgrades.

ClickHouse supports a current-state layout with ReplacingMergeTree and an append-only event log with MergeTree.

engine applies only when ETL creates or recreates a table. ClickHouse cannot change a table's engine, so if the configured engine differs from an existing table, ETL fails the write rather than silently keeping the old layout. Drop and resynchronize the table, or revert engine.

When ETL has no destination metadata for a source table, it creates the ClickHouse table and, for ReplacingMergeTree, its __current view. If either name already exists, ETL fails the table instead of reusing an object it cannot prove it owns. This happens when a table is re-added to the publication, when ETL state is reset against the same database, when two pipelines share a database, or when a resync after a rename would copy into another table's destination. Drop the conflicting object or use another database.

Limitations

• The default ReplacingMergeTree layout requires a source primary key and ClickHouse 23.5 or newer.
• In either engine, publish every primary-key column if the source has a key.
• In either engine, the source primary key must not be DEFERRABLE. A deferrable key lets one statement move a row into a key before the key's previous row moves out, which per-key current state cannot represent. ETL fails the table during its initial copy. A schema change that makes an existing key deferrable fails instead, and replication of every table stops until you resynchronize this one.
• For updates and deletes on tables with a primary key, replica identity must be DEFAULT, FULL, or USING INDEX with the same identity columns as the primary key. ETL rejects other index identities. With NOTHING, Postgres rejects published updates and deletes before they reach ETL. A primary-key-changing update writes a tombstone for the old key followed by the new row.
• MergeTree keeps an append-only event log; current state needs a limit 1 by query over cdc_lsn and cdc_tx_ordinal.
• A replicated TRUNCATE clears the destination table in either engine, including MergeTree event history.
• A source type change that changes the mapped ClickHouse type, such as integer to bigint, fails the schema change before ETL alters anything, as does timestamp to timestamptz, which shares a ClickHouse type but converts existing values through the session time zone. Replication stops until you resynchronize the table. Changes that keep the ClickHouse type, such as a longer varchar, continue without DDL. Every insert names its columns and types, so ClickHouse also rejects writes to a column whose type changed outside ETL.
• PostgreSQL timestamp and timestamptz columns, and their arrays, map to DateTime64(6, 'UTC'). A timestamp value reads back as its source wall-clock value in any server or session time zone. Tables created by earlier ETL versions use DateTime64(6) for timestamp and fail with a column type mismatch; follow the upgrade steps in the ClickHouse destination guide.
• ClickHouse Date32 and DateTime64 columns support 1900-01-01 through 2299-12-31. ETL writes dates and timestamps outside that range exactly, and ClickHouse stores them, but reads them back clamped to the nearest limit, without an error. For example, 1800-06-15 reads back as 1900-01-01. Filters match neither the source value nor the clamped one. Keep replicated dates and timestamps inside this range.
• ClickHouse Array columns are non-nullable. As with BigQuery, ETL stores a top-level PostgreSQL NULL array as an empty array, so the two source values are indistinguishable in the destination. NULL array elements are preserved.
• ETL sends any value RowBinary can encode; ClickHouse applies its own conversion and rejection rules.
• You may add MATERIALIZED, ALIAS, or EPHEMERAL columns to a table ETL created; ETL ignores them when writing. A resynchronization recreates the table without them.
• Dotted nested subcolumns can be renamed only within the same parent, matching ClickHouse's documented RENAME COLUMN limitation.
• Tombstone cleanup and OPTIMIZE ... FINAL CLEANUP are operator-managed. Follow ClickHouse's cleanup requirements to avoid resurrecting deleted rows.

DuckLake

This status applies to the managed service, not local or self-hosted use.

DuckLake writes through DuckDB to a file or PostgreSQL catalog and local or object storage.

Limitations

• Data storage URLs currently support file, s3, and gs schemes.
• Deployments must account for the required DuckDB extensions and the catalog, storage, and maintenance services they configure.
• Primary-key sorting requires source tables to have a primary key; other sorting modes do not add that requirement.
• DuckLake adds streaming columns as nullable. Supported literal defaults are stored as metadata; ETL does not emit DEFAULT(NULL) when no source default exists, although a catalog client may display an omitted default as NULL.

Snowflake

This status applies to the managed service, not local or self-hosted use.

Snowflake uses direct Snowpipe Streaming and key-pair authentication.

Resynchronizing after a source table rename

A source rename preserves the recorded destination table name during streaming. Resynchronization drops that recorded table, then creates a fresh destination from the current source name. If that name is already occupied, ETL fails the copy instead of writing into another table's destination.

For example, suppose public.orders is renamed to public.ordersold, and public.ordersnew is renamed to public.orders. The first table still owns PUBLIC_ORDERS, so resynchronizing the second table fails when it tries to create PUBLIC_ORDERS.

To recover, resynchronize public.ordersold first and wait for it to finish. Provided PUBLIC_ORDERSOLD is available, this rebuilds the first table there and frees PUBLIC_ORDERS. Then retry resynchronizing public.orders. Resynchronization deletes previous destination data and CDC history. Existing source rows are copied again only when initial copying is enabled. Update any queries that depend on the first table's previous destination name.

If the existing destination must remain unchanged, use a new pipeline targeting an unused destination schema. Do not manually delete or rename a destination that another replicated table still uses.

Limitations

• Setup requires a Snowflake user and role with the documented warehouse, database, and schema privileges.
• The effective QUOTED_IDENTIFIERS_IGNORE_CASE parameter must be FALSE. Connectivity validation rejects other settings because ETL preserves quoted source column names exactly. See Snowflake identifier resolution.
• Schema evolution support is still in progress. Only defaults that ETL can translate safely at table creation or column addition are applied. Later existing-column default changes are skipped because Snowflake default provenance cannot be inferred safely.
• json and jsonb use native VARIANT storage and preserve SQL NULL separately from JSON null. ETL omits destination defaults for these columns and removes existing defaults before opening or refreshing their streaming channels. PostgreSQL still applies source defaults before replication. If a destination default cannot be removed, channel setup fails before writing. Existing destination columns mapped from json or jsonb must have physical type VARIANT; incompatible types are rejected during channel setup.
• PostgreSQL json[] and jsonb[] arrays containing SQL NULL elements are rejected during encoding because the ingestion format cannot distinguish them from JSON null elements. JSON null elements, empty arrays, and a whole SQL NULL array are supported.
• Earlier versions could merge SQL NULL and JSON null. The encoding fix does not repair existing rows; restoring that distinction requires the source data or a fresh copy.
• Validate channel recovery, committed offsets, and account-specific resource limits before production use.

Iceberg

The Apache Iceberg implementation is retained for compatibility and is not recommended for new deployments.

Limitations

• Schema-change DDL is not supported. A newer relation schema is rejected even when only a default, nullability, mapped-equivalent type, or publication mask changed. Existing tables must match the exact generated Iceberg schema.
• Apache Iceberg supports native schema evolution, but this deprecated ETL adapter does not implement it and does not translate PostgreSQL defaults.
• The destination is deprecated and is not receiving the same product investment as the active destination implementations.

Custom destinations and upstream support

Adding an official destination is a long-term maintenance commitment. It requires durable and idempotent writes, schema-evolution behavior, restart and recovery coverage, integration infrastructure, credential handling, ongoing dependency updates, and operational support. For that reason, maintainers are careful about accepting new destination implementations upstream. Open an issue or discussion before investing in an upstream implementation; acceptance is not guaranteed.

You do not need to contribute a destination upstream to use ETL. Implement the Destination trait in your own Rust project, wire it into Pipeline::new, and maintain it alongside your application. See Custom Implementations for a complete example.