Standalone Replicator

The standalone replicator is the ready-made etl-replicator application for running Supabase ETL without embedding the library in another Rust program.

What the replicator is

The replicator packages one ETL pipeline, a Postgres-backed state store, and one configured built-in destination into a long-lived process. It loads the source, destination, batching, retry, and operational settings from files and environment variables, starts replication, and handles graceful shutdown. It does not require Kubernetes or the managed Supabase product.

Use the standalone replicator when a built-in destination and the standard Postgres-backed store fit your deployment. Embed the etl crate instead when you need a custom destination, a custom store, or application-specific runtime orchestration. See First Pipeline for the library path.

Fastest local start

ClickHouse is the default destination. cargo x init starts Postgres and ClickHouse; no cloud account is required.

git clone https://github.com/supabase/etl.git
cd etl
cargo x init
cargo x setup replicator
cargo x seed
cargo x run replicator

That writes gitignored config under crates/etl-replicator/configuration/, seeds a publication named seed_pub, and starts replication into local ClickHouse at http://localhost:8123 (etl / etl). Seed tables appear as public_users, public_orders, and public_events, plus __current views:

curl -sS 'http://localhost:8123/?user=etl&password=etl' \
  --data-binary 'SELECT count() FROM "public_users__current"'

Stop the replicator with Ctrl+C.

The rest of this guide shows how to build the binary yourself and point it at ClickHouse with your own configuration directory. Other built-in destinations use the same binary with a different --features flag; see Destinations.

Prerequisites

• Rust 1.95.0, as pinned by rust-toolchain.toml.
• PostgreSQL 14 through 18 configured for logical replication, including a replication user and publication. Complete Configure Postgres first, or use the local Postgres from cargo x init.
• ClickHouse 23.5 or newer. Local Docker from cargo x init is enough.

PostgreSQL 15 or newer is recommended for column and row publication filters, and PostgreSQL 16 or newer is required when replication connects to a physical read replica.

Build the binary

Clone the repository and build only the destination features you need. ClickHouse is the local default:

git clone https://github.com/supabase/etl.git
cd etl
cargo build --release -p etl-replicator --no-default-features --features clickhouse

The executable is written to target/release/etl-replicator. Available destination features are clickhouse, bigquery, ducklake, snowflake, and the currently deprecated iceberg implementation. Review Destinations before choosing a module.

Create the configuration

The replicator reads a required base.yaml plus an environment-specific file from a configuration directory. Keep that directory outside the repository so credentials cannot be committed accidentally.

Create /absolute/path/to/etl-config/base.yaml with the ClickHouse destination and Postgres source settings:

destination:
  clickhouse:
    url: "http://localhost:8123"
    user: "etl"
    password: "etl"
    database: default

pipeline:
  id: 1
  publication_name: seed_pub
  pg_connection:
    host: localhost
    hostaddr: null
    port: 5430
    name: etl_testdata
    username: postgres
    password: postgres
    tls:
      enabled: false
      trusted_root_certs: ""

Those host, port, and password values match the published cargo x init Docker defaults. For a remote ClickHouse, change url, user, password, and database, and enable TLS on both connections.

Create /absolute/path/to/etl-config/prod.yaml for production overrides. It may be empty when base.yaml contains the complete configuration:

{}

The default environment is prod. Set APP_ENVIRONMENT to dev, staging, or prod to load the corresponding file instead. Configuration values can be overridden with APP_-prefixed environment variables; use __ between nested keys. For example:

export APP_PIPELINE__PG_CONNECTION__PASSWORD='placeholder-password'
export APP_DESTINATION__CLICKHOUSE__PASSWORD='placeholder-clickhouse-password'

Do not put real secrets in tracked files, shell history, logs, or process arguments. Enable TLS and provide trusted root certificates for networked production connections.

Cloud destinations such as BigQuery need a service-account key instead of a ClickHouse password:

export APP_DESTINATION__BIG_QUERY__SERVICE_ACCOUNT_KEY='placeholder-service-account-json'

Run the replicator

APP_CONFIG_DIR must be an absolute path:

APP_CONFIG_DIR=/absolute/path/to/etl-config \
  APP_ENVIRONMENT=prod \
  ./target/release/etl-replicator

The process runs until it receives a shutdown signal or encounters a terminal pipeline error. Run it under a service manager that restarts failed processes, preserves logs, and provides resource limits.

Recovery and operations

Validate destination behavior, resource limits, and recovery procedures in a non-production environment before operating the replicator on production data.

• ETL provides at-least-once delivery. Destination writes must tolerate retries without producing incorrect duplicate state.
• Pipeline state and checkpoints are stored in Postgres. When replication uses a read-only physical replica, configure a separate writable store_pg_connection.
• The default invalidated_slot_behavior is error, which stops startup and requires operator intervention. Set it to recreate only when automatically resetting table states and repeating initial copies is acceptable.
• Monitor PostgreSQL replication-slot WAL retention, destination capacity, process memory, replication lag, and terminal errors.
• Back up configuration metadata, but never include credentials in diagnostic bundles or public issues.

See Architecture for the runtime model and Extension Points if you need a custom destination or state store.