Standalone Replicator

The standalone replicator is the ready-made etl-replicator application for running Supabase ETL without embedding the library in another Rust program.

What the replicator is

The replicator packages one ETL pipeline, a Postgres-backed state store, and one configured built-in destination into a long-lived process. It loads the source, destination, batching, retry, and operational settings from files and environment variables, starts replication, and handles graceful shutdown. It does not require Kubernetes or the managed Supabase product.

Use the standalone replicator when a built-in destination and the standard Postgres-backed store fit your deployment. Embed the etl crate instead when you need a custom destination, a custom store, or application-specific runtime orchestration. See First Pipeline for the library path.

Fastest local start

ClickHouse is the default destination. cargo x init starts Postgres and ClickHouse; no cloud account is required. First clone the repository and complete the toolchain prerequisites in DEVELOPMENT.md. Run the following commands from the repository root.

git clone https://github.com/supabase/etl.git
cd etl
cargo x init
cargo x setup replicator
cargo x seed
cargo x run replicator

That writes gitignored config under crates/etl-replicator/configuration/, seeds a publication named seed_pub, and starts replication into local ClickHouse at http://localhost:8123 (etl / etl). Seed tables appear as public_users, public_orders, and public_events, plus __current views:

curl -sS 'http://localhost:8123/?user=etl&password=etl' \
  --data-binary 'SELECT count() FROM "public_users__current"'

Stop the replicator with Ctrl+C.

The rest of this guide shows how to build the binary yourself and point it at ClickHouse with your own configuration directory. Other built-in destinations use the same binary with a different --features flag; see Destinations.

Prerequisites

• Rust 1.95.0, as pinned by rust-toolchain.toml.
• PostgreSQL 14 through 18 configured for logical replication, including a replication user and publication. Complete Configure Postgres first, or use the local Postgres from cargo x init.
• ClickHouse 23.5 or newer. Local Docker from cargo x init is enough.

PostgreSQL 15 or newer is recommended for column and row publication filters, and PostgreSQL 16 or newer is required when replication connects to a physical read replica.

Build the binary

Clone the repository and build only the destination features you need. ClickHouse is the local default:

git clone https://github.com/supabase/etl.git
cd etl
cargo build --release -p etl-replicator --no-default-features --features clickhouse

The executable is written to target/release/etl-replicator. Available destination features are clickhouse, bigquery, ducklake, snowflake, and the currently deprecated iceberg implementation. Review Destinations before choosing a module.

Create the configuration

The replicator reads a required base.yaml plus an environment-specific file from a configuration directory. Keep that directory outside the repository so credentials cannot be committed accidentally.

Create /absolute/path/to/etl-config/base.yaml with the ClickHouse destination and Postgres source settings:

destination:
  clickhouse:
    url: "http://localhost:8123"
    user: "etl"
    password: "etl"
    database: default

pipeline:
  id: 1
  publication_name: seed_pub
  pg_connection:
    host: localhost
    hostaddr: null
    port: 5430
    name: etl_testdata
    username: postgres
    password: postgres
    tls:
      enabled: false
      trusted_root_certs: ""

Those host, port, and password values match the published cargo x init Docker defaults. For a remote ClickHouse, change url, user, password, and database, and enable TLS on both connections. Keep credentials out of url: a value with embedded credentials, such as https://alice:secret@clickhouse.example:8443, or with a query string is rejected when the configuration is loaded, because the URL is treated as non-secret configuration.

Create /absolute/path/to/etl-config/prod.yaml for production overrides. It may be empty when base.yaml contains the complete configuration:

{}

The default environment is prod. Set APP_ENVIRONMENT to dev, staging, or prod to load the corresponding file instead. Configuration values can be overridden with APP_-prefixed environment variables; use __ between nested keys. For example:

export APP_PIPELINE__PG_CONNECTION__PASSWORD='placeholder-password'
export APP_DESTINATION__CLICKHOUSE__PASSWORD='placeholder-clickhouse-password'

Do not put real secrets in tracked files, shell history, logs, or process arguments. Enable TLS and provide trusted root certificates for networked production connections.

Cloud destinations such as BigQuery need a service-account key instead of a ClickHouse password:

export APP_DESTINATION__BIG_QUERY__SERVICE_ACCOUNT_KEY='placeholder-service-account-json'

Run the replicator

APP_CONFIG_DIR must be an absolute path:

APP_CONFIG_DIR=/absolute/path/to/etl-config \
  APP_ENVIRONMENT=prod \
  ./target/release/etl-replicator

The process runs until it receives a shutdown signal or encounters a terminal pipeline error. Run it under a service manager that restarts failed processes, preserves logs, and provides resource limits.

Shutdown

SIGINT (Ctrl+C) and SIGTERM request shutdown. Listeners are installed before asynchronous initialization, so pending startup can be cancelled. If a signal interrupts startup of a constructed pipeline, its destination is cleaned up. Initialization errors return immediately.

Once running, the pipeline stops intake and drains pending apply work under existing retry and durability rules. Interrupted initial copies restart from a fresh snapshot. Workers finish before destination cleanup and background tasks are joined. Enabled probes report stopping while liveness remains healthy.

Requested task cancellations are silently accepted. Panics, task errors, and unexpected cancellations return immediately, requesting abort of remaining owned tasks without awaiting further cleanup. This does not roll back writes or guarantee that native work has stopped. Recovery uses persisted checkpoints; uncheckpointed work may be replayed even after a successful exit. Persisted table errors remain stopped across process restarts, including when shutdown interrupted a timed retry; restarting the process does not clear them.

There is no internal shutdown grace timer or second-signal force-exit behavior. Destination or store operations can delay exit. Configure an external termination limit, such as Kubernetes' grace period followed by SIGKILL, which cannot run cleanup.

Activity probes

Enable the optional HTTP listener with a top-level configuration block:

health:
  port: 9001
  stall_timeout_ms: 300000

Omitting health disables it; health: {} uses these defaults. The listener accepts IPv4 and IPv6 connections on all interfaces, falling back to IPv4 when IPv6 sockets are unsupported.

Use another port, such as 19001, when running alongside the local Docker stack, which reserves port 9001 for ClickHouse.

• /livez: returns 503 when monitored work stalls; otherwise 200, including initialization, intentional waits, and graceful shutdown.
• /readyz: returns 200 once the pipeline is running with monitored activity and no stalls; otherwise 503. Initial sync need not be complete.

Probes observe completed apply-loop iterations and destination copy batches, not durable replication progress. Parallel copy workers share one observation per table; the observation includes source commit and the final destination durability barrier. Slot acquisition and intentional catchup waits are exempt. With no observations, the process stays live but unready. Apply loops allow the greater of stall_timeout_ms and PostgreSQL's wal_sender_timeout (60-second fallback when disabled or unavailable). Tune the stall timeout above expected copy batch and finalization durations.

For Kubernetes, use /livez for startup and liveness, and /readyz for readiness on the configured port. Configure probe timing and failure thresholds in the Pod specification. Enable probes only with a replicator image that supports these endpoints and has its health listener enabled.

Recovery and operations

Validate destination behavior, resource limits, and recovery procedures in a non-production environment before operating the replicator on production data.

• ETL provides at-least-once delivery. Destination writes must tolerate retries without producing incorrect duplicate state.
• Pipeline state and checkpoints are stored in Postgres. When replication uses a read-only physical replica, configure a separate writable store_pg_connection.
• The default invalidated_slot_behavior is error, which stops startup and requires operator intervention. Set it to recreate only when automatically resetting table states and repeating initial copies is acceptable.
• Monitor PostgreSQL replication-slot WAL retention, destination capacity, process memory, replication lag, and terminal errors.
• Back up configuration metadata, but never include credentials in diagnostic bundles or public issues.

See Architecture for the runtime model and Extension Points if you need a custom destination or state store.