Standalone Replicator

The standalone replicator is the ready-made etl-replicator application for running Supabase ETL without embedding the library in another Rust program.

What the replicator is

The replicator packages one ETL pipeline, a Postgres-backed state store, and one configured built-in destination into a long-lived process. It loads the source, destination, batching, retry, and operational settings from files and environment variables, starts replication, and handles graceful shutdown. It does not require Kubernetes or the managed Supabase product.

Use the standalone replicator when a built-in destination and the standard Postgres-backed store fit your deployment. Embed the etl crate instead when you need a custom destination, a custom store, or application-specific runtime orchestration. See First Pipeline for the library path.

Fastest local start

ClickHouse is the default destination. cargo x init starts Postgres and ClickHouse; no cloud account is required. First clone the repository and complete the toolchain prerequisites in DEVELOPMENT.md. Run the following commands from the repository root.

git clone https://github.com/supabase/etl.git
cd etl
cargo x init
cargo x setup replicator
cargo x seed
cargo x run replicator

That writes gitignored config under crates/etl-replicator/configuration/, seeds a publication named seed_pub, and starts replication into local ClickHouse at http://localhost:8123 (etl / etl). Seed tables appear as public_users, public_orders, and public_events, plus __current views:

curl -sS 'http://localhost:8123/?user=etl&password=etl' \
  --data-binary 'SELECT count() FROM "public_users__current"'

Stop the replicator with Ctrl+C.

The rest of this guide shows how to build the binary yourself and point it at ClickHouse with your own configuration directory. Other built-in destinations use the same binary with a different --features flag; see Destinations.

Prerequisites

• Rust 1.95.0, as pinned by rust-toolchain.toml.
• PostgreSQL 14 through 18 configured for logical replication, including a replication user and publication. Complete Configure Postgres first, or use the local Postgres from cargo x init.
• ClickHouse 23.5 or newer. Local Docker from cargo x init is enough.

PostgreSQL 15 or newer is recommended for column and row publication filters, and PostgreSQL 16 or newer is required when replication connects to a physical read replica.

Build the binary

Clone the repository and build only the destination features you need. ClickHouse is the local default:

git clone https://github.com/supabase/etl.git
cd etl
cargo build --release -p etl-replicator --no-default-features --features clickhouse

The executable is written to target/release/etl-replicator. Available destination features are clickhouse, bigquery, ducklake, snowflake, and the currently deprecated iceberg implementation. Review Destinations before choosing a module.

Create the configuration

The replicator reads a required base.yaml plus an environment-specific file from a configuration directory. Keep that directory outside the repository so credentials cannot be committed accidentally.

Create /absolute/path/to/etl-config/base.yaml with the ClickHouse destination and Postgres source settings:

destination:
  clickhouse:
    url: "http://localhost:8123"
    user: "etl"
    password: "etl"
    database: default

pipeline:
  id: "03dg2o5teb2eucxr9ggh3uie6"
  publication_name: seed_pub
  pg_connection:
    host: localhost
    hostaddr: null
    port: 5430
    name: etl_testdata
    username: postgres
    password: postgres
    tls:
      enabled: false
      trusted_root_certs: ""

Those host, port, and password values match the published cargo x init Docker defaults. For a remote ClickHouse, change url, user, password, and database, and enable TLS on both connections. Keep credentials out of url: a value with embedded credentials, such as https://alice:secret@clickhouse.example:8443, or with a query string is rejected when the configuration is loaded, because the URL is treated as non-secret configuration.

Create /absolute/path/to/etl-config/prod.yaml for production overrides. It may be empty when base.yaml contains the complete configuration:

{}

The default environment is prod. Set APP_ENVIRONMENT to dev, staging, or prod to load the corresponding file instead. Configuration values can be overridden with APP_-prefixed environment variables; use __ between nested keys. For example:

export APP_PIPELINE__PG_CONNECTION__PASSWORD='placeholder-password'
export APP_DESTINATION__CLICKHOUSE__PASSWORD='placeholder-clickhouse-password'

Do not put real secrets in tracked files, shell history, logs, or process arguments. Enable TLS and provide trusted root certificates for networked production connections.

Cloud destinations such as BigQuery need a service-account key instead of a ClickHouse password:

export APP_DESTINATION__BIG_QUERY__SERVICE_ACCOUNT_KEY='placeholder-service-account-json'

Pipeline intervals

The pipeline's intervals are independent. Omitted settings use these defaults:

All three refresh intervals must be greater than zero. The 100 ms memory interval keeps backpressure responsive without reading operating-system files on the hot path. The 10 s copy-slot interval balances invalid-slot detection against one check per copying table. Lag sampling uses one query per pipeline per tick; the 10 s default provides useful monitoring resolution without a query per worker. Lower lag intervals improve freshness at the cost of more source queries. They do not control replication batching, PostgreSQL feedback, or the Prometheus scrape interval.

For example, sample lag every 5 s while keeping the other defaults:

pipeline:
  replication_lag_refresh_interval_ms: 5000

A numeric settle_durable_interval_ms is the minimum age of an accepted streaming write before a later eligible keepalive may force durability. It is not a periodic flush timer or an upper bound on durability latency. 0 adds no age threshold, like null. Terminal table-sync completion forces durability without this wait; shutdown does not introduce a new durability barrier.

Active apply loops also enforce a separate 60 s maximum deferral, measured from the first Accepted result. After it expires, the next write must confirm cumulative durability, using buffered events or an empty barrier. This works during continuous traffic and paused source intake, and takes precedence over a longer idle minimum. ETL waits for any in-flight write before dispatching the confirmation request, so destination latency can still extend the total wait. Checkpoints advance only through completed source commits covered by confirmation.

Previously, table_sync_monitor_refresh_interval_ms also controlled lag sampling. If you customized it for metric freshness, set replication_lag_refresh_interval_ms explicitly when upgrading; otherwise lag sampling now uses its independent 10 s default.

Run the replicator

APP_CONFIG_DIR must be an absolute path:

APP_CONFIG_DIR=/absolute/path/to/etl-config \
  APP_ENVIRONMENT=prod \
  ./target/release/etl-replicator

The process runs until it receives a shutdown signal or encounters a terminal pipeline error. Run it under a service manager that restarts failed processes, preserves logs, and provides resource limits.

Shutdown

SIGINT (Ctrl+C) and SIGTERM request shutdown. Listeners are installed before asynchronous initialization, so pending startup can be cancelled. If a signal interrupts startup of a constructed pipeline, its destination is cleaned up. Initialization errors return immediately.

Once running, the pipeline stops intake and drains pending apply work under existing retry and durability rules. Interrupted initial copies restart from a fresh snapshot. The pipeline joins the apply worker, then table-sync workers, then shuts down the destination. Lag and memory sampling remain alive through this sequence and are stopped and joined afterward. Enabled probes report stopping while liveness remains healthy.

Lag and memory monitors are optional: a monitor panic is logged without stopping replication. If lag sampling stops, lag becomes unavailable. If memory sampling stops, memory gauges become unavailable and emergency memory backpressure is released; batch sizing retains the last sampled capacity. A failed PostgreSQL lag query marks lag unavailable and retries. Background schema pruning is also best-effort; if it fails, obsolete schema versions may be retained. PostgreSQL feedback remains required: sender failure stops the current replication attempt and follows the worker's retry policy.

Requested task cancellations are silently accepted. Worker panics, task errors, and unexpected cancellations return immediately, requesting abort of remaining owned tasks without awaiting further cleanup. This does not roll back writes or guarantee that native work has stopped. Recovery uses persisted checkpoints; uncheckpointed work may be replayed even after a successful exit. Persisted table errors remain stopped across process restarts, including when shutdown interrupted a timed retry; restarting the process does not clear them.

There is no internal shutdown grace timer or second-signal force-exit behavior. Destination or store operations can delay exit. Configure an external termination limit, such as Kubernetes' grace period followed by SIGKILL, which cannot run cleanup.

Activity probes

Enable the optional HTTP listener with a top-level configuration block:

health:
  port: 9001
  stall_timeout_ms: 300000

Omitting health disables it; health: {} uses these defaults. The listener accepts IPv4 and IPv6 connections on all interfaces, falling back to IPv4 when IPv6 sockets are unsupported.

Use another port, such as 19001, when running alongside the local Docker stack, which reserves port 9001 for ClickHouse.

• /livez: returns 503 when monitored work stalls; otherwise 200, including initialization, intentional waits, and graceful shutdown.
• /readyz: returns 200 once the pipeline is running with monitored activity and no stalls; otherwise 503. Initial sync need not be complete.

Probes observe completed apply-loop iterations and destination copy batches, not durable replication progress. Parallel copy workers share one observation per table; the observation includes source commit and the final destination durability barrier. Slot acquisition and intentional catchup waits are exempt. With no observations, the process stays live but unready. Apply loops allow the greater of stall_timeout_ms and PostgreSQL's wal_sender_timeout (60-second fallback when disabled or unavailable). Tune the stall timeout above expected copy batch and finalization durations.

For Kubernetes, use /livez for startup and liveness, and /readyz for readiness on the configured port. Configure probe timing and failure thresholds in the Pod specification. Enable probes only with a replicator image that supports these endpoints and has its health listener enabled.

Recovery and operations

Validate destination behavior, resource limits, and recovery procedures in a non-production environment before operating the replicator on production data.

• ETL provides at-least-once delivery. Destination writes must tolerate retries without producing incorrect duplicate state.
• Pipeline state and checkpoints are stored in Postgres. When replication uses a read-only physical replica, configure a separate writable store_pg_connection.
• The default invalidated_slot_behavior is error, which stops startup and requires operator intervention. Set it to recreate only when automatically resetting table states and repeating initial copies is acceptable.
• Monitor PostgreSQL replication-slot WAL retention, destination capacity, process memory, replication lag, and terminal errors.
• Back up configuration metadata, but never include credentials in diagnostic bundles or public issues.

Outstanding CDC durability age

etl_destination_durability_age_seconds measures the live age of the main apply worker's oldest outstanding destination-write interval. Timing starts before a nonempty write_events call, including a call or result that never returns. Later Accepted results and PostgreSQL heartbeats preserve the original start. A successfully handled Durable result covers earlier accepted writes in that ordered stream and clears the interval. Empty barriers can confirm existing work but do not start a new interval on an idle source.

An independent pipeline-owned task refreshes the gauge every second, including while dispatch, confirmation, retry backoff, or shutdown draining is pending. The gauge has pipeline_id, worker_type="apply", and replication_path="cdc" labels; the standalone replicator also supplies its global project and destination labels. Initial copy and table-sync catchup are excluded. Main-worker CDC remains observed while other tables are copying.

Retries retain the original age until a durable replay commit covers the prior attempt's dispatched WAL frontier. This is conservative: partially replayed work may remain in the measured interval. Errors and cancelled writes do not clear the interval. Completed latency histograms keep their existing semantics.

Zero means no observed outstanding work in this process. Negative values mean observation is unavailable, including before the main apply loop starts and after the sampling task stops. Process restarts lose the in-memory age; missing metrics, a stopped process, or a new process exporting zero do not prove previous work became durable. Old images do not export this metric.

Recommended alerts are age above 300 seconds for 120 seconds and above 1,200 seconds for 60 seconds, independently per pipeline. Scrape, evaluation, and notification delays are additional. Keep separate scrape/workload health checks for absent series. This measures outstanding CDC durability, not a complete source-to-query freshness SLO or a single provider request's latency. Use WAL lag and source retention alongside it.

Replication slot creation

etl_replication_slot_creation_duration_seconds records client elapsed time for each CREATE_REPLICATION_SLOT command. It includes PostgreSQL waiting for older transactions, but excludes connection setup, table-lock acquisition, deleting the previous slot, committing the surrounding transaction, and copying rows. Reusing an existing slot emits no creation sample. Labels identify pipeline_id, worker_type (apply or table_sync), and outcome (success, error, or cancelled). Each retry is a separate attempt. Cancellation means the client stopped awaiting the command, not that PostgreSQL has confirmed its termination.

etl_replication_slot_creation_age_seconds reports the oldest active command for each pipeline and worker type. An independent task updates it every second, so a pending command does not freeze its own age. Concurrent table-sync attempts aggregate by oldest age, not by summing their ages. Completing or canceling one attempt leaves the other attempts observable. Zero means idle; -1 means the sampler has stopped and must be filtered out. Retries and process restarts reset age. Missing telemetry or zero age does not prove slot creation succeeded.

Duration is published only when an attempt ends; use live age to see ongoing waits. The duration's exported _sum / _count gives an observation-weighted mean over the process lifetime, including attempts completed before the first scrape. A rate(_sum) / rate(_count) query instead measures recent observations but can miss startup completions before the first scrape. Keep worker types and outcomes separate, and show gaps when there are no observations. Summary quantiles cannot be pooled across processes: a maximum p99 is the worst reported per-process p99, not a global percentile.

Replication lag

etl_replication_wal_lag_bytes measures replication lag: the source WAL head minus progress safe to acknowledge, including durable destination work and idle WAL. etl_replication_receive_wal_lag_bytes measures receive lag: the same head minus the worker's consumed position. The source head is durably flushed WAL on a primary and replayed WAL on a standby.

Labels identify pipeline_id, worker_type (apply or table_sync), and replication_path (copy or cdc). Table-sync copy and CDC lag are separate maxima, so initial-copy backlog is distinct from catchup in the apply loop. Copy exposes only replication lag, held at its fixed snapshot boundary through final durability and the wait to start CDC. Copying rows does not consume that WAL backlog.

These are WAL address distances, not payload sizes or elapsed latency. Unknown values are -1; filter them out rather than treating them as zero. Query PostgreSQL directly for slot acknowledgment and retention details.

See Architecture for the runtime model and Extension Points if you need a custom destination or state store.