Keel Migrate: run and cut over

Licensed module

This is the second phase of Keel Migrate, a module licensed per tenant (Plans and licences). It builds on Plan and prove: a run always executes an approved plan version.

Letting Keel execute the migration is optional. Your private agent does the work inside your network:

1. It bulk-loads the approved plan into the target.
2. It keeps the target in sync with change data capture until you cut over.
3. You follow a guided cutover that switches Keel's own serving to the target, with a rollback window.

This is the only part of Keel that writes to customer systems: the Migrate write plane (ADR-0021, see Platform and modules). The page shows it with a dark Write plane banner.

What stays in your network

• Rows move source → target through the agent. They never pass through or persist in Keel's cloud.
• The agent reports counts, sizes, timings, rows/s, lag, quarantine counts, reconciliation checksums and at most 50 business keys. It never reports row values or driver messages. A contract test posts rows with unique marker values and proves that no marker appears in anything the agent sends.
• The agent only talks to the Keel URL it enrolled with. It holds the write credential's secret, and Keel stores only the secret's name.

Before you start

Prerequisite	Details
An approved plan	Plan and prove, steps 1–4
A target data source	A run writes to a registered target database. Semantic-model and uploaded targets can't be run by Keel.
A private agent that reaches both sides	The sources must be connected Through a private agent, on the same agent that holds the write credential.
A keel_stage schema on the target	A DBA creates it and grants it to the write credential. It holds staging and delta tables, the chunk ledger (resume state) and the poison-row quarantine. Drop it after cutover.
A write credential	A separate login on the target with the rights below and nothing more
Change capture set up on the source (optional)	See Change capture

Supported engines:

Role	Engines
Source	PostgreSQL, SQL Server, Oracle
Target	PostgreSQL, SQL Server, Snowflake, BigQuery

The other connectors (MySQL, Redshift, Databricks) are not yet write-plane engines. A run reads one source through one agent.

Cloud targets.

• Snowflake: the agent writes each chunk to a local gzip CSV, PUTs it to the staging table's stage and loads it with COPY INTO in the KEEL_STAGE schema.
• BigQuery: the agent uses load jobs into a keel_stage dataset, which an administrator creates in the target's location. Chunks are at least 100,000 rows, because of the load-job quota.
• Both: Keel checks the write credential's grants (Snowflake roles, BigQuery dataset access) and refuses one that can write elsewhere.
• Network: the agent reaches the warehouse directly; warehouse traffic never passes through Keel.
• Verification so far: both cloud targets have been verified with unit tests and, for BigQuery, an emulator. Confirm a first run on your account.

1. Register the write credential and test its scope

Run & cutover › Write credential › Register credential asks for the following:

• Agent: the agent that reads the sources, so rows never leave your network.
• Secret name: the name of a secret the agent already holds, as the file /run/secrets/keel/<name> or the environment variable KEEL_SECRET_<NAME>. Never enter the secret itself. It must differ from every read credential of the sources and the target.
• User and Target schemas (comma-separated). Keel adds its staging schema, keel_stage.

Test scope asks the agent to check what the credential can do. A run can only be scheduled once the check says Accepted. The check refuses a credential that can write outside the target schemas and keel_stage, including through role memberships, ownership, default privileges or server roles. It also reports rights a run lacks.

Target	The credential needs	Refused when it has
PostgreSQL	USAGE, CREATE on keel_stage; USAGE on each target schema; SELECT, INSERT, UPDATE, DELETE on the target tables	SUPERUSER, CREATEDB, CREATEROLE, pg_write_all_data, database ownership or CREATE, rights to write to another schema, table or sequence, default privileges that grant writes outside the scope
SQL Server	CREATE TABLE; ALTER, SELECT, INSERT, UPDATE, DELETE ON SCHEMA::keel_stage; SELECT, INSERT, UPDATE, DELETE on the target schemas	sysadmin and other server roles, CONTROL SERVER, db_owner, db_datawriter, db_ddladmin, database-level write or schema rights, write rights on another schema or object

2. Decide who can run it

• Who can run it: an Owner or Admin grants migrate:execute per migration to Owners, Admins and Stewards. API keys and agents never execute. Only people with the grant can start, pause, resume or retry a run, or perform cutover steps 1–3.
• Four-eyes on the switch (Owner or Admin): when it is on, a second Steward or Owner, never the requester, approves the switch and any exceptions at the final reconciliation.

Switching serving and rolling back always need a Steward or Owner (ApproveMigrations).

3. Start a run

Start run… asks for three things:

• Change capture. See the table below.
• Chunk rows. 1,000–1,000,000 rows per chunk.
• Parallelism. 1–16 tables at once.

The agent picks the run up within a minute. It then does the following:

1. Starts change capture before the load, so nothing committed during the load is missed.
2. Copies each source table into keel_stage in key-range chunks. PostgreSQL uses binary COPY and SQL Server uses SqlBulkCopy. Each chunk commits in one transaction together with its ledger row.
3. When every table is staged, runs Keel's generated SQL on the target. The loads are idempotent and never truncate, and parents load before children, following the target's own foreign keys.
4. Applies captured changes until the cutover. Each window of changes is applied in one transaction, compacted per key, and its position is saved with it.

Resuming. If the agent is killed or restarted, the run resumes from the chunk ledger on the target: committed chunks are not loaded twice. Keel notices a run whose agent is online but no longer reports it, and re-issues it automatically, up to 3 times an hour.

Failures. Transient errors are retried with backoff. Rows that fail with a data error are isolated and moved to the quarantine, which holds their business key and error code, never their values. More than 100 poison rows in one chunk fails the chunk instead. A failed chunk or load leaves the run Load failed · retry after fixing and sends a critical notification. Fix the cause, then use Retry failed chunks. Pause and Resume stop and continue after the current chunk.

Monitoring. The Load and sync panel shows the following, and polls for updates:

• bulk-load progress, rows/s and ETA;
• per-table chunks and status (Pending, Loading, Loaded, Syncing, In sync, Failed, Paused);
• change-capture lag (an alert fires above 60 s), changes per minute and quarantined rows.

The timeline lists every step. Keel reconciles after the bulk load and every 15 minutes while syncing.

Change capture

Mode	For	Source prerequisites
SQL Server change tracking (recommended for SQL Server)	SQL Server; no SQL Server Agent needed	ALTER on the database and tables, or a DBA enables tracking; every table needs a primary key. VIEW SERVER STATE (or VIEW DATABASE STATE on Azure SQL) for exact lag. Keel enables 3 days of retention unless tracking is already on.
SQL Server CDC	SQL Server with the SQL Server Agent running	sysadmin / db_owner to enable CDC, or a DBA does it. For tables with (n)varchar(max), varbinary(max), xml, text, ntext or image columns, max text repl size must be -1. Keel refuses otherwise and never changes that server setting.
PostgreSQL logical replication	PostgreSQL	wal_level = logical, a free replication slot and WAL sender, a role with REPLICATION and SELECT, table ownership to create the publication, and pg_hba.conf access for the agent's replication connection. The slot is dropped when the run ends.
Oracle LogMiner	Oracle (built-in LogMiner, no extra software)	ARCHIVELOG mode, minimal supplemental logging, and the key columns logged per table (ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS or (ALL)). A role with LOGMINING, SELECT ANY TRANSACTION, EXECUTE on DBMS_LOGMNR and SELECT on the V$LOGMNR_*, V$LOG*, V$ARCHIVED_LOG and V$TRANSACTION views. Keel checks each prerequisite and says what the DBA must do; it never changes database configuration. Archived logs must be kept long enough, otherwise the run needs a reload.
Incremental polling	Any engine with a watermark column (a rowversion or last-modified column)	SELECT only. Captures inserts and updates only: deletes are not captured, and reconciliation flags them.
None	A one-time load	Nothing is synced before the cutover

Setting up change capture is the only change Keel makes on a source, and a DBA can do it instead. Keel turns off only the tracking or CDC that it turned on itself. A PostgreSQL replication slot keeps WAL until Keel acknowledges applied changes, so stop a run rather than abandoning it, and monitor pg_replication_slots. The full privilege tables are in docs/10-devops.md in the repository.

4. Cut over

Once the bulk load is done and changes are being captured, Start cutover opens a checklist of five steps. Each step is confirmed on its own, in order, and audited with its actor and plan version.

1. Freeze the source. Stop your applications writing to the source, then confirm. Keel checks with the agent's reports: it needs a report from the last 2 minutes, no applied changes for 5 minutes, and lag within that window. Until then, the step stays open and says what Keel saw, for example "changes were applied 40 s ago".

2. Final sync. The agent applies the last captured changes and waits for zero lag.

3. Final reconciliation. Every reconciliation check runs again. The switch stays blocked while any Critical check fails. With four-eyes on, a second person approves Critical checks that pass only by exception.

4. Switch serving. Before you confirm, Keel shows the impact: which models and endpoints move to the target, which references can't be carried over (no table mapping, no column mapping, SQL-defined entities), and which certified metrics are at risk because the plan transforms or drops their columns. Then Keel does the following:

   • It rebinds each affected model to the target tables and columns on a cutover/… branch. Metric, dimension and entity names don't change.
   • It merges and releases through the normal change-request path.
   • It repoints the serving endpoints that follow the latest release. MCP and the SQL interface follow their endpoints, and caches are dropped.

   Certified metrics at risk block the switch unless a Steward or Owner uses Force switch…, which is recorded in their name. With four-eyes on, one person requests the switch and a second Steward or Owner approves it.

5. Rollback window. It stays open for 72 hours by default. During the window the agent keeps capturing changes on the source.

   • Roll back (Steward or Owner) repoints serving to the source in under a minute. It is an endpoint pointer swap: the run ends, change capture stops, and a new cutover needs a new run.
   • Close window ends the window early: after that you can no longer roll back, and the agent releases change capture.

Your switch steps lists what Keel can't see, such as repointing your own applications and BI tools. Tick the items off as your teams finish. This list is not saved.

What the switch doesn't do: it doesn't translate SQL functions between dialects (the preview warns you), it doesn't rewrite SQL-defined entities, and it doesn't move endpoints pinned to an older release. Model history is not reverted by a rollback: the cutover release stays published.

Measured so far

• A full cutover end to end on PostgreSQL, with the real agent and control plane:
  • 62,000 rows loaded, with the agent killed and resumed mid-load and no duplicates;
  • about 1,160 changes synced at zero lag;
  • the switch in about 0.3 s, and a rollback in about 0.6 s.
• SQL Server → PostgreSQL: 200,000 rows in CI, killed and resumed, reconciled exactly.
• Change-capture lag at 500 changes/s: p95 0.35 s with PostgreSQL logical replication and 4.3 s with SQL Server change tracking.

Current limits

• Sources: PostgreSQL, SQL Server and Oracle. Targets: PostgreSQL, SQL Server, Snowflake and BigQuery. Oracle is not a target yet, and capturing a 19c PDB from the CDB root isn't supported.
• A run reads one source through one agent.
• There is no run history (only the newest or active run) and no "abandon run" action: a failed run keeps its change capture so that a retry can resume.
• An automatic re-issue can't replace a runner that is alive but hung.
• A Full re-load into a populated target can't delete a parent row that a child row still references.
• The run page polls rather than streams.
• The 10-million-row load soak and the 10-minute CDC soak run nightly. Full-scale results for the 10-million-row target are pending their first run. SQL Server CDC is tested only on x64 SQL Server.