Keel Migrate: plan and prove

Licensed module

Keel Migrate is licensed per tenant (Plans and licences). Without a licence, Migrations is not in the navigation and its routes answer 404. The dev demo tenant is licensed.

Use this phase to move a system to a new target, such as a legacy SQL Server ERP to Snowflake, or into the tables behind a semantic model:

1. Keel proposes how every source table and column maps to the target, with its evidence.
2. People review the mapping and approve it as a plan.
3. Keel generates the dbt project and SQL scripts, which you run with your own tools.
4. After your pipeline has loaded the target, Keel reconciles both sides and a Steward signs the result off.

Keel issues no write statement in this phase. It reads the source and the target with read-only connections. To let Keel load and sync the target itself, see Run and cut over.

Who does what

Action	Permission	Roles
Create migrations, set targets, infer, review and edit mappings, code maps, commit plans, generate code, run reconciliation	EditMigrations	Owner, Admin, Steward, Modeler
Approve a plan version, record exceptions, sign off a reconciliation	ApproveMigrations	Owner, Admin, Steward
See migrations	ReadCatalog	everyone

1. Create a migration and set its target

Migrations › New migration. Give it a name, pick 1 to 10 sources (connected and scanned data sources) and a strategy:

• Lift and shift: the same tables on a new platform.
• Re-model: the legacy schema mapped onto a new target model.
• Consolidate: several sources feed one target model.

Then Set target. There are three kinds:

Target	What Keel maps against
Data source	An existing database, harvested like any source. Scan it first.
Semantic model	The tables behind a model's entities
Upload DDL / dbt	CREATE TABLE statements (.sql) or a dbt project (schema .yml files, or a zip), up to 10 MB, with a dialect. The files are parsed in memory and not kept.

For a model or an upload target, you can also name a database to reconcile against. Keel reads it with a read-only role to compare counts, keys and totals.

The overview shows five stages: Target → Mapping → Generate → Reconcile → Run & cutover. It also shows tiles for tables and columns mapped, the share suggested by Keel, and the last reconciliation.

2. Infer mappings

Infer mappings runs a job that proposes the following:

• Table matches. They come from names (after expanding legacy abbreviations such as CUST_MST → customer master or ORD_HDR → order header), column-set overlap, table roles and row counts. An extension table that shares the key (ORD_HDR_EXT) joins its primary table as a many-to-one match.
• Column matches. They come from names, type fit, meaning, key roles, profiled values and glossary terms.
• Transforms. Date parsing (a varchar YYYYMMDD column into a date, with the share of profiled values that parse), trim, case, cast, code maps for low-cardinality codes, split and concatenation of names, unit conversions, surrogate keys and defaults for target columns with no source.
• Gaps. Target columns with no source, and source columns with no target. Dropped sensitive columns are highlighted.

Every proposal carries a confidence (0–1) and its evidence, and stays Proposed until someone decides. Sources must be scanned first, and the target must have tables.

3. Review in the mapping workbench

Mapping lists the tables with their coverage. For each table there is a source → transform → target grid with a confidence and a decision per column.

• Select a column mapping to see its evidence and a live SQL or dbt model preview of the generated code.
• Edit mapping changes the transform. The options are None (copy), Cast, Trim, Upper / lower case, Default value, Code map, Concatenate, Split, Parse date, Unit conversion, Currency conversion, Surrogate key, Hash and Custom SQL ({0}, {1} … stand for the source columns). Saving an edit counts as your decision.
• Accept all ≥ 0.90 accepts the high-confidence proposals of the table in one step.
• The keyboard works too: J / K move, A accepts, R rejects, N marks needs work.
• Code maps are value translation tables, for example status A/I/X → active/inactive/closed. Each shows its coverage of the profiled source values, lists the profiled values without a pair, and has a default for unmapped values. Saving a code map accepts it.

A restricted source column keeps its classification in the target. When the plan is approved, Keel proposes the matching mask policy in Governance.

Current limits

Confidence is conservative: correct matches often score 0.68–0.89, so Accept all ≥ 0.90 takes only part of a plan. Discovery does not harvest CHECK (col IN (…)) value sets, so a harvested target gets few inferred code maps. Uploaded DDL/dbt targets carry their value sets. Mapping inference uses heuristics only; there is no LLM rationale yet. The UI has no plan-diff or plan-history view yet, and no editor for table-level join or filter logic (the API has one).

4. Commit and approve a plan

Commit plan vN takes an immutable, versioned snapshot of the accepted mappings. A plan needs at least one accepted table mapping, and committing is refused when nothing changed since the last version. A Steward, Admin or Owner then approves the latest version. Code generation and runs always work from an approved version.

On approval, Keel adds lineage edges from each source column to its target column, with the transform, and proposes masks for restricted columns.

5. Generate the code

Generate… (in Generated code) builds from the approved plan. There are two choices:

• Output: dbt, SQL or both.
  • The dbt project has three layers: staging (a view per source table), intermediate (renames, casts, transforms, joins, deduplication, code-map seeds) and marts (the target tables). Loads are incremental on the business key where configured. Tests come from the target constraints.
  • SQL is DDL plus idempotent loads in run order: MERGE, INSERT … ON CONFLICT or the dialect's equivalent. The loads never truncate.
• Delivery: a zip download, or a Git branch and pull request in the workspace's repository (Settings › Integrations). The Git option is offered only when a repository is connected.

Output is available for all eight dialects and is byte-identical for the same plan. Anything Keel can't express exactly is generated as NULL with a warning, which is also listed in the generated README. Keel never generates SQL that loads something else. Keel never runs this code. You run it with dbt or your own pipeline, or you let Keel run the migration (Run and cut over).

6. Reconcile and sign off

After your pipeline has loaded the target, open Reconciliation and select Run reconciliation. Keel runs read-only aggregate queries on both sides. Every statement is a single SELECT or WITH. It then compares the results per table:

Check	Severity
Row count	Critical
Key match (missing and extra keys, when the target has a business key)	Critical
Checksum per column (SHA-256 over canonical text, comparable across engines)	Critical for key and amount columns, else Warning
Sum / min / max of numeric, non-sensitive columns	Critical for amounts, else Warning
Null count	Warning
Code map coverage	Warning

Each failing check shows the difference and a likely cause, for example "Target is missing 20 rows (0.2% of the source)", "Sum differs by 1.37% — check the currency conversion" or "values look scaled by 10^2". It also shows up to 50 sample failing keys, and none when the key column is sensitive. Fix the mapping → takes you back to the workbench.

Sign off is blocked while any Critical check fails. You can instead record an exception for the check, with a reason of 10 to 1,000 characters that goes to the audit log. After sign-off, the migration's mappings and plan can no longer change.

What leaves your databases

Row counts, digest sums, SUM/MIN/MAX of non-sensitive numeric columns, and at most 50 canonical business keys of failing rows. Never other row values, and never values of sensitive columns. Driver error messages are not copied into the results, because they can quote values.

Engine coverage

Code generation and reconciliation have run end to end against real PostgreSQL, and SQL Server has been a reconciliation source. The other six dialects are covered by golden-file tests. Some transforms can't be recomputed on every engine, for example split on SQL Server and Oracle. These checks show as Not checked, and a Critical one still blocks sign-off until it gets an exception. Reconciliation against Azure SQL Edge on arm64 is slow.

Everything in this guide is audited: target changes, decisions, plans and approvals, generations, runs, exceptions and sign-offs.