Data quality

Keel suggests, runs and scores data-quality rules on the tables behind your model, opens incidents when critical checks fail, and turns the results into each metric's trust score.

Rules

Type	Checks	Typical suggestion
NotNull, Unique	Keys and grain columns	Primary keys, grains
ReferentialIntegrity	Orphan rows on a join	Accepted joins (0% tolerance for declared foreign keys, 1% for inferred)
AcceptedValues	A known value set	Low-cardinality, non-sensitive columns (≤ 20 values)
Range, NonNegative	Numeric bounds	Observed min..max ± 10%; amounts and quantities
Regex	A pattern	Emails, ISO codes (SQL Server uses a LIKE fallback)
Freshness	Lag of the newest timestamp	Fact timestamps (limit = 2 × the observed cadence)
RowCountDelta, VolumeAnomaly	Volume changes	Every table (50% delta); same-weekday baseline, robust z-score > 3.5
DistributionDrift	PSI against a baseline	Measures (baseline from the first run)
CustomSql	Your query returning the failing rows	—

Rules belong to one of seven dimensions: Completeness, Uniqueness, Validity, Integrity, Timeliness, Consistency, Reasonableness. Severity is Critical, Warning or Info; only Critical failures open incidents and cap trust.

Suggested rules

After every successful scan Keel suggests rules from profiles and classifications — as proposals. Review them in Data quality › Review suggested rules (confidence, evidence, bulk accept ≥ 0.85); rejected suggestions never come back. Accepted rules run on their schedule: freshness hourly, everything else daily at 02:00 UTC (five-field UTC cron).

Custom SQL

A custom rule is one read-only SELECT or WITH query that returns the failing rows. A tokenizer rejects DDL/DML, INTO, SET, FOR UPDATE/SHARE and multiple statements (POST …/quality/custom-sql:validate checks it without saving). Keel runs it as SELECT COUNT(*) FROM (…) q with bound parameters on the source.

curl -X POST "$KEEL/api/v1/workspaces/$WS/quality/rules" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{
        "dataSourceId": "'"$SOURCE"'", "schema": "public", "table": "invoice", "column": null,
        "name": "invoice_total_positive", "type": "CustomSql", "severity": "Critical", "dimension": "Validity",
        "description": "Invoices must have a positive total.",
        "parameters": { "sql": "SELECT invoice_id FROM public.invoice WHERE total <= 0" },
        "schedule": "0 6 * * *"
      }'

Runs and privacy

Rules run pushed down on the source through its dialect (all eight connector kinds). A run stores the measured value and at most 50 failing rows' key columns — never values of Confidential, Restricted or PII columns; with no safe key, counts only. Re-run check (:run) runs one rule now; assets:run runs every rule of a table.

Scorecards

Scores 0–100 per dimension and overall, with a daily trend, for any asset: workspace, source, table, column, entity, metric or domain (GET …/quality/scorecard?asset=). Entity, metric and domain scores roll up through lineage.

Incidents

A Critical failure opens an incident (one per rule) and notifies Owners, Admins and Stewards (in-app, email, and any subscribed Slack/Teams/webhook channel). Acknowledge assigns it to you; it resolves itself after 3 consecutive passing runs, or by hand. Open incidents appear on Home and as a dot in the navigation.

In change requests

The change-request data quality check reads stored results for the tables behind changed metrics. It warns; it never blocks the merge.

Known gaps

• Importing dbt tests and Great Expectations suites as rules is part of M9 (Imports).
• Incident summaries don't carry the threshold and unit; the dashboard shows the latest 50 results without paging; acknowledgers and assignees show as ids.