Skip to content

Operate / Operations

Operations ​

For Anchor's platform operators: the operator console, tenant offboarding, consented support sessions and the status endpoint (F23). Backups, cross-region recovery and infrastructure-as-code depend on the cloud Anchor chooses and are documented as runbooks and targets, not built yet — see Backups and restore.

Platform operators ​

Operators are Anchor staff, not a tenant role: an operator has no access to any tenant's data by being one. The deployment lists their IdP subjects in configuration, so granting operator rights is a reviewed deployment change:

json
{
  "Keel": {
    "Operators": {
      "Subjects": ["2c0d…-okta-sub-of-an-operator"],
      "RequiredAmr": "hwk"
    }
  }
}

RequiredAmr (optional) makes operator rights depend on the sign-in method: with hwk, the token's amr claim must say a hardware key was used. Operator routes live under /api/v1/ops; everyone else gets 403 Operations.OperatorRequired.

The operator console ​

/ops in the web app (shown to operators only):

  • Tenants — plan, region, licensed modules and feature flags, workspaces, members, sources, models, last activity (the newest audit record) and whether support consent is active. Counts only: no tenant data.
  • Offboard — see below.
  • Support session — start a read-only session in a tenant whose Owner granted consent.
  • Health — the component health of the status endpoint.
  • Operator log — offboarding requests and tombstones, exports, support sessions, newest first.

Module licences are granted with the licence verb.

Offboarding ​

Offboarding deletes a tenant: every row with its tenant_id in every schema — tables are discovered from the PostgreSQL catalog, so a table added by any module later is covered without code changes — plus child rows without a tenant column, the tenant row, the people who belonged to that tenant only, its secrets in the secret store (local or Key Vault) and its cache entries. The tenant's own audit chain goes with it; a tombstone in the platform operator log records who offboarded which tenant, the rows deleted per table and the verification.

From the console (a platform job run by the worker), or from the deployment's command line:

bash
dotnet Keel.Api.dll tenant offboard --tenant acme-corp --confirm acme-corp --export
dotnet Keel.Api.dll tenant verify --tenant-id 0192f0c1-…   # rows left per tenant table (exit code 0 when none)
  1. Export (optional, recommended): a zip of the tenant's metadata — manifest.json and one JSON Lines file per table (models and revisions, glossary, policies, quality rules, the audit chain, …) — written to Keel:Operations:ExportDirectory. Secrets are never exported. The tombstone keeps the file's SHA-256. Hand it to the customer, then delete it (the runbook says within 30 days).
  2. Delete, in one transaction.
  3. Verify: a catalog-driven count proves zero rows remain for the tenant in every table with a tenant_id; the job fails (and the tombstone says tenant.offboarding-incomplete) otherwise.

Backups still hold the tenant until they age out (35 days in the F23 target). The runbook is docs/runbooks/offboarding.md in the repository.

Not built yet

The customer-initiated deletion request with a 7-day grace period (F23 FR5) is not built: offboarding is operator-initiated. After a verified purge every Owner is emailed a certificate of deletion (id and SHA-256 kept in the tombstone; the addresses are kept nowhere). The export is JSON Lines per table, not the model-as-YAML / glossary-CSV formats F23 lists.

Support sessions ​

An operator can look at a customer's tenant only while the customer's Owner has granted time-boxed consent (Settings › Security › Support access: 1 hour to 7 days, with a reason). Then:

  • The operator starts a session from the console with a reason (a ticket number). It lasts at most one hour and never beyond the consent. Revoking consent ends every open session at once.
  • The session is read-only: it sees the tenant as a Viewer (catalog and audit log only — never serving queries); every command is refused in the application pipeline, and every non-GET request at the HTTP edge.
  • The app shows a banner for the whole session.
  • Everything is in the customer's audit log, with the operator as the actor: consent granted and revoked, session started and ended, and every request the session made (Support.SessionViewed, with method, route, path and status; refused changes as Support.SessionChangeRefused).

Without active consent a session cannot start (403 Support.NoConsent), and a session header that does not resolve to the caller's own active session is refused (403 Support.SessionInvalid) rather than ignored.

Status ​

GET /status is public, cached for 10 seconds, and answers 503 when a component is down so a plain HTTP monitor works too:

json
{
  "status": "operational",
  "generatedAt": "2026-09-30T12:00:00Z",
  "components": [
    { "name": "api", "status": "operational" },
    { "name": "database", "status": "operational" },
    { "name": "redis", "status": "operational" },
    { "name": "worker-queue", "status": "operational", "detail": "0 due job(s); oldest waiting 0 s.", "metric": 0 },
    { "name": "scheduler", "status": "operational", "metric": 42 }
  ]
}
  • worker-queue is degraded when a due job has waited more than 5 minutes (down after 50) — workers are stopped or overloaded.
  • scheduler is degraded when its every-minute supervision job has not been queued for 5 minutes (metric is the age in seconds).
  • redis is unknown when no Redis is configured (a cache only).

Status page ​

Keel serves its own public status page at /status-page in the web app (point status.[DOMAIN] at it): six components (App, API & MCP, Scans, Agents gateway, Assistant, Exports & webhooks) with 90-day uptime bars recorded every five minutes by the scheduler, open incidents with their timeline, past incidents and email subscriptions (double opt-in). Operators open incidents and post updates in the operator console (/ops › Status page). Settings: Keel:Status:* (region, other regions, the API URL the worker probes, the link key).

Contact support ​

The help panel's Contact support tab sends the person's text with the tenant, route id, trace id and app version — never data — to Keel:Support:Email, with a copy to the requester (5 requests per person per hour).

Anchor Keel is a working product name: one platform, a shared core and licensed modules (Keel Semantic, Keel Migrate). Documentation built from the repository — facts are derived from code, ADRs and the decisions log.