Eval Gate Operations

Eval Gate Operations

Operational guide for the eval-gate configuration the Agent CI/CD pipeline exposes: who owns what, how a vertical onboards, how judge templates are deprecated, and how a bad change is rolled back or a post-deploy regression is handled.

The eval gate itself (schema, retrieval API, gate engine) is documented in docs/specs/PC-5.md. This guide is the operations layer: the policy and procedure a human follows, not the code.

Ownership model

The pipeline supplies eval-gate configuration (judge templates, threshold conventions, the retrieval + gate API). It does not own a vertical’s data project. A vertical brings its own Opik project; the pipeline supplies the templates and the API to wire eval into it.

Surface	Owned by	Notes
Judge rule files (configs/rules/*.yaml)	Platform (this pipeline)	Reusable judge templates verticals instantiate
Threshold conventions (evaluation_manifest.yaml thresholds:)	Platform	Per-composite, per-milestone, 0.0-1.0 normalized scale
Retrieval + gate API (get_metric_by_id, get_threshold, evaluate_gate)	Platform	Read-only against a project; never writes to it
A vertical’s Opik project + its traces	The vertical	The pipeline reads via the API; it holds no write access
Which judges apply to which category	The vertical (via the manifest it maintains)	Reviewed on the normal PR surface

No-write guarantee. The retrieval API reads judge config from the file-backed registry and resolves thresholds from the manifest + composites. It constructs no Opik client and performs no write to any vertical’s project. A vertical onboards by syncing the templates into its own project (via the rules CLI) and calling the API; it does not fork a project starter.

How a vertical onboards

1. Sync the judge templates it needs into its own Opik project with the rules CLI (cli/opik/rules.py create).
2. Declare which judges apply to which category in the evaluation manifest.
3. Call the retrieval API (get_metrics_for_category, get_metric_by_id) and, when gating a rollout, evaluate_gate(milestone, sample, judge_ids).

The manifest may reference XML prompt-component IDs as the judge source; the source-of-truth shift from standalone rule files to XML components is owned by the prompt-variant pipeline (PC-6), not this pipeline. This pipeline consumes whichever the manifest declares.

Judge-template deprecation policy

A judge template change ranges from a typo fix to a rubric rewrite that moves every score. The discipline scales with the blast radius:

• Minor change (wording, a clarifying example): normal PR review.

• Major change (a rubric rewrite that shifts the score distribution): the author MUST acknowledge re-baselining in the PR. The PR template carries a checkbox:

  • This change alters a judge’s scoring behavior. Thresholds were re-baselined (or a follow-up to re-baseline is filed) and the baseline_source / calibration_ref on affected thresholds are updated.

  A cross-vertical template change additionally requires platform-team review.

Why a checkbox, not a CI script (v1). “Major rubric change” is a judgment about intent and distribution shift, not a property a deterministic linter can decide. A CI rubric-diff detector would either over-fire on harmless wording or miss a semantically large change behind a small diff. The schema validator (validate_rule_file) enforces what is deterministic — the file conforms to the schema; the deprecation judgment stays a human review gate. If churn data later shows a rule of thumb that catches major changes reliably, automate then.

Post-deploy regression response

When a judge or gate surfaces a regression after deploy, the response is tiered by which floor was breached:

Trigger	Response
Safety / refusal floor breach (a safety-composite judge below its floor)	Automatic ramp-down to 0% on the affected agent/component + page on-call. Ramp-down (not rollback) is the safer failure mode — reversible if the judge run was a false positive.
Quality floor breach (a quality-composite judge below its floor)	Slack alert + Grafana annotation + scheduled triage within 24h. No page; a quality regression does not justify a 3am response.

Both tiers append an entry to docs/post-deploy-regressions.md for trend analysis.

Automatic ramp-down depends on the sub-agent lifecycle ramp primitive (PF-1). Until that lands, the automatic path degrades to: alert fires, an operator performs the ramp-down manually via the feature-flag tool. The tiering and the tracking-note discipline apply regardless; only the automatic actuation waits on PF-1.

Lifecycle-transition audit (recording the dev → test → promote → rollback transition itself) is not part of this pipeline. This pipeline observes its own eval gates; transition audit is owned by the rollout/trace-store work (tracked separately under the PC-6 / PS-5 line).

Rollback procedure

A bad judge or threshold change is rolled back without redeploying the agent:

1. Revert the config in git. Revert the commit that changed the rule file or the manifest threshold. This is a normal PR (or a fast-forward revert on an incident).
2. Sync the reverted config into the Opik project with the rules CLI (cli/opik/rules.py create --name <rule> --file configs/rules/<rule>.yaml).
3. Refresh the runtime cache so the change takes effect immediately: call the cache-refresh affordance (refresh_caches() in the judge-metric module). No agent redeploy is required — the manifest + rule files are read through the cache, and the refresh invalidates it.

Restore-time contract. The above restores the prior judge behavior within 15 minutes (revert + sync + refresh; no build, no deploy). The concrete measurement is operational and tracked against incidents.

Scope note. The refresh covers the manifest (thresholds) and the rule files (read uncached per lookup). It does NOT reload the composite definitions or fallback thresholds in agent_config.yaml — a change there needs a process restart. Threshold edits flow through the manifest, which the refresh does cover.