Evaluation in CI

Run deterministic tests on every change. Run live-provider evaluations when the change requires semantic evidence, using explicit credentials, concurrency and cost limits. Keep the application and judge configuration with the result so comparisons have a defined meaning.

Establish the baseline

Create a representative suite, run it, inspect individual cases and approve the resulting immutable baseline. Include failure and authorization cases alongside successful requests. Stable case IDs identify the logical contract; revisions identify its contents.

Save candidate and baseline result exports, then compare them:

cayu eval compare baseline.eval-result.json result.eval-result.json \
  --json --output comparison.json
cayu eval report result.eval-result.json --html --output report.html

Comparison exits 0 for a compatible result without regression, 1 for a compatible regression and 2 when the result cannot support a conclusive decision. Treat 2 as a check requiring attention. Changed evidence, authority, assertion semantics, judge configuration or trial policy can make two results incomparable.

Retain the proof

Archive suite revision, candidate version, result JSON, comparison and report. Record whether the candidate was newly executed or a retained session was graded. Retain every repeated trial, including evaluator errors and unavailable evidence.

Use first evaluation for authoring and launch, judges for semantic quality, and eval CLI for all flags.