cayu.evals.runner

Public Cayu API. Definitions are grouped by their public source module, and definitions from private implementation modules appear on their package’s page; adding a symbol does not move existing links. The import shown at each entry is a supported public namespace. Read the Python API overview and install the required extra.

`EvalCase`

from cayu.evals import EvalCase

Bases: BaseModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
`id`	`str`	`required`
`request`	`RunRequest`	`required`
`assertions`	`list[EvalAssertion]`	`Field(default_factory=list)`
`metadata`	`dict[str, Any]`	`Field(default_factory=dict)`

validate_id (method)

EvalCase.validate_id(value: str, info) -> str

copy_request (method)

EvalCase.copy_request(value: RunRequest) -> RunRequest

validate_assertions (method)

EvalCase.validate_assertions(value) -> list[EvalAssertion]

copy_metadata (method)

EvalCase.copy_metadata(value: dict[str, Any]) -> dict[str, Any]

Release source · Runtime guarantees · Capability guides

`EvalPlan`

from cayu.evals import EvalPlan

Trusted project eval entry point in exactly one execution mode.

Decorators: @dataclass(frozen=True).

Fields and attributes

Name	Type	Default
`app`	`CayuApp \| None`	`None`
`suite`	`EvalSuite \| None`	`None`
`corpus_target`	`CorpusTarget \| None`	`None`
`workflow_target`	`WorkflowEvalTarget \| None`	`None`
`execution_profile_policy`	`EvalExecutionProfilePolicyV1 \| None`	`None`

Release source · Runtime guarantees · Capability guides

`EvalSuite`

from cayu.evals import EvalSuite

Bases: BaseModel.

Fields and attributes

Name	Type	Default
`id`	`str`	`required`
`cases`	`list[EvalCase]`	`required`
`metadata`	`dict[str, Any]`	`Field(default_factory=dict)`

validate_id (method)

EvalSuite.validate_id(value: str, info) -> str

validate_cases (method)

EvalSuite.validate_cases(value) -> list[EvalCase]

copy_metadata (method)

EvalSuite.copy_metadata(value: dict[str, Any]) -> dict[str, Any]

Release source · Runtime guarantees · Capability guides

`evaluate_assertions`

from cayu.evals import evaluate_assertions

Run assertions against a Trajectory — the replay entry point.

Wraps the trajectory in an EvalContext and evaluates the assertions, returning their results. Pair with load_trajectory to re-check a saved run without a live runtime: evaluate_assertions(load_trajectory(path), assertions).

async evaluate_assertions(trajectory: Trajectory, assertions: Iterable[EvalAssertion], *, suite_id: str='replay', case_id: str='replay') -> tuple[EvalAssertionResult, ...]

Release source · Runtime guarantees · Capability guides

`run_eval_case`

from cayu.evals import run_eval_case

Run one case one or more times and retain every concrete trial.

Every trial receives a fresh concrete session ID; an authored request session ID is never reused as concrete state, but remains result provenance and the causal-accounting fallback. Aggregate fields are deterministic projections of the ordered result.trials tuple. No trial is selected as a representative and no trial evidence is overwritten.

async run_eval_case(app: CayuApp, case: EvalCase, *, suite_id: str, retain_trajectory: bool=False, retain_final_output: bool=True, timeout_seconds: float | None=None, trials: int=1, trial_policy: EvalSuiteTrialPolicyV1 | None=None) -> EvalCaseResult

Release source · Runtime guarantees · Capability guides

`run_eval_plan`

from cayu.evals import run_eval_plan

async run_eval_plan(plan: EvalPlan, *, corpus: EvalCorpusDocument | None=None, suite_id: str | None=None, retain_trajectory: bool=False, max_concurrency: int | None=None, case_timeout_seconds: float | None=None, trials: int | None=None, execution_capacity: EvalExecutionCapacity | None=None) -> EvalRun | CorpusExecutionResult

Release source · Runtime guarantees · Capability guides

`run_eval_suite`

from cayu.evals import run_eval_suite

Run every case in the suite and aggregate the results.

max_concurrency runs up to that many concrete case trials at once. When omitted, an explicit trial policy wins, then app.config.evals.max_concurrency (default 1 = sequential). Results always keep suite and trial order. Concurrent suites reject positional ScriptedModelProvider instances shared by multiple trials. Unused registrations and isolated per-case providers are allowed. Overlapping invocations reserve participating positional providers even at concurrency 1. Request-aware factories support shared providers.

case_timeout_seconds bounds each case’s run; a case that exceeds it is cancelled and recorded as EvalStatus.ERROR instead of stalling the suite.

trials runs each case N times and retains every outcome. When supplied, trial_policy fixes the minimum-pass decision and concurrency ceiling before execution; otherwise an all-pass policy records the requested concurrency.

retain_final_output=False discards raw trial output after assertions have evaluated. It cannot be combined with trajectory retention because a retained trajectory must remain lossless.

async run_eval_suite(app: CayuApp, suite: EvalSuite, *, retain_trajectory: bool=False, retain_final_output: bool=True, max_concurrency: int | None=None, case_timeout_seconds: float | None=None, trials: int=1, execution_capacity: EvalExecutionCapacity | None=None, trial_policy: EvalSuiteTrialPolicyV1 | None=None) -> EvalRun

Release source · Runtime guarantees · Capability guides

`run_workflow_eval_suite`

from cayu.evals import run_workflow_eval_suite

Run a typed application-owned workflow as the root eval target.

async run_workflow_eval_suite(target: WorkflowEvalTarget, suite: EvalSuite, *, retain_trajectory: bool=False, retain_final_output: bool=True, max_concurrency: int | None=None, case_timeout_seconds: float | None=None, trials: int=1, execution_capacity: EvalExecutionCapacity | None=None, trial_policy: EvalSuiteTrialPolicyV1 | None=None) -> EvalRun

Release source · Runtime guarantees · Capability guides