Skip to content

cayu.evals.reporting

Public Cayu API. Definitions are grouped by their public source module, and definitions from private implementation modules appear on their package’s page; adding a symbol does not move existing links. The import shown at each entry is a supported public namespace. Read the Python API overview and install the required extra.

compare_eval_runs

Section titled “compare_eval_runs”
from cayu.evals import compare_eval_runs

Compare two runs and flag regressions.

score_tolerance (>= 0) is the amount a score may drop before it counts as a regression: a current score below baseline - score_tolerance regresses, so a stochastic wobble (e.g. 0.83 -> 0.82) inside the tolerance no longer fails a baseline comparison every run. A move to a more severe status is always flagged regardless of tolerance.

compare_eval_runs(baseline: EvalRun, current: EvalRun, *, score_tolerance: float=0.0) -> EvalRunComparison

Release source · Runtime guarantees · Capability guides

comparison_to_json

Section titled “comparison_to_json”
from cayu.evals import comparison_to_json
comparison_to_json(comparison: EvalRunComparison, *, indent: int | None=2) -> str

Release source · Runtime guarantees · Capability guides

eval_run_to_json

Section titled “eval_run_to_json”
from cayu.evals import eval_run_to_json
eval_run_to_json(run: EvalRun, *, indent: int | None=2) -> str

Release source · Runtime guarantees · Capability guides

EvalCaseComparison

Section titled “EvalCaseComparison”
from cayu.evals import EvalCaseComparison

Bases: BaseModel.

Fields and attributes

Section titled “Fields and attributes”

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name Type Default
case_id str required
baseline_status EvalStatus | None None
current_status EvalStatus | None None
baseline_score StrictFloat | None Field(default=None, ge=0.0, le=1.0, allow_inf_nan=False)
current_score StrictFloat | None Field(default=None, ge=0.0, le=1.0, allow_inf_nan=False)
regressions tuple[str, ...] Field(default_factory=tuple)

validate_case_id (method)

Section titled “validate_case_id (method)”
EvalCaseComparison.validate_case_id(value: str, info) -> str

validate_regressions (method)

Section titled “validate_regressions (method)”
EvalCaseComparison.validate_regressions(value: tuple[str, ...]) -> tuple[str, ...]

Release source · Runtime guarantees · Capability guides

EvalRunComparison

Section titled “EvalRunComparison”
from cayu.evals import EvalRunComparison

Bases: BaseModel.

Fields and attributes

Section titled “Fields and attributes”

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name Type Default
baseline_run_id str required
current_run_id str required
baseline_suite_id str required
current_suite_id str required
baseline_status EvalStatus required
current_status EvalStatus required
baseline_score StrictFloat | None Field(default=None, ge=0.0, le=1.0, allow_inf_nan=False)
current_score StrictFloat | None Field(default=None, ge=0.0, le=1.0, allow_inf_nan=False)
memory_attribution_support Literal['unsupported'] 'unsupported'
regressions tuple[str, ...] Field(default_factory=tuple)
cases tuple[EvalCaseComparison, ...] Field(default_factory=tuple)

validate_ids (method)

Section titled “validate_ids (method)”
EvalRunComparison.validate_ids(value: str, info) -> str

validate_regressions (method)

Section titled “validate_regressions (method)”
EvalRunComparison.validate_regressions(value: tuple[str, ...]) -> tuple[str, ...]

detach_cases (method)

Section titled “detach_cases (method)”
EvalRunComparison.detach_cases(value)

Release source · Runtime guarantees · Capability guides

load_eval_run

Section titled “load_eval_run”
from cayu.evals import load_eval_run
load_eval_run(path: str | Path) -> EvalRun

Release source · Runtime guarantees · Capability guides

load_trajectory

Section titled “load_trajectory”
from cayu.evals import load_trajectory
load_trajectory(path: str | Path) -> Trajectory

Release source · Runtime guarantees · Capability guides

render_comparison_html

Section titled “render_comparison_html”
from cayu.evals import render_comparison_html
render_comparison_html(comparison: EvalRunComparison) -> str

Release source · Runtime guarantees · Capability guides

render_html_report

Section titled “render_html_report”
from cayu.evals import render_html_report
render_html_report(run: EvalRun) -> str

Release source · Runtime guarantees · Capability guides

trajectory_to_json

Section titled “trajectory_to_json”
from cayu.evals import trajectory_to_json
trajectory_to_json(trajectory: Trajectory, *, indent: int | None=2) -> str

Release source · Runtime guarantees · Capability guides

write_eval_run_json

Section titled “write_eval_run_json”
from cayu.evals import write_eval_run_json
write_eval_run_json(run: EvalRun, path: str | Path) -> None

Release source · Runtime guarantees · Capability guides

write_html_report

Section titled “write_html_report”
from cayu.evals import write_html_report
write_html_report(run: EvalRun, path: str | Path) -> None

Release source · Runtime guarantees · Capability guides

write_trajectory_json

Section titled “write_trajectory_json”
from cayu.evals import write_trajectory_json
write_trajectory_json(trajectory: Trajectory, path: str | Path) -> None

Release source · Runtime guarantees · Capability guides