cayu.evals.assertions

Public Cayu API. Definitions are grouped by their public source module, and definitions from private implementation modules appear on their package’s page; adding a symbol does not move existing links. The import shown at each entry is a supported public namespace. Read the Python API overview and install the required extra.

ArtifactCreated

from cayu.evals import ArtifactCreated

Bases: EvalAssertion.

name (property)

ArtifactCreated.name() -> str

assertion_revision (property)

ArtifactCreated.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

ArtifactCreated.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

ArtifactCreated.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async ArtifactCreated.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

ArtifactCreated.required_probes() -> ProbeRequirements

passed (method)

ArtifactCreated.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

ArtifactCreated.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

ArtifactCreated.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

ArtifactCreated.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

ArtifactCreated.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

ArtifactCreated.__init__(*, filename: str | None=None, content_type: str | None=None, scope: ArtifactScope | str | None=None, min_count: int=1) -> None

Release source · Runtime guarantees · Capability guides

ChildSessionCompleted

from cayu.evals import ChildSessionCompleted

Bases: EvalAssertion.

name (property)

ChildSessionCompleted.name() -> str

assertion_revision (property)

ChildSessionCompleted.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

ChildSessionCompleted.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

ChildSessionCompleted.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async ChildSessionCompleted.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

ChildSessionCompleted.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

ChildSessionCompleted.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

ChildSessionCompleted.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

ChildSessionCompleted.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

ChildSessionCompleted.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

ChildSessionCompleted.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

ChildSessionCompleted.__init__(*, agent_name: str | None=None, min_count: int=1) -> None

Release source · Runtime guarantees · Capability guides

EvalAssertion

from cayu.evals import EvalAssertion

Assertion over a completed Cayu eval case.

Bases: ABC.

name (property)

EvalAssertion.name() -> str

assertion_revision (property)

EvalAssertion.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

EvalAssertion.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

EvalAssertion.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async EvalAssertion.evaluate(context: EvalContext) -> EvalAssertionResult

Evaluate this assertion against a Cayu runtime context.

required_probes (method)

EvalAssertion.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

EvalAssertion.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

EvalAssertion.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

EvalAssertion.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

EvalAssertion.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

EvalAssertion.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

Release source · Runtime guarantees · Capability guides

EventNotOccurred

from cayu.evals import EventNotOccurred

Bases: EventOccurred.

name (property)

EventNotOccurred.name() -> str

assertion_revision (property)

EventNotOccurred.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

EventNotOccurred.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

EventNotOccurred.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async EventNotOccurred.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

EventNotOccurred.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

EventNotOccurred.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

EventNotOccurred.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

EventNotOccurred.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

EventNotOccurred.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

EventNotOccurred.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

EventNotOccurred.__init__(event_type: EventType | str) -> None

Release source · Runtime guarantees · Capability guides

EventOccurred

from cayu.evals import EventOccurred

Bases: EvalAssertion.

name (property)

EventOccurred.name() -> str

assertion_revision (property)

EventOccurred.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

EventOccurred.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

EventOccurred.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async EventOccurred.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

EventOccurred.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

EventOccurred.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

EventOccurred.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

EventOccurred.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

EventOccurred.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

EventOccurred.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

EventOccurred.__init__(event_type: EventType | str, *, min_count: int=1, max_count: int | None=None) -> None

Release source · Runtime guarantees · Capability guides

EventPayloadContains

from cayu.evals import EventPayloadContains

Bases: EvalAssertion.

name (property)

EventPayloadContains.name() -> str

assertion_revision (property)

EventPayloadContains.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

EventPayloadContains.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

EventPayloadContains.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async EventPayloadContains.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

EventPayloadContains.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

EventPayloadContains.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

EventPayloadContains.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

EventPayloadContains.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

EventPayloadContains.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

EventPayloadContains.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

EventPayloadContains.__init__(event_type: EventType | str, expected: Mapping[str, Any], *, min_count: int=1) -> None

Release source · Runtime guarantees · Capability guides

FinalOutputContains

from cayu.evals import FinalOutputContains

Bases: EvalAssertion.

name (property)

FinalOutputContains.name() -> str

assertion_revision (property)

FinalOutputContains.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

FinalOutputContains.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

FinalOutputContains.reads_final_output_only() -> bool

evaluate (method)

async FinalOutputContains.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

FinalOutputContains.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

FinalOutputContains.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

FinalOutputContains.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

FinalOutputContains.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

FinalOutputContains.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

FinalOutputContains.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

FinalOutputContains.__init__(text: str) -> None

Release source · Runtime guarantees · Capability guides

FinalOutputMatches

from cayu.evals import FinalOutputMatches

Bases: EvalAssertion.

name (property)

FinalOutputMatches.name() -> str

assertion_revision (property)

FinalOutputMatches.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

FinalOutputMatches.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

FinalOutputMatches.reads_final_output_only() -> bool

evaluate (method)

async FinalOutputMatches.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

FinalOutputMatches.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

FinalOutputMatches.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

FinalOutputMatches.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

FinalOutputMatches.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

FinalOutputMatches.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

FinalOutputMatches.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

FinalOutputMatches.__init__(pattern: str) -> None

Release source · Runtime guarantees · Capability guides

MaxEstimatedCost

from cayu.evals import MaxEstimatedCost

Bases: EvalAssertion.

name (property)

MaxEstimatedCost.name() -> str

assertion_revision (property)

MaxEstimatedCost.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

MaxEstimatedCost.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

MaxEstimatedCost.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async MaxEstimatedCost.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

MaxEstimatedCost.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

MaxEstimatedCost.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

MaxEstimatedCost.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

MaxEstimatedCost.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

MaxEstimatedCost.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

MaxEstimatedCost.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

MaxEstimatedCost.__init__(maximum: Decimal | str | int | float, pricing: PriceBook, *, currency: str='USD') -> None

Release source · Runtime guarantees · Capability guides

MaxModelSteps

from cayu.evals import MaxModelSteps

Bases: EvalAssertion.

name (property)

MaxModelSteps.name() -> str

assertion_revision (property)

MaxModelSteps.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

MaxModelSteps.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

MaxModelSteps.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async MaxModelSteps.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

MaxModelSteps.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

MaxModelSteps.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

MaxModelSteps.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

MaxModelSteps.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

MaxModelSteps.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

MaxModelSteps.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

MaxModelSteps.__init__(maximum: int) -> None

Release source · Runtime guarantees · Capability guides

MaxToolCalls

from cayu.evals import MaxToolCalls

Bases: EvalAssertion.

name (property)

MaxToolCalls.name() -> str

assertion_revision (property)

MaxToolCalls.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

MaxToolCalls.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

MaxToolCalls.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async MaxToolCalls.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

MaxToolCalls.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

MaxToolCalls.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

MaxToolCalls.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

MaxToolCalls.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

MaxToolCalls.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

MaxToolCalls.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

MaxToolCalls.__init__(maximum: int) -> None

Release source · Runtime guarantees · Capability guides

MaxTotalTokens

from cayu.evals import MaxTotalTokens

Bases: EvalAssertion.

name (property)

MaxTotalTokens.name() -> str

assertion_revision (property)

MaxTotalTokens.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

MaxTotalTokens.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

MaxTotalTokens.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async MaxTotalTokens.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

MaxTotalTokens.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

MaxTotalTokens.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

MaxTotalTokens.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

MaxTotalTokens.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

MaxTotalTokens.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

MaxTotalTokens.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

MaxTotalTokens.__init__(maximum: int) -> None

Release source · Runtime guarantees · Capability guides

SessionCompleted

from cayu.evals import SessionCompleted

Bases: SessionStatusIs.

name (property)

SessionCompleted.name() -> str

assertion_revision (property)

SessionCompleted.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

SessionCompleted.evaluates_failed_session() -> bool

reads_final_output_only (property)

SessionCompleted.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async SessionCompleted.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

SessionCompleted.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

SessionCompleted.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

SessionCompleted.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

SessionCompleted.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

SessionCompleted.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

SessionCompleted.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

SessionCompleted.__init__() -> None

Release source · Runtime guarantees · Capability guides

SessionFailed

from cayu.evals import SessionFailed

Bases: SessionStatusIs.

name (property)

SessionFailed.name() -> str

assertion_revision (property)

SessionFailed.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

SessionFailed.evaluates_failed_session() -> bool

reads_final_output_only (property)

SessionFailed.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async SessionFailed.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

SessionFailed.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

SessionFailed.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

SessionFailed.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

SessionFailed.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

SessionFailed.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

SessionFailed.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

SessionFailed.__init__() -> None

Release source · Runtime guarantees · Capability guides

SessionInterrupted

from cayu.evals import SessionInterrupted

Bases: SessionStatusIs.

name (property)

SessionInterrupted.name() -> str

assertion_revision (property)

SessionInterrupted.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

SessionInterrupted.evaluates_failed_session() -> bool

reads_final_output_only (property)

SessionInterrupted.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async SessionInterrupted.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

SessionInterrupted.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

SessionInterrupted.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

SessionInterrupted.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

SessionInterrupted.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

SessionInterrupted.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

SessionInterrupted.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

SessionInterrupted.__init__() -> None

Release source · Runtime guarantees · Capability guides

SessionStatusIs

from cayu.evals import SessionStatusIs

Bases: EvalAssertion.

name (property)

SessionStatusIs.name() -> str

assertion_revision (property)

SessionStatusIs.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

SessionStatusIs.evaluates_failed_session() -> bool

reads_final_output_only (property)

SessionStatusIs.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async SessionStatusIs.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

SessionStatusIs.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

SessionStatusIs.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

SessionStatusIs.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

SessionStatusIs.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

SessionStatusIs.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

SessionStatusIs.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

SessionStatusIs.__init__(status: SessionStatus | str) -> None

Release source · Runtime guarantees · Capability guides

ToolArgsContain

from cayu.evals import ToolArgsContain

Bases: EvalAssertion.

name (property)

ToolArgsContain.name() -> str

assertion_revision (property)

ToolArgsContain.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

ToolArgsContain.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

ToolArgsContain.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async ToolArgsContain.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

ToolArgsContain.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

ToolArgsContain.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

ToolArgsContain.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

ToolArgsContain.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

ToolArgsContain.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

ToolArgsContain.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

ToolArgsContain.__init__(tool_name: str, expected: Mapping[str, Any]) -> None

Release source · Runtime guarantees · Capability guides

ToolCalled

from cayu.evals import ToolCalled

Bases: EvalAssertion.

name (property)

ToolCalled.name() -> str

assertion_revision (property)

ToolCalled.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

ToolCalled.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

ToolCalled.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async ToolCalled.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

ToolCalled.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

ToolCalled.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

ToolCalled.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

ToolCalled.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

ToolCalled.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

ToolCalled.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

ToolCalled.__init__(tool_name: str, *, min_count: int=1, max_count: int | None=None) -> None

Release source · Runtime guarantees · Capability guides

ToolNotCalled

from cayu.evals import ToolNotCalled

Bases: ToolCalled.

name (property)

ToolNotCalled.name() -> str

assertion_revision (property)

ToolNotCalled.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

ToolNotCalled.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

ToolNotCalled.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async ToolNotCalled.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

ToolNotCalled.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

ToolNotCalled.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

ToolNotCalled.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

ToolNotCalled.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

ToolNotCalled.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

ToolNotCalled.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

ToolNotCalled.__init__(tool_name: str) -> None

Release source · Runtime guarantees · Capability guides

ToolResultContains

from cayu.evals import ToolResultContains

Bases: EvalAssertion.

name (property)

ToolResultContains.name() -> str

assertion_revision (property)

ToolResultContains.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

ToolResultContains.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

ToolResultContains.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async ToolResultContains.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

ToolResultContains.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

ToolResultContains.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

ToolResultContains.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

ToolResultContains.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

ToolResultContains.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

ToolResultContains.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

ToolResultContains.__init__(tool_name: str, text: str) -> None

Release source · Runtime guarantees · Capability guides

ToolsCalledInOrder

from cayu.evals import ToolsCalledInOrder

Require exact model-requested tool names in transcript order.

Bases: EvalAssertion.

name (property)

ToolsCalledInOrder.name() -> str

assertion_revision (property)

ToolsCalledInOrder.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

ToolsCalledInOrder.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

ToolsCalledInOrder.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async ToolsCalledInOrder.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

ToolsCalledInOrder.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

ToolsCalledInOrder.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

ToolsCalledInOrder.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

ToolsCalledInOrder.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

ToolsCalledInOrder.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

ToolsCalledInOrder.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

ToolsCalledInOrder.__init__(tool_names: Iterable[str]) -> None

Release source · Runtime guarantees · Capability guides

TranscriptContains

from cayu.evals import TranscriptContains

Bases: EvalAssertion.

name (property)

TranscriptContains.name() -> str

assertion_revision (property)

TranscriptContains.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

TranscriptContains.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

TranscriptContains.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async TranscriptContains.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

TranscriptContains.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

TranscriptContains.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

TranscriptContains.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

TranscriptContains.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

TranscriptContains.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

TranscriptContains.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

TranscriptContains.__init__(text: str) -> None

Release source · Runtime guarantees · Capability guides

UsageRecorded

from cayu.evals import UsageRecorded

Bases: EvalAssertion.

name (property)

UsageRecorded.name() -> str

assertion_revision (property)

UsageRecorded.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

UsageRecorded.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

UsageRecorded.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async UsageRecorded.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

UsageRecorded.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

UsageRecorded.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

UsageRecorded.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

UsageRecorded.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

UsageRecorded.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

UsageRecorded.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

UsageRecorded.__init__(*, min_total_tokens: int=1) -> None

Release source · Runtime guarantees · Capability guides

WorkspaceFileContains

from cayu.evals import WorkspaceFileContains

Assert a workspace file contains expected text when the case finishes.

Like WorkspaceFileExists, this reads the workspace as it stands at case end, not “produced by this case”: a file left by an earlier case in a shared workspace counts. Register an environment factory for per-case isolation. See docs/evals.md “Workspace isolation”.

Bases: EvalAssertion.

name (property)

WorkspaceFileContains.name() -> str

assertion_revision (property)

WorkspaceFileContains.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

WorkspaceFileContains.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

WorkspaceFileContains.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async WorkspaceFileContains.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

WorkspaceFileContains.required_probes() -> ProbeRequirements

passed (method)

WorkspaceFileContains.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

WorkspaceFileContains.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

WorkspaceFileContains.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

WorkspaceFileContains.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

WorkspaceFileContains.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

WorkspaceFileContains.__init__(path: str, text: str, *, encoding: str='utf-8') -> None

Release source · Runtime guarantees · Capability guides

WorkspaceFileExists

from cayu.evals import WorkspaceFileExists

Assert a path is present in the workspace when the case finishes.

This means “present now”, not “this case created it”: cases in a suite share the app’s workspace unless the caller registers an environment factory (fresh workspace per case), so a file left by an earlier case will satisfy this. See docs/evals.md “Workspace isolation”.

Bases: EvalAssertion.

name (property)

WorkspaceFileExists.name() -> str

assertion_revision (property)

WorkspaceFileExists.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

WorkspaceFileExists.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

WorkspaceFileExists.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async WorkspaceFileExists.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

WorkspaceFileExists.required_probes() -> ProbeRequirements

passed (method)

WorkspaceFileExists.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

WorkspaceFileExists.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

WorkspaceFileExists.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

WorkspaceFileExists.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

WorkspaceFileExists.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

WorkspaceFileExists.__init__(path: str) -> None

Release source · Runtime guarantees · Capability guides