cayu.evals.models

Public Cayu API. Definitions are grouped by their public source module, and definitions from private implementation modules appear on their package’s page; adding a symbol does not move existing links. The import shown at each entry is a supported public namespace. Read the Python API overview and install the required extra.

EVAL_SCHEMA_VERSION

from cayu.evals import EVAL_SCHEMA_VERSION

EVAL_SCHEMA_VERSION = 11

Release source · Runtime guarantees · Capability guides

EvalAssertionResult

from cayu.evals import EvalAssertionResult

Bases: BaseModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
name	str	required
assertion_revision	str | None	None
outcome	EvalOutcome	required
score	StrictFloat | None	Field(default=None, ge=0.0, le=1.0)
threshold	StrictFloat | None	Field(default=None, ge=0.0, le=1.0)
message	str	''
metadata	dict[str, Any]	Field(default_factory=dict)
cost_summary	SessionCostSummary | None	None

check_outcome_score_consistency (method)

EvalAssertionResult.check_outcome_score_consistency() -> EvalAssertionResult

passed (property)

EvalAssertionResult.passed() -> bool

Return the boolean view used by assertion consumers.

validate_name (method)

EvalAssertionResult.validate_name(value: str, info) -> str

validate_assertion_revision (method)

EvalAssertionResult.validate_assertion_revision(value: str | None, info) -> str | None

validate_message (method)

EvalAssertionResult.validate_message(value: str, info) -> str

copy_metadata (method)

EvalAssertionResult.copy_metadata(value)

copy_cost_summary (method)

EvalAssertionResult.copy_cost_summary(value: SessionCostSummary | None) -> SessionCostSummary | None

Release source · Runtime guarantees · Capability guides

EvalCaseContractV1

from cayu.evals import EvalCaseContractV1

One immutable case revision selected before an eval run starts.

Bases: BaseModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
case_id	StrictStr	required
case_revision	StrictStr	required

validate_case_id (method)

EvalCaseContractV1.validate_case_id(value: str, info) -> str

validate_case_revision (method)

EvalCaseContractV1.validate_case_revision(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

EvalCaseResult

from cayu.evals import EvalCaseResult

Bases: BaseModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
case_id	str	required
status	EvalStatus	required
authored_session_id	str | None	None
trials	tuple[EvalTrialResult, ...]	Field(min_length=1)
trial_policy	EvalSuiteTrialPolicyV1	required
score	StrictFloat | None	Field(default=None, ge=0.0, le=1.0)
assertions	tuple[EvalAssertionResult, ...]	Field(default_factory=tuple)
error	str | None	None
unavailable_reason	str | None	None
started_at	datetime	required
completed_at	datetime	required
duration_ms	StrictInt	Field(default=0, ge=0)
metadata	dict[str, Any]	Field(default_factory=dict)

from_trials (method)

EvalCaseResult.from_trials(*, case_id: str, trials: Iterable[EvalTrialResult], authored_session_id: str | None=None, started_at: datetime | None=None, completed_at: datetime | None=None, metadata: dict[str, Any] | None=None, trial_policy: EvalSuiteTrialPolicyV1 | None=None) -> EvalCaseResult

validate_case_id (method)

EvalCaseResult.validate_case_id(value: str, info) -> str

validate_session_id (method)

EvalCaseResult.validate_session_id(value: str | None, info) -> str | None

normalize_error (method)

EvalCaseResult.normalize_error(value: object) -> str | None

copy_json_data (method)

EvalCaseResult.copy_json_data(value, info)

copy_trial_policy (method)

EvalCaseResult.copy_trial_policy(value: object) -> object

validate_aggregate_contract (method)

EvalCaseResult.validate_aggregate_contract() -> EvalCaseResult

Release source · Runtime guarantees · Capability guides

EvalContext

from cayu.evals import EvalContext

The assertion’s view of a run: the Trajectory under test + case identity.

Where Trajectory is the serializable run record, EvalContext is what an assertion’s evaluate() receives — it exposes the trajectory’s data (session / events / transcript / usage_summary / final_output / probes) plus the suite_id / case_id / metadata, and carries no live app handle (assertions run offline). root_evidence_available is true by default for deliberately synthetic assertion contexts; public replay and fresh-run entry points set it from the durable root session so missing evidence cannot become a conclusive empty observation. It is also the intended home for future dataset expectations (a reference of expected values an assertion compares the trajectory against).

Decorators: @dataclass(frozen=True).

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
trajectory	Trajectory	required
suite_id	str	required
case_id	str	required
metadata	dict[str, Any]	required
root_evidence_available	bool	True

session (property)

EvalContext.session() -> Session | None

events (property)

EvalContext.events() -> tuple[Event, ...]

transcript (property)

EvalContext.transcript() -> tuple[Message, ...]

usage_summary (property)

EvalContext.usage_summary() -> SessionUsageSummary | None

final_output (property)

EvalContext.final_output() -> str

probes (property)

EvalContext.probes() -> TrajectoryProbes

probe_session (property)

EvalContext.probe_session() -> Session | None

Session that owns the unambiguous structural probe projection.

Release source · Runtime guarantees · Capability guides

EvalOutcome

from cayu.evals import EvalOutcome

One evaluated assertion outcome.

unavailable means the assertion could not obtain complete evidence; error means execution or evaluation failed. Neither has a numeric score.

Bases: StrEnum.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
PASSED	class attribute	'passed'
FAILED	class attribute	'failed'
UNAVAILABLE	class attribute	'unavailable'
ERROR	class attribute	'error'

Release source · Runtime guarantees · Capability guides

EvalRun

from cayu.evals import EvalRun

Bases: BaseModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[11]	EVAL_SCHEMA_VERSION
run_id	str	Field(default_factory=lambda: str(uuid4()))
suite_id	str	required
status	EvalStatus	required
score	StrictFloat | None	Field(default=None, ge=0.0, le=1.0)
cases	tuple[EvalCaseResult, ...]	Field(min_length=1)
started_at	datetime	Field(default_factory=lambda: datetime.now(UTC))
completed_at	datetime	Field(default_factory=lambda: datetime.now(UTC))
duration_ms	StrictInt	Field(default=0, ge=0)
metadata	dict[str, Any]	Field(default_factory=dict)
run_contract	EvalRunContractV1 | EvalRunContractV2 | None	None

require_versioned_memory_evidence (method)

EvalRun.require_versioned_memory_evidence(value: object) -> object

Reject versioned mappings that silently omit required memory evidence.

Standalone Python EvalTrialResult construction retains its explicit typed-unavailable default for extension boundaries. A durable EvalRun mapping has a schema discriminator, so absence there is malformed rather than evidence that may be inferred as missing.

validate_ids (method)

EvalRun.validate_ids(value: str, info) -> str

copy_metadata (method)

EvalRun.copy_metadata(value: dict[str, Any]) -> dict[str, Any]

revalidate_run_contract (method)

EvalRun.revalidate_run_contract(value)

validate_aggregate_contract (method)

EvalRun.validate_aggregate_contract() -> EvalRun

Release source · Runtime guarantees · Capability guides

EvalRunContractV1

from cayu.evals import EvalRunContractV1

Portable execution contract fixed before a corpus-backed run dispatches.

Bases: BaseModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[1]	1
corpus_revision	StrictStr	required
target_key	StrictStr	required
suite_id	StrictStr	required
suite_revision	StrictStr	required
evidence_policy_revision	StrictStr	required
pricing_profile_fingerprint	StrictStr | None	None
trials	StrictInt	Field(ge=1, le=100)
timeout_seconds	StrictInt	Field(ge=1, le=3600)
cases	tuple[EvalCaseContractV1, ...]	Field(min_length=1, max_length=_EVAL_RUN_CONTRACT_MAX_CASES)

validate_schema_version_type (method)

EvalRunContractV1.validate_schema_version_type(value: object) -> object

validate_ids (method)

EvalRunContractV1.validate_ids(value: str, info) -> str

validate_revision (method)

EvalRunContractV1.validate_revision(value: str | None, info) -> str | None

revalidate_cases (method)

EvalRunContractV1.revalidate_cases(value)

validate_case_contract (method)

EvalRunContractV1.validate_case_contract() -> EvalRunContractV1

Release source · Runtime guarantees · Capability guides

EvalRunContractV2

from cayu.evals import EvalRunContractV2

Portable execution contract including the immutable suite trial policy.

Bases: BaseModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[2]	2
corpus_revision	StrictStr	required
target_key	StrictStr	required
suite_id	StrictStr	required
suite_revision	StrictStr	required
evidence_policy_revision	StrictStr	required
pricing_profile_fingerprint	StrictStr | None	None
trials	StrictInt	Field(ge=1, le=100)
timeout_seconds	StrictInt	Field(ge=1, le=3600)
trial_policy	EvalSuiteTrialPolicyV1	required
cases	tuple[EvalCaseContractV1, ...]	Field(min_length=1, max_length=_EVAL_RUN_CONTRACT_MAX_CASES)

validate_schema_version_type (method)

EvalRunContractV2.validate_schema_version_type(value: object) -> object

validate_ids (method)

EvalRunContractV2.validate_ids(value: str, info) -> str

validate_revision (method)

EvalRunContractV2.validate_revision(value: str | None, info) -> str | None

copy_trial_policy (method)

EvalRunContractV2.copy_trial_policy(value: object) -> object

revalidate_cases (method)

EvalRunContractV2.revalidate_cases(value)

validate_contract (method)

EvalRunContractV2.validate_contract() -> EvalRunContractV2

Release source · Runtime guarantees · Capability guides

EvalStatus

from cayu.evals import EvalStatus

Bases: StrEnum.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
PASSED	class attribute	'passed'
FAILED	class attribute	'failed'
UNAVAILABLE	class attribute	'unavailable'
ERROR	class attribute	'error'
SKIPPED	class attribute	'skipped'

Release source · Runtime guarantees · Capability guides

EvalTrialResult

from cayu.evals import EvalTrialResult

Lossless result for one concrete, independently executed case trial.

Bases: BaseModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
trial_number	StrictInt	Field(ge=1)
status	EvalStatus	required
session_id	str | None	None
score	StrictFloat | None	Field(default=None, ge=0.0, le=1.0)
final_output	str	''
structured_output	dict[str, Any] | None	Field(default=None, exclude_if=lambda value: value is None)
assertions	tuple[EvalAssertionResult, ...]	Field(default_factory=tuple)
error	str | None	None
unavailable_reason	str | None	None
operation_outcomes	OperationOutcomeSummary | None	Field(default=None, exclude_if=lambda value: value is None)
execution_status	Literal['completed', 'failed'] | None	Field(default=None, exclude_if=lambda value: value is None)
execution_failure_category	Literal['provider_failure', 'environment_failure', 'execution_failure'] | None	Field(default=None, exclude_if=lambda value: value is None)
failure_evidence	FailureEvidence | None	Field(default=None, exclude_if=lambda value: value is None)
failure_capture	WorkflowFailureCapture | None	Field(default=None, exclude_if=lambda value: value is None)
capture_bounds	SessionTrajectoryBounds | None	Field(default=None, exclude_if=lambda value: value is None)
capture_diagnostic	WorkflowCaptureDiagnostic | None	Field(default=None, exclude_if=lambda value: value is None)
workflow_attempt	WorkflowAttemptAnchor | None	Field(default=None, exclude_if=lambda value: value is None)
retained_workflow_output	RetainedWorkflowEvalOutput | None	Field(default=None, exclude_if=lambda value: value is None, repr=False)
workflow_output_retention	Literal['retained', 'disabled', 'limit_exceeded'] | None	Field(default=None, exclude_if=lambda value: value is None)
evidence_complete	StrictBool	False
events_count	StrictInt	Field(default=0, ge=0)
usage_summary	dict[str, Any] | None	None
usage_evidence_state	Literal['complete', 'partial', 'unavailable'] | None	Field(default=None, exclude_if=lambda value: value is None)
memory_attribution	EvalMemoryAttributionEvidenceV1	Field(default_factory=lambda: EvalMemoryAttributionEvidenceV1.unavailable(EvalMemoryEvidenceLimitation.MISSING))
started_at	datetime	required
completed_at	datetime	required
duration_ms	StrictInt	Field(default=0, ge=0)
trajectory	Trajectory | None	Field(default=None, exclude=True, repr=False)

validate_session_id (method)

EvalTrialResult.validate_session_id(value: str | None, info) -> str | None

validate_final_output (method)

EvalTrialResult.validate_final_output(value: str, info) -> str

copy_structured_output (method)

EvalTrialResult.copy_structured_output(value)

revalidate_trajectory (method)

EvalTrialResult.revalidate_trajectory(value)

revalidate_eval_memory_attribution (method)

EvalTrialResult.revalidate_eval_memory_attribution(value)

normalize_diagnostic (method)

EvalTrialResult.normalize_diagnostic(value: object, info) -> str | None

copy_usage_summary (method)

EvalTrialResult.copy_usage_summary(value)

validate_result_contract (method)

EvalTrialResult.validate_result_contract() -> EvalTrialResult

Release source · Runtime guarantees · Capability guides

ProbeRequirements

from cayu.evals import ProbeRequirements

What an assertion needs captured from the live environment to evaluate offline.

The runner unions these across a case’s assertions and snapshots the result into the Trajectory while the environment is still live, so assertions never touch the app.

Decorators: @dataclass(frozen=True).

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
workspace_paths	frozenset[str]	frozenset()
workspace_structure_paths	frozenset[str]	frozenset()
artifact_scopes	frozenset[ArtifactScope]	frozenset()
artifact_requirements	tuple[ArtifactProbeRequirement, ...]	()

merged_with (method)

ProbeRequirements.merged_with(other: ProbeRequirements) -> ProbeRequirements

Release source · Runtime guarantees · Capability guides

Trajectory

from cayu.evals import Trajectory

The serializable record of one completed run — and the eval assertion substrate.

Composes already-serializable cayu types (events / transcript / usage / session) with a captured probe snapshot and the recursive sub-agent trajectories, so a whole run — its sub-agent tree included — can be persisted, reloaded, and replayed against assertions without a live runtime. It is the single object that flows through the lifecycle: a run produces it (EvalTrialResult.trajectory), write_trajectory_json / load_trajectory move it to and from disk, and evaluate_assertions re-checks it (the replay path). Distinct from EvalContext, which is the assertion’s view of a Trajectory.

Bases: BaseModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
session	Session | None	None
events	tuple[Event, ...]	Field(default_factory=tuple)
transcript	tuple[Message, ...]	Field(default_factory=tuple)
usage_summary	SessionUsageSummary | None	None
memory_attribution	MemoryAttribution | None	None
final_output	str	''
workflow_output	WorkflowEvalOutputEvidenceV1 | None	Field(default=None, exclude_if=lambda value: value is None)
probes	TrajectoryProbes	Field(default_factory=TrajectoryProbes)
children	tuple[Trajectory, ...]	Field(default_factory=tuple)
children_incomplete	StrictBool	False
metadata	dict[str, Any]	Field(default_factory=dict)

initial_input_message_start_index (property)

Trajectory.initial_input_message_start_index() -> int | None

Runtime-attested transcript index where fresh-run caller input begins.

initial_input_message_count (property)

Trajectory.initial_input_message_count() -> int | None

Runtime-attested fresh-run input count, absent from serialized trajectories.

initial_input_messages_sha256 (property)

Trajectory.initial_input_messages_sha256() -> str | None

Runtime-attested fresh-run input digest, absent from serialized trajectories.

structured_output_requested (property)

Trajectory.structured_output_requested() -> bool | None

Runtime-attested structured-output mode, absent from serialized trajectories.

input_redactions_applied (property)

Trajectory.input_redactions_applied() -> bool | None

Runtime-attested input-redaction mode, absent from serialized trajectories.

revalidate_session (method)

Trajectory.revalidate_session(value)

revalidate_events (method)

Trajectory.revalidate_events(value)

revalidate_transcript (method)

Trajectory.revalidate_transcript(value)

revalidate_usage_summary (method)

Trajectory.revalidate_usage_summary(value)

revalidate_memory_attribution (method)

Trajectory.revalidate_memory_attribution(value)

revalidate_probes (method)

Trajectory.revalidate_probes(value)

revalidate_children (method)

Trajectory.revalidate_children(value)

validate_final_output (method)

Trajectory.validate_final_output(value: str, info) -> str

revalidate_workflow_output (method)

Trajectory.revalidate_workflow_output(value)

copy_metadata (method)

Trajectory.copy_metadata(value)

validate_session_attribution (method)

Trajectory.validate_session_attribution() -> Trajectory

Release source · Runtime guarantees · Capability guides

TRAJECTORY_SCHEMA_VERSION

from cayu.evals import TRAJECTORY_SCHEMA_VERSION

TRAJECTORY_SCHEMA_VERSION = 5

Release source · Runtime guarantees · Capability guides

TrajectoryProbes

from cayu.evals import TrajectoryProbes

Serializable snapshot of the live-environment data assertions need.

Captured by the runner before the environment is torn down. The *_available flags distinguish “no workspace / artifact store configured” from “configured but the file / artifact is absent”, which the workspace and artifact assertions report differently.

Bases: BaseModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
workspace_available	StrictBool	False
workspace_files	dict[str, bytes | None]	Field(default_factory=dict)
workspace_file_stats	dict[str, WorkspaceFileProbe]	Field(default_factory=dict)
workspace_unavailable_paths	tuple[str, ...]	Field(default_factory=tuple)
workspace_structures	dict[str, WorkspaceStructuralProbe]	Field(default_factory=dict, max_length=64)
artifacts_available	StrictBool	False
artifact_scopes_captured	tuple[ArtifactScope, ...]	Field(default_factory=tuple)
artifact_scopes_truncated	tuple[ArtifactScope, ...]	Field(default_factory=tuple)
artifact_scopes_unavailable	tuple[ArtifactScope, ...]	Field(default_factory=tuple)
artifacts	tuple[ArtifactMetadata, ...]	Field(default_factory=tuple)
artifact_content_probes	tuple[ArtifactContentProbe, ...]	Field(default_factory=tuple)

validate_workspace_unavailable_paths (method)

TrajectoryProbes.validate_workspace_unavailable_paths(value) -> tuple[str, ...]

validate_workspace_file_paths (method)

TrajectoryProbes.validate_workspace_file_paths(value)

validate_artifact_scope_sets (method)

TrajectoryProbes.validate_artifact_scope_sets(value, info) -> tuple[ArtifactScope, ...]

revalidate_workspace_file_stats (method)

TrajectoryProbes.revalidate_workspace_file_stats(value)

revalidate_workspace_structures (method)

TrajectoryProbes.revalidate_workspace_structures(value)

revalidate_artifacts (method)

TrajectoryProbes.revalidate_artifacts(value)

revalidate_artifact_content_probes (method)

TrajectoryProbes.revalidate_artifact_content_probes(value)

validate_capture_provenance (method)

TrajectoryProbes.validate_capture_provenance() -> TrajectoryProbes

Release source · Runtime guarantees · Capability guides

WORKSPACE_PROBE_MAX_BYTES

from cayu.evals import WORKSPACE_PROBE_MAX_BYTES

WORKSPACE_PROBE_MAX_BYTES = 1 << 20

Release source · Runtime guarantees · Capability guides

WorkspaceFileProbe

from cayu.evals import WorkspaceFileProbe

Stat + integrity record for a probed workspace file.

Companion to TrajectoryProbes.workspace_files (the captured, possibly-truncated bytes): records the file’s true size on disk, whether the captured window was truncated at WORKSPACE_PROBE_MAX_BYTES, and a sha256 of the captured bytes. Direct Python assertions may use that prefix identity; portable structural assertions use WorkspaceStructuralProbe and never expose a truncated prefix digest as whole-file identity.

Bases: BaseModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
total_bytes	StrictInt	Field(ge=0)
truncated	StrictBool	False
sha256	str	required

validate_sha256 (method)

WorkspaceFileProbe.validate_sha256(value: str, info) -> str

Release source · Runtime guarantees · Capability guides