cayu.evals.trial_policy

Public Cayu API. Definitions are grouped by their public source module, and definitions from private implementation modules appear on their package’s page; adding a symbol does not move existing links. The import shown at each entry is a supported public namespace. Read the Python API overview and install the required extra.

`EVAL_SUITE_RUN_EXPOSURE_SCHEMA_VERSION`

from cayu.evals import EVAL_SUITE_RUN_EXPOSURE_SCHEMA_VERSION

EVAL_SUITE_RUN_EXPOSURE_SCHEMA_VERSION = 1

Release source · Runtime guarantees · Capability guides

`EVAL_SUITE_TRIAL_POLICY_SCHEMA_VERSION`

from cayu.evals import EVAL_SUITE_TRIAL_POLICY_SCHEMA_VERSION

EVAL_SUITE_TRIAL_POLICY_SCHEMA_VERSION = 1

Release source · Runtime guarantees · Capability guides

`EvalCandidateCostBudgetV1`

from cayu.evals import EvalCandidateCostBudgetV1

One configured per-trial observed-cost interruption threshold.

Bases: EvalMaximumCostTotalV1.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
`currency`	`StrictStr`	`Field(min_length=1, max_length=16)`
`amount`	`StrictStr`	`Field(min_length=1, max_length=128)`

validate_currency (method)

EvalCandidateCostBudgetV1.validate_currency(value: str) -> str

validate_amount (method)

EvalCandidateCostBudgetV1.validate_amount(value: str) -> str

from_decimal (method)

EvalCandidateCostBudgetV1.from_decimal(*, currency: str, amount: Decimal) -> Self

validate_positive_budget (method)

EvalCandidateCostBudgetV1.validate_positive_budget() -> EvalCandidateCostBudgetV1

Release source · Runtime guarantees · Capability guides

`EvalCaseReliabilityV1`

from cayu.evals import EvalCaseReliabilityV1

Lossless bounded distribution and policy decision for one published case.

Bases: _PortablePolicyModel.

Fields and attributes

Name	Type	Default
`schema_version`	`Literal[1]`	`1`
`trial_policy_revision`	`StrictStr`	`required`
`outcome`	`EvalReliabilityOutcome`	`required`
`total_trials`	`StrictInt`	`Field(ge=1, le=EVAL_SUITE_MAX_TRIALS)`
`passed_trials`	`StrictInt`	`Field(ge=0, le=EVAL_SUITE_MAX_TRIALS)`
`candidate_failed_trials`	`StrictInt`	`Field(ge=0, le=EVAL_SUITE_MAX_TRIALS)`
`runtime_error_trials`	`StrictInt`	`Field(ge=0, le=EVAL_SUITE_MAX_TRIALS)`
`evaluator_error_trials`	`StrictInt`	`Field(ge=0, le=EVAL_SUITE_MAX_TRIALS)`
`unavailable_trials`	`StrictInt`	`Field(ge=0, le=EVAL_SUITE_MAX_TRIALS)`
`cancelled_trials`	`StrictInt`	`Field(ge=0, le=EVAL_SUITE_MAX_TRIALS)`
`scored_trials`	`StrictInt`	`Field(ge=0, le=EVAL_SUITE_MAX_TRIALS)`
`minimum_score`	`StrictFloat \| None`	`Field(default=None, ge=0.0, le=1.0)`
`mean_score`	`StrictFloat \| None`	`Field(default=None, ge=0.0, le=1.0)`
`maximum_score`	`StrictFloat \| None`	`Field(default=None, ge=0.0, le=1.0)`
`variability`	`EvalVariabilityKind`	`required`

validate_schema_version_type (method)

EvalCaseReliabilityV1.validate_schema_version_type(value: object) -> object

validate_policy_revision (method)

EvalCaseReliabilityV1.validate_policy_revision(value: str, info) -> str

validate_distribution (method)

EvalCaseReliabilityV1.validate_distribution() -> EvalCaseReliabilityV1

create (method)

EvalCaseReliabilityV1.create(*, policy: EvalSuiteTrialPolicyV1, trials: Iterable[tuple[str, float | None, EvalTrialDiagnosticCode]], uses_model_judge: bool) -> EvalCaseReliabilityV1

Release source · Runtime guarantees · Capability guides

`EvalExecutionProfileExposureV1`

from cayu.evals import EvalExecutionProfileExposureV1

One exact candidate profile bound to the cases it will execute.

Bases: _PortablePolicyModel.

Fields and attributes

Name	Type	Default
`case_ids`	`tuple[StrictStr, ...]`	`Field(min_length=1, max_length=1000)`
`execution_profile_revision`	`StrictStr`	`required`
`execution_profile_comparison_revision`	`StrictStr`	`required`
`candidate_cost_budget`	`EvalCandidateCostBudgetV1 \| None`	`None`

validate_case_ids (method)

EvalExecutionProfileExposureV1.validate_case_ids(value: tuple[str, ...]) -> tuple[str, ...]

validate_profile_revision (method)

EvalExecutionProfileExposureV1.validate_profile_revision(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

`EvalJudgeProfileExposureV1`

from cayu.evals import EvalJudgeProfileExposureV1

One current judge route accepted for every assertion that names its key.

Bases: _PortablePolicyModel.

Fields and attributes

Name	Type	Default
`profile_key`	`StrictStr`	`required`
`judge_profile_revision`	`StrictStr`	`required`
`judge_profile_comparison_revision`	`StrictStr`	`required`

validate_profile_key (method)

EvalJudgeProfileExposureV1.validate_profile_key(value: str) -> str

validate_profile_revision (method)

EvalJudgeProfileExposureV1.validate_profile_revision(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

`EvalMaximumCostExposureV1`

from cayu.evals import EvalMaximumCostExposureV1

Bases: _PortablePolicyModel.

Fields and attributes

Name	Type	Default
`state`	`Literal['priced', 'unavailable', 'not_applicable']`	`required`
`totals`	`tuple[EvalMaximumCostTotalV1, ...]`	`Field(default=(), max_length=16)`
`unavailable_reason`	`EvalMaximumCostUnavailableReason \| None`	`None`
`pricing_profile_fingerprints`	`tuple[StrictStr, ...]`	`Field(default=(), max_length=64)`

validate_pricing_revisions (method)

EvalMaximumCostExposureV1.validate_pricing_revisions(value: tuple[str, ...]) -> tuple[str, ...]

validate_state (method)

EvalMaximumCostExposureV1.validate_state() -> EvalMaximumCostExposureV1

Release source · Runtime guarantees · Capability guides

`EvalMaximumCostTotalV1`

from cayu.evals import EvalMaximumCostTotalV1

Bases: _PortablePolicyModel.

Fields and attributes

Name	Type	Default
`currency`	`StrictStr`	`Field(min_length=1, max_length=16)`
`amount`	`StrictStr`	`Field(min_length=1, max_length=128)`

validate_currency (method)

EvalMaximumCostTotalV1.validate_currency(value: str) -> str

validate_amount (method)

EvalMaximumCostTotalV1.validate_amount(value: str) -> str

from_decimal (method)

EvalMaximumCostTotalV1.from_decimal(*, currency: str, amount: Decimal) -> Self

Release source · Runtime guarantees · Capability guides

`EvalMaximumCostUnavailableReason`

from cayu.evals import EvalMaximumCostUnavailableReason

EvalMaximumCostUnavailableReason = Literal['no_candidate_cost_ceiling', 'candidate_cost_not_hard_bounded', 'candidate_pricing_incomplete', 'judge_cost_not_hard_bounded', 'judge_pricing_incomplete']

Release source · Runtime guarantees · Capability guides

`EvalSuiteRunExposureV1`

from cayu.evals import EvalSuiteRunExposureV1

Exact maximum configured work accepted for one authored-suite selection.

Bases: _PortablePolicyModel.

Fields and attributes

Name	Type	Default
`schema_version`	`Literal[1]`	`EVAL_SUITE_RUN_EXPOSURE_SCHEMA_VERSION`
`revision`	`StrictStr`	`required`
`selection_revision`	`StrictStr`	`required`
`trial_policy_revision`	`StrictStr`	`required`
`execution_profiles`	`tuple[EvalExecutionProfileExposureV1, ...]`	`Field(min_length=1, max_length=1000)`
`judge_profiles`	`tuple[EvalJudgeProfileExposureV1, ...]`	`Field(default=(), max_length=64)`
`candidate_trials`	`StrictInt`	`Field(ge=1, le=100000)`
`judge_evaluations`	`StrictInt`	`Field(ge=0, le=6400000)`
`max_concurrency`	`StrictInt`	`Field(ge=1, le=EVAL_SUITE_MAX_CONCURRENCY)`
`maximum_candidate_model_steps`	`StrictInt`	`Field(ge=1)`
`maximum_candidate_total_tokens`	`StrictInt \| None`	`Field(default=None, ge=1)`
`maximum_judge_input_tokens`	`StrictInt \| None`	`Field(default=None, ge=0)`
`maximum_judge_output_tokens`	`StrictInt \| None`	`Field(default=None, ge=0)`
`maximum_judge_total_tokens`	`StrictInt \| None`	`Field(default=None, ge=0)`
`candidate_cost`	`EvalMaximumCostExposureV1`	`required`
`judge_cost`	`EvalMaximumCostExposureV1`	`required`

validate_schema_version_type (method)

EvalSuiteRunExposureV1.validate_schema_version_type(value: object) -> object

validate_revisions (method)

EvalSuiteRunExposureV1.validate_revisions(value: str, info) -> str

validate_execution_profiles (method)

EvalSuiteRunExposureV1.validate_execution_profiles(value: tuple[EvalExecutionProfileExposureV1, ...]) -> tuple[EvalExecutionProfileExposureV1, ...]

validate_judge_profiles (method)

EvalSuiteRunExposureV1.validate_judge_profiles(value: tuple[EvalJudgeProfileExposureV1, ...]) -> tuple[EvalJudgeProfileExposureV1, ...]

validate_contract (method)

EvalSuiteRunExposureV1.validate_contract() -> EvalSuiteRunExposureV1

comparison_revision (property)

EvalSuiteRunExposureV1.comparison_revision() -> str

Return the cross-release identity of the accepted execution contract.

create (method)

EvalSuiteRunExposureV1.create(**values: Any) -> EvalSuiteRunExposureV1

Release source · Runtime guarantees · Capability guides

`EvalSuiteTrialPolicyV1`

from cayu.evals import EvalSuiteTrialPolicyV1

Immutable decision rule for the complete retained trial set of each case.

Bases: _PortablePolicyModel.

Fields and attributes

Name	Type	Default
`schema_version`	`Literal[1]`	`EVAL_SUITE_TRIAL_POLICY_SCHEMA_VERSION`
`revision`	`StrictStr`	`required`
`trial_count`	`StrictInt`	`Field(ge=1, le=EVAL_SUITE_MAX_TRIALS)`
`minimum_passed_trials`	`StrictInt`	`Field(ge=1, le=EVAL_SUITE_MAX_TRIALS)`
`max_concurrency`	`StrictInt`	`Field(ge=1, le=EVAL_SUITE_MAX_CONCURRENCY)`
`require_zero_runtime_errors`	`Literal[True]`	`True`
`require_zero_evaluator_errors`	`Literal[True]`	`True`
`require_complete_required_evidence`	`Literal[True]`	`True`

validate_schema_version_type (method)

EvalSuiteTrialPolicyV1.validate_schema_version_type(value: object) -> object

validate_required_flag_types (method)

EvalSuiteTrialPolicyV1.validate_required_flag_types(value: object, info) -> object

validate_revision_shape (method)

EvalSuiteTrialPolicyV1.validate_revision_shape(value: str, info) -> str

validate_contract (method)

EvalSuiteTrialPolicyV1.validate_contract() -> EvalSuiteTrialPolicyV1

create (method)

EvalSuiteTrialPolicyV1.create(*, trial_count: int=1, minimum_passed_trials: int | None=None, max_concurrency: int=1) -> EvalSuiteTrialPolicyV1

Release source · Runtime guarantees · Capability guides