cayu.evals.suite_authoring

Public Cayu API. Definitions are grouped by their public source module, and definitions from private implementation modules appear on their package’s page; adding a symbol does not move existing links. The import shown at each entry is a supported public namespace. Read the Python API overview and install the required extra.

add_eval_case

from cayu.evals import add_eval_case

Return a new suite revision containing one new stable case ID.

add_eval_case(document: EvalSuiteDocumentV1, case: EvalCaseDraftV1) -> EvalSuiteDocumentV1

Release source · Runtime guarantees · Capability guides

compile_eval_suite_authoring_draft

from cayu.evals import compile_eval_suite_authoring_draft

Compile the exact draft wire version selected by the caller.

compile_eval_suite_authoring_draft(draft: EvalSuiteDraft) -> EvalSuiteDocument

Release source · Runtime guarantees · Capability guides

compile_eval_suite_draft

from cayu.evals import compile_eval_suite_draft

Canonicalize bounded editor material into one immutable suite revision.

compile_eval_suite_draft(draft: EvalSuiteDraftV1) -> EvalSuiteDocumentV1

Release source · Runtime guarantees · Capability guides

compile_eval_suite_draft_v2

from cayu.evals import compile_eval_suite_draft_v2

Canonicalize V2 editor material without losing structured judge fields.

compile_eval_suite_draft_v2(draft: EvalSuiteDraftV2) -> EvalSuiteDocumentV2

Release source · Runtime guarantees · Capability guides

compile_eval_suite_draft_v3

from cayu.evals import compile_eval_suite_draft_v3

Canonicalize V3 material with its explicit suite trial policy.

compile_eval_suite_draft_v3(draft: EvalSuiteDraftV3) -> EvalSuiteDocumentV3

Release source · Runtime guarantees · Capability guides

duplicate_eval_case

from cayu.evals import duplicate_eval_case

Copy one case under an explicit new identity without aliasing history.

duplicate_eval_case(document: EvalSuiteDocumentV1, case_id: str, *, new_case_id: str, new_name: str | None=None) -> EvalSuiteDocumentV1

Release source · Runtime guarantees · Capability guides

EVAL_SUITE_AUTHORING_MAX_BYTES

from cayu.evals import EVAL_SUITE_AUTHORING_MAX_BYTES

EVAL_SUITE_AUTHORING_MAX_BYTES = EVAL_CORPUS_MAX_BYTES

Release source · Runtime guarantees · Capability guides

EVAL_SUITE_AUTHORING_SCHEMA_VERSION

from cayu.evals import EVAL_SUITE_AUTHORING_SCHEMA_VERSION

EVAL_SUITE_AUTHORING_SCHEMA_VERSION = 1

Release source · Runtime guarantees · Capability guides

EVAL_SUITE_AUTHORING_V2_SCHEMA_VERSION

from cayu.evals import EVAL_SUITE_AUTHORING_V2_SCHEMA_VERSION

EVAL_SUITE_AUTHORING_V2_SCHEMA_VERSION = 2

Release source · Runtime guarantees · Capability guides

EVAL_SUITE_AUTHORING_V3_SCHEMA_VERSION

from cayu.evals import EVAL_SUITE_AUTHORING_V3_SCHEMA_VERSION

EVAL_SUITE_AUTHORING_V3_SCHEMA_VERSION = 3

Release source · Runtime guarantees · Capability guides

eval_suite_document_from_json

from cayu.evals import eval_suite_document_from_json

Load one bounded authored-suite document from strict JSON text.

eval_suite_document_from_json(source: str) -> EvalSuiteDocument

Release source · Runtime guarantees · Capability guides

eval_suite_document_to_json

from cayu.evals import eval_suite_document_to_json

Return deterministic, human-readable authored-suite JSON.

eval_suite_document_to_json(document: EvalSuiteDocument) -> str

Release source · Runtime guarantees · Capability guides

eval_suite_selection

from cayu.evals import eval_suite_selection

Freeze either every case or an explicit non-empty subset by revision.

eval_suite_selection(document: EvalSuiteDocument, case_ids: Sequence[str] | None=None) -> EvalSuiteSelectionV1

Release source · Runtime guarantees · Capability guides

EVAL_SUITE_SELECTION_SCHEMA_VERSION

from cayu.evals import EVAL_SUITE_SELECTION_SCHEMA_VERSION

EVAL_SUITE_SELECTION_SCHEMA_VERSION = 1

Release source · Runtime guarantees · Capability guides

EvalCaseDefinitionV1

from cayu.evals import EvalCaseDefinitionV1

One immutable authored case revision with truthful optional provenance.

Bases: _EvalCaseAuthoringMaterial.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
name	StrictStr	required
description	StrictStr | None	None
source	EvaluationSourceIdentityV1 | None	None
stimulus	EvalCaseStimulusV1	required
assertions	tuple[EvalSuiteAuthoringAssertionSpecV1, ...]	Field(min_length=1, max_length=EVAL_CORPUS_MAX_ASSERTIONS_PER_CASE)
revision	StrictStr	required

validate_id (method)

EvalCaseDefinitionV1.validate_id(value: str, info) -> str

validate_name (method)

EvalCaseDefinitionV1.validate_name(value: str, info) -> str

validate_description (method)

EvalCaseDefinitionV1.validate_description(value: str | None, info) -> str | None

validate_stimulus (method)

EvalCaseDefinitionV1.validate_stimulus(value: EvalCaseStimulusV1) -> EvalCaseStimulusV1

validate_source (method)

EvalCaseDefinitionV1.validate_source(value: EvaluationSourceIdentityV1 | None) -> EvaluationSourceIdentityV1 | None

validate_assertions_are_ordered (method)

EvalCaseDefinitionV1.validate_assertions_are_ordered(value: object, info) -> object

validate_assertions (method)

EvalCaseDefinitionV1.validate_assertions(value: tuple[EvalSuiteAuthoringAssertionSpecV1, ...]) -> tuple[EvalSuiteAuthoringAssertionSpecV1, ...]

validate_assertion_ids (method)

EvalCaseDefinitionV1.validate_assertion_ids() -> _EvalCaseAuthoringMaterial

validate_revision_shape (method)

EvalCaseDefinitionV1.validate_revision_shape(value: str, info) -> str

validate_revision (method)

EvalCaseDefinitionV1.validate_revision() -> EvalCaseDefinitionV1

create (method)

EvalCaseDefinitionV1.create(draft: EvalCaseDraftV1) -> EvalCaseDefinitionV1

Release source · Runtime guarantees · Capability guides

EvalCaseDefinitionV2

from cayu.evals import EvalCaseDefinitionV2

One immutable V2 case revision with structured-judge fidelity.

Bases: _EvalCaseAuthoringMaterialV2.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
name	StrictStr	required
description	StrictStr | None	None
source	EvaluationSourceIdentityV1 | None	None
stimulus	EvalCaseStimulusV1	required
assertions	tuple[EvalSuiteAuthoringAssertionSpecV2, ...]	Field(min_length=1, max_length=EVAL_CORPUS_MAX_ASSERTIONS_PER_CASE)
revision	StrictStr	required

validate_id (method)

EvalCaseDefinitionV2.validate_id(value: str, info) -> str

validate_name (method)

EvalCaseDefinitionV2.validate_name(value: str, info) -> str

validate_description (method)

EvalCaseDefinitionV2.validate_description(value: str | None, info) -> str | None

validate_stimulus (method)

EvalCaseDefinitionV2.validate_stimulus(value: EvalCaseStimulusV1) -> EvalCaseStimulusV1

validate_source (method)

EvalCaseDefinitionV2.validate_source(value: EvaluationSourceIdentityV1 | None) -> EvaluationSourceIdentityV1 | None

validate_assertions_are_ordered (method)

EvalCaseDefinitionV2.validate_assertions_are_ordered(value: object, info) -> object

validate_assertions (method)

EvalCaseDefinitionV2.validate_assertions(value: tuple[EvalSuiteAuthoringAssertionSpecV2, ...]) -> tuple[EvalSuiteAuthoringAssertionSpecV2, ...]

validate_assertion_ids (method)

EvalCaseDefinitionV2.validate_assertion_ids() -> _EvalCaseAuthoringMaterialV2

validate_revision_shape (method)

EvalCaseDefinitionV2.validate_revision_shape(value: str, info) -> str

validate_revision (method)

EvalCaseDefinitionV2.validate_revision() -> EvalCaseDefinitionV2

create (method)

EvalCaseDefinitionV2.create(draft: EvalCaseDraftV2) -> EvalCaseDefinitionV2

Release source · Runtime guarantees · Capability guides

EvalCaseDraftV1

from cayu.evals import EvalCaseDraftV1

Bounded, revision-free case material accepted by authoring previews.

Bases: _EvalCaseAuthoringMaterial.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
name	StrictStr	required
description	StrictStr | None	None
source	EvaluationSourceIdentityV1 | None	None
stimulus	EvalCaseStimulusV1	required
assertions	tuple[EvalSuiteAuthoringAssertionSpecV1, ...]	Field(min_length=1, max_length=EVAL_CORPUS_MAX_ASSERTIONS_PER_CASE)

validate_id (method)

EvalCaseDraftV1.validate_id(value: str, info) -> str

validate_name (method)

EvalCaseDraftV1.validate_name(value: str, info) -> str

validate_description (method)

EvalCaseDraftV1.validate_description(value: str | None, info) -> str | None

validate_stimulus (method)

EvalCaseDraftV1.validate_stimulus(value: EvalCaseStimulusV1) -> EvalCaseStimulusV1

validate_source (method)

EvalCaseDraftV1.validate_source(value: EvaluationSourceIdentityV1 | None) -> EvaluationSourceIdentityV1 | None

validate_assertions_are_ordered (method)

EvalCaseDraftV1.validate_assertions_are_ordered(value: object, info) -> object

validate_assertions (method)

EvalCaseDraftV1.validate_assertions(value: tuple[EvalSuiteAuthoringAssertionSpecV1, ...]) -> tuple[EvalSuiteAuthoringAssertionSpecV1, ...]

validate_assertion_ids (method)

EvalCaseDraftV1.validate_assertion_ids() -> _EvalCaseAuthoringMaterial

from_case (method)

EvalCaseDraftV1.from_case(case: EvalCaseDefinitionV1) -> EvalCaseDraftV1

Release source · Runtime guarantees · Capability guides

EvalCaseDraftV2

from cayu.evals import EvalCaseDraftV2

Revision-free V2 case material accepted by authoring previews.

Bases: _EvalCaseAuthoringDraftMaterialV2.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
name	StrictStr	required
description	StrictStr | None	None
source	EvaluationSourceIdentityV1 | None	None
stimulus	EvalCaseStimulusV1	required
assertions	tuple[EvalSuiteAuthoringAssertionDraftSpecV2, ...]	Field(min_length=1, max_length=EVAL_CORPUS_MAX_ASSERTIONS_PER_CASE)

validate_id (method)

EvalCaseDraftV2.validate_id(value: str, info) -> str

validate_name (method)

EvalCaseDraftV2.validate_name(value: str, info) -> str

validate_description (method)

EvalCaseDraftV2.validate_description(value: str | None, info) -> str | None

validate_stimulus (method)

EvalCaseDraftV2.validate_stimulus(value: EvalCaseStimulusV1) -> EvalCaseStimulusV1

validate_source (method)

EvalCaseDraftV2.validate_source(value: EvaluationSourceIdentityV1 | None) -> EvaluationSourceIdentityV1 | None

validate_assertions_are_ordered (method)

EvalCaseDraftV2.validate_assertions_are_ordered(value: object, info) -> object

validate_assertions (method)

EvalCaseDraftV2.validate_assertions(value: tuple[EvalSuiteAuthoringAssertionDraftSpecV2, ...]) -> tuple[EvalSuiteAuthoringAssertionDraftSpecV2, ...]

validate_assertion_ids (method)

EvalCaseDraftV2.validate_assertion_ids() -> _EvalCaseAuthoringDraftMaterialV2

from_case (method)

EvalCaseDraftV2.from_case(case: EvalCaseDefinitionV1 | EvalCaseDefinitionV2) -> EvalCaseDraftV2

Release source · Runtime guarantees · Capability guides

EvalCaseStimulusV1

from cayu.evals import EvalCaseStimulusV1

EvalCaseStimulusV1: TypeAlias = Annotated[EvalSimpleInputStimulusV1 | EvalScenarioStimulusV1, Field(discriminator='kind')]

Release source · Runtime guarantees · Capability guides

EvalScenarioStimulusV1

from cayu.evals import EvalScenarioStimulusV1

A content-addressed reference to one separately persisted scenario.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
kind	Literal['scenario']	'scenario'
scenario_id	StrictStr	required
scenario_revision	StrictStr	required

validate_scenario_id (method)

EvalScenarioStimulusV1.validate_scenario_id(value: str, info) -> str

validate_scenario_revision (method)

EvalScenarioStimulusV1.validate_scenario_revision(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

EvalSelectedCaseV1

from cayu.evals import EvalSelectedCaseV1

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
revision	StrictStr	required

validate_id (method)

EvalSelectedCaseV1.validate_id(value: str, info) -> str

validate_revision (method)

EvalSelectedCaseV1.validate_revision(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

EvalSimpleInputStimulusV1

from cayu.evals import EvalSimpleInputStimulusV1

One ordinary fresh invocation authored without captured provenance.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
kind	Literal['simple_input']	'simple_input'
input	RunInputSpec	required

validate_input (method)

EvalSimpleInputStimulusV1.validate_input(value: RunInputSpec) -> RunInputSpec

Release source · Runtime guarantees · Capability guides

EvalSuiteDocumentV1

from cayu.evals import EvalSuiteDocumentV1

One immutable, authority-free authored suite and its exact case revisions.

Bases: _SchemaV1PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[1]	EVAL_SUITE_AUTHORING_SCHEMA_VERSION
revision	StrictStr	required
target_key	StrictStr	required
suite	EvalSuiteSpec	required
cases	tuple[EvalCaseDefinitionV1, ...]	Field(min_length=1, max_length=EVAL_CORPUS_MAX_CASES)

validate_schema_version_type (method)

EvalSuiteDocumentV1.validate_schema_version_type(value: object) -> object

validate_revision_shape (method)

EvalSuiteDocumentV1.validate_revision_shape(value: str, info) -> str

validate_target_key (method)

EvalSuiteDocumentV1.validate_target_key(value: str, info) -> str

validate_cases_are_ordered (method)

EvalSuiteDocumentV1.validate_cases_are_ordered(value: object, info) -> object

validate_suite (method)

EvalSuiteDocumentV1.validate_suite(value: EvalSuiteSpec) -> EvalSuiteSpec

validate_cases (method)

EvalSuiteDocumentV1.validate_cases(value: tuple[EvalCaseDefinitionV1, ...]) -> tuple[EvalCaseDefinitionV1, ...]

validate_contract (method)

EvalSuiteDocumentV1.validate_contract() -> EvalSuiteDocumentV1

Release source · Runtime guarantees · Capability guides

EvalSuiteDocumentV2

from cayu.evals import EvalSuiteDocumentV2

Immutable V2 suite preserving structured rubric and judge contracts.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[2]	EVAL_SUITE_AUTHORING_V2_SCHEMA_VERSION
revision	StrictStr	required
target_key	StrictStr	required
suite	EvalSuiteSpec	required
cases	tuple[EvalCaseDefinitionV2, ...]	Field(min_length=1, max_length=EVAL_CORPUS_MAX_CASES)

validate_revision_shape (method)

EvalSuiteDocumentV2.validate_revision_shape(value: str, info) -> str

validate_target_key (method)

EvalSuiteDocumentV2.validate_target_key(value: str, info) -> str

validate_cases_are_ordered (method)

EvalSuiteDocumentV2.validate_cases_are_ordered(value: object, info) -> object

validate_suite (method)

EvalSuiteDocumentV2.validate_suite(value: EvalSuiteSpec) -> EvalSuiteSpec

validate_cases (method)

EvalSuiteDocumentV2.validate_cases(value: tuple[EvalCaseDefinitionV2, ...]) -> tuple[EvalCaseDefinitionV2, ...]

validate_contract (method)

EvalSuiteDocumentV2.validate_contract() -> EvalSuiteDocumentV2

Release source · Runtime guarantees · Capability guides

EvalSuiteDocumentV3

from cayu.evals import EvalSuiteDocumentV3

Immutable V3 suite with an explicit revisioned trial decision policy.

Bases: EvalSuiteDocumentV2.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[3]	EVAL_SUITE_AUTHORING_V3_SCHEMA_VERSION
revision	StrictStr	required
target_key	StrictStr	required
suite	EvalSuiteSpec	required
cases	tuple[EvalCaseDefinitionV2, ...]	Field(min_length=1, max_length=EVAL_CORPUS_MAX_CASES)

validate_revision_shape (method)

EvalSuiteDocumentV3.validate_revision_shape(value: str, info) -> str

validate_target_key (method)

EvalSuiteDocumentV3.validate_target_key(value: str, info) -> str

validate_cases_are_ordered (method)

EvalSuiteDocumentV3.validate_cases_are_ordered(value: object, info) -> object

validate_suite (method)

EvalSuiteDocumentV3.validate_suite(value: EvalSuiteSpec) -> EvalSuiteSpec

validate_cases (method)

EvalSuiteDocumentV3.validate_cases(value: tuple[EvalCaseDefinitionV2, ...]) -> tuple[EvalCaseDefinitionV2, ...]

validate_contract (method)

EvalSuiteDocumentV3.validate_contract() -> EvalSuiteDocumentV2

validate_schema_version_type (method)

EvalSuiteDocumentV3.validate_schema_version_type(value: object) -> object

validate_v3_contract (method)

EvalSuiteDocumentV3.validate_v3_contract() -> EvalSuiteDocumentV3

Release source · Runtime guarantees · Capability guides

EvalSuiteDraftV1

from cayu.evals import EvalSuiteDraftV1

Revision-free suite material edited by an SDK or Control Plane client.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
target_key	StrictStr	required
name	StrictStr	required
description	StrictStr | None	None
trial_request	TrialRequestSpec	Field(default_factory=TrialRequestSpec)
cases	tuple[EvalCaseDraftV1, ...]	Field(min_length=1, max_length=EVAL_CORPUS_MAX_CASES)

validate_ids (method)

EvalSuiteDraftV1.validate_ids(value: str, info) -> str

validate_name (method)

EvalSuiteDraftV1.validate_name(value: str, info) -> str

validate_description (method)

EvalSuiteDraftV1.validate_description(value: str | None, info) -> str | None

validate_cases_are_ordered (method)

EvalSuiteDraftV1.validate_cases_are_ordered(value: object, info) -> object

validate_trial_request (method)

EvalSuiteDraftV1.validate_trial_request(value: TrialRequestSpec) -> TrialRequestSpec

validate_cases (method)

EvalSuiteDraftV1.validate_cases(value: tuple[EvalCaseDraftV1, ...]) -> tuple[EvalCaseDraftV1, ...]

validate_case_ids (method)

EvalSuiteDraftV1.validate_case_ids() -> EvalSuiteDraftV1

from_document (method)

EvalSuiteDraftV1.from_document(document: EvalSuiteDocumentV1) -> EvalSuiteDraftV1

Release source · Runtime guarantees · Capability guides

EvalSuiteDraftV2

from cayu.evals import EvalSuiteDraftV2

Explicit V2 editor material with structured-judge authoring support.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[2]	EVAL_SUITE_AUTHORING_V2_SCHEMA_VERSION
id	StrictStr	required
target_key	StrictStr	required
name	StrictStr	required
description	StrictStr | None	None
trial_request	TrialRequestSpec	Field(default_factory=TrialRequestSpec)
cases	tuple[EvalCaseDraftV2, ...]	Field(min_length=1, max_length=EVAL_CORPUS_MAX_CASES)

validate_ids (method)

EvalSuiteDraftV2.validate_ids(value: str, info) -> str

validate_name (method)

EvalSuiteDraftV2.validate_name(value: str, info) -> str

validate_description (method)

EvalSuiteDraftV2.validate_description(value: str | None, info) -> str | None

validate_cases_are_ordered (method)

EvalSuiteDraftV2.validate_cases_are_ordered(value: object, info) -> object

validate_trial_request (method)

EvalSuiteDraftV2.validate_trial_request(value: TrialRequestSpec) -> TrialRequestSpec

validate_cases (method)

EvalSuiteDraftV2.validate_cases(value: tuple[EvalCaseDraftV2, ...]) -> tuple[EvalCaseDraftV2, ...]

validate_case_ids (method)

EvalSuiteDraftV2.validate_case_ids() -> EvalSuiteDraftV2

from_document (method)

EvalSuiteDraftV2.from_document(document: EvalSuiteDocumentV1 | EvalSuiteDocumentV2) -> EvalSuiteDraftV2

Release source · Runtime guarantees · Capability guides

EvalSuiteDraftV3

from cayu.evals import EvalSuiteDraftV3

V3 editor material with revision-free suite trial settings.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[3]	EVAL_SUITE_AUTHORING_V3_SCHEMA_VERSION
id	StrictStr	required
target_key	StrictStr	required
name	StrictStr	required
description	StrictStr | None	None
trial_request	EvalSuiteTrialRequestDraftV3	Field(default_factory=EvalSuiteTrialRequestDraftV3)
cases	tuple[EvalCaseDraftV2, ...]	Field(min_length=1, max_length=EVAL_CORPUS_MAX_CASES)

validate_schema_version_type (method)

EvalSuiteDraftV3.validate_schema_version_type(value: object) -> object

validate_ids (method)

EvalSuiteDraftV3.validate_ids(value: str, info) -> str

validate_name (method)

EvalSuiteDraftV3.validate_name(value: str, info) -> str

validate_description (method)

EvalSuiteDraftV3.validate_description(value: str | None, info) -> str | None

validate_cases_are_ordered (method)

EvalSuiteDraftV3.validate_cases_are_ordered(value: object, info) -> object

validate_trial_request (method)

EvalSuiteDraftV3.validate_trial_request(value: EvalSuiteTrialRequestDraftV3) -> EvalSuiteTrialRequestDraftV3

validate_cases (method)

EvalSuiteDraftV3.validate_cases(value: tuple[EvalCaseDraftV2, ...]) -> tuple[EvalCaseDraftV2, ...]

validate_case_ids (method)

EvalSuiteDraftV3.validate_case_ids() -> EvalSuiteDraftV3

from_document (method)

EvalSuiteDraftV3.from_document(document: EvalSuiteDocumentV1 | EvalSuiteDocumentV2 | EvalSuiteDocumentV3) -> EvalSuiteDraftV3

Release source · Runtime guarantees · Capability guides

EvalSuiteSelectionV1

from cayu.evals import EvalSuiteSelectionV1

Content identity for a full-suite or explicit-subset launch request.

Bases: _SchemaV1PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[1]	EVAL_SUITE_SELECTION_SCHEMA_VERSION
revision	StrictStr	required
suite_document_revision	StrictStr	required
suite_id	StrictStr	required
suite_revision	StrictStr	required
mode	Literal['full_suite', 'subset']	required
cases	tuple[EvalSelectedCaseV1, ...]	Field(min_length=1, max_length=EVAL_CORPUS_MAX_CASES)

validate_schema_version_type (method)

EvalSuiteSelectionV1.validate_schema_version_type(value: object) -> object

validate_revisions (method)

EvalSuiteSelectionV1.validate_revisions(value: str, info) -> str

validate_suite_id (method)

EvalSuiteSelectionV1.validate_suite_id(value: str, info) -> str

validate_cases_are_ordered (method)

EvalSuiteSelectionV1.validate_cases_are_ordered(value: object, info) -> object

validate_cases (method)

EvalSuiteSelectionV1.validate_cases(value: tuple[EvalSelectedCaseV1, ...]) -> tuple[EvalSelectedCaseV1, ...]

validate_contract (method)

EvalSuiteSelectionV1.validate_contract() -> EvalSuiteSelectionV1

Release source · Runtime guarantees · Capability guides

EvalSuiteTrialRequestDraftV3

from cayu.evals import EvalSuiteTrialRequestDraftV3

Revision-free trial settings edited by SDK and Control Plane clients.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
trials	StrictInt	Field(default=1, ge=1, le=EVAL_CORPUS_MAX_TRIALS)
timeout_seconds	StrictInt	Field(default=300, ge=1, le=EVAL_CORPUS_MAX_TIMEOUT_SECONDS)
minimum_passed_trials	StrictInt	Field(default=1, ge=1, le=EVAL_CORPUS_MAX_TRIALS)
max_concurrency	StrictInt	Field(default=1, ge=1, le=EVAL_SUITE_MAX_CONCURRENCY)

validate_policy_bounds (method)

EvalSuiteTrialRequestDraftV3.validate_policy_bounds() -> EvalSuiteTrialRequestDraftV3

from_request (method)

EvalSuiteTrialRequestDraftV3.from_request(request: TrialRequestSpec) -> EvalSuiteTrialRequestDraftV3

Release source · Runtime guarantees · Capability guides

PublicJudgeReferenceDraftV1

from cayu.evals import PublicJudgeReferenceDraftV1

Revision-free public reference truth accepted from an editor.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[1]	1
kind	Literal['public_reference']	'public_reference'
id	StrictStr	required
expected_answer	StrictStr | None	None
expected_facts	tuple[StrictStr, ...]	Field(default=(), max_length=EVAL_CORPUS_MAX_JUDGE_REFERENCE_FACTS)

validate_id (method)

PublicJudgeReferenceDraftV1.validate_id(value: str, info) -> str

validate_expected_facts_are_ordered (method)

PublicJudgeReferenceDraftV1.validate_expected_facts_are_ordered(value: object, info) -> object

validate_contract (method)

PublicJudgeReferenceDraftV1.validate_contract() -> PublicJudgeReferenceDraftV1

from_reference (method)

PublicJudgeReferenceDraftV1.from_reference(reference: PublicJudgeReferenceV1) -> PublicJudgeReferenceDraftV1

compile (method)

PublicJudgeReferenceDraftV1.compile() -> PublicJudgeReferenceV1

Release source · Runtime guarantees · Capability guides

revise_eval_case

from cayu.evals import revise_eval_case

Replace one case only when the caller reviewed its current revision.

revise_eval_case(document: EvalSuiteDocumentV1, replacement: EvalCaseDraftV1, *, expected_case_revision: str) -> EvalSuiteDocumentV1

Release source · Runtime guarantees · Capability guides

StructuredModelJudgeAssertionDraftV1

from cayu.evals import StructuredModelJudgeAssertionDraftV1

Revision-free rubric assertion material compiled only by the server.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
kind	Literal['structured_model_judge']	'structured_model_judge'
description	StrictStr | None	None
judge_profile_key	StrictStr	required
judge_profile_revision	StrictStr	required
rubric	StructuredRubricDraftV1	required
reference	JudgeReferenceDraftV1 | None	None
threshold	StrictStr	'0.5'
evidence	EvalJudgeEvidenceSelectionV1	Field(default_factory=EvalJudgeEvidenceSelectionV1)

validate_ids (method)

StructuredModelJudgeAssertionDraftV1.validate_ids(value: str, info) -> str

validate_description (method)

StructuredModelJudgeAssertionDraftV1.validate_description(value: str | None, info) -> str | None

validate_judge_profile_revision (method)

StructuredModelJudgeAssertionDraftV1.validate_judge_profile_revision(value: str, info) -> str

validate_threshold (method)

StructuredModelJudgeAssertionDraftV1.validate_threshold(value: str, info) -> str

from_assertion (method)

StructuredModelJudgeAssertionDraftV1.from_assertion(assertion: StructuredModelJudgeAssertionSpec) -> StructuredModelJudgeAssertionDraftV1

compile (method)

StructuredModelJudgeAssertionDraftV1.compile() -> StructuredModelJudgeAssertionSpec

Release source · Runtime guarantees · Capability guides

StructuredRubricDraftV1

from cayu.evals import StructuredRubricDraftV1

Revision-free structured rubric material accepted from an editor.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[1]	1
id	StrictStr	required
criteria	tuple[StructuredRubricCriterionV1, ...]	Field(min_length=1, max_length=EVAL_CORPUS_MAX_JUDGE_CRITERIA)

validate_id (method)

StructuredRubricDraftV1.validate_id(value: str, info) -> str

validate_criteria_are_ordered (method)

StructuredRubricDraftV1.validate_criteria_are_ordered(value: object, info) -> object

validate_criteria (method)

StructuredRubricDraftV1.validate_criteria(value: tuple[StructuredRubricCriterionV1, ...]) -> tuple[StructuredRubricCriterionV1, ...]

validate_contract (method)

StructuredRubricDraftV1.validate_contract() -> StructuredRubricDraftV1

from_rubric (method)

StructuredRubricDraftV1.from_rubric(rubric: StructuredRubricV1) -> StructuredRubricDraftV1

compile (method)

StructuredRubricDraftV1.compile() -> StructuredRubricV1

Release source · Runtime guarantees · Capability guides

validate_eval_suite_selection

from cayu.evals import validate_eval_suite_selection

Match one selection to the exact immutable suite and case revisions.

validate_eval_suite_selection(selection: EvalSuiteSelectionV1, document: EvalSuiteDocument) -> EvalSuiteSelectionV1

Release source · Runtime guarantees · Capability guides

validate_expected_eval_suite_revision

from cayu.evals import validate_expected_eval_suite_revision

validate_expected_eval_suite_revision(document: EvalSuiteDocumentV1, expected_revision: str) -> EvalSuiteDocumentV1

Release source · Runtime guarantees · Capability guides