cayu.evals.scenario

Public Cayu API. Definitions are grouped by their public source module, and definitions from private implementation modules appear on their package’s page; adding a symbol does not move existing links. The import shown at each entry is a supported public namespace. Read the Python API overview and install the required extra.

`compile_eval_scenario`

from cayu.evals import compile_eval_scenario

Compile portable stimuli without resolving any execution authority.

compile_eval_scenario(scenario: EvalScenarioDocumentV2) -> CompiledEvalScenarioV2

Release source · Runtime guarantees · Capability guides

`CompiledEvalScenarioV2`

from cayu.evals import CompiledEvalScenarioV2

Validated authority-free template consumed by later launch preflight.

Decorators: @dataclass(frozen=True, slots=True).

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
`document`	`EvalScenarioDocumentV2`	`required`
`initial`	`ScenarioInitialInputEventV2`	`required`
`steps`	`tuple[ScenarioQueuedInputEventV2 \| ScenarioResumedInputEventV2 \| ScenarioApprovalCheckpointEventV2, ...]`	`required`

revision (property)

CompiledEvalScenarioV2.revision() -> str

target_key (property)

CompiledEvalScenarioV2.target_key() -> str

artifact_requirement (method)

CompiledEvalScenarioV2.artifact_requirement(requirement_id: str) -> ScenarioArtifactRequirementV2

secret_requirement (method)

CompiledEvalScenarioV2.secret_requirement(requirement_id: str) -> ScenarioSecretRequirementV2

Release source · Runtime guarantees · Capability guides

`eval_scenario_from_json`

from cayu.evals import eval_scenario_from_json

Load one bounded scenario-v2 JSON document from text.

eval_scenario_from_json(source: str) -> EvalScenarioDocumentV2

Release source · Runtime guarantees · Capability guides

`EVAL_SCENARIO_MAX_ARTIFACT_REQUIREMENTS`

from cayu.evals import EVAL_SCENARIO_MAX_ARTIFACT_REQUIREMENTS

EVAL_SCENARIO_MAX_ARTIFACT_REQUIREMENTS = 128

Release source · Runtime guarantees · Capability guides

`EVAL_SCENARIO_MAX_BYTES`

from cayu.evals import EVAL_SCENARIO_MAX_BYTES

EVAL_SCENARIO_MAX_BYTES = 8 << 20

Release source · Runtime guarantees · Capability guides

`EVAL_SCENARIO_MAX_EVENTS`

from cayu.evals import EVAL_SCENARIO_MAX_EVENTS

EVAL_SCENARIO_MAX_EVENTS = 1024

Release source · Runtime guarantees · Capability guides

`EVAL_SCENARIO_MAX_JSON_PART_BYTES`

from cayu.evals import EVAL_SCENARIO_MAX_JSON_PART_BYTES

EVAL_SCENARIO_MAX_JSON_PART_BYTES = 256 << 10

Release source · Runtime guarantees · Capability guides

`EVAL_SCENARIO_MAX_MESSAGES_PER_EVENT`

from cayu.evals import EVAL_SCENARIO_MAX_MESSAGES_PER_EVENT

EVAL_SCENARIO_MAX_MESSAGES_PER_EVENT = 32

Release source · Runtime guarantees · Capability guides

`EVAL_SCENARIO_MAX_PARTS_PER_MESSAGE`

from cayu.evals import EVAL_SCENARIO_MAX_PARTS_PER_MESSAGE

EVAL_SCENARIO_MAX_PARTS_PER_MESSAGE = 32

Release source · Runtime guarantees · Capability guides

`EVAL_SCENARIO_MAX_SECRET_REQUIREMENTS`

from cayu.evals import EVAL_SCENARIO_MAX_SECRET_REQUIREMENTS

EVAL_SCENARIO_MAX_SECRET_REQUIREMENTS = 128

Release source · Runtime guarantees · Capability guides

`EVAL_SCENARIO_MAX_TEXT_CHARS`

from cayu.evals import EVAL_SCENARIO_MAX_TEXT_CHARS

EVAL_SCENARIO_MAX_TEXT_CHARS = 65536

Release source · Runtime guarantees · Capability guides

`EVAL_SCENARIO_MAX_TOTAL_ARTIFACT_BYTES`

from cayu.evals import EVAL_SCENARIO_MAX_TOTAL_ARTIFACT_BYTES

EVAL_SCENARIO_MAX_TOTAL_ARTIFACT_BYTES = 1 << 30

Release source · Runtime guarantees · Capability guides

`EVAL_SCENARIO_MAX_TOTAL_TEXT_CHARS`

from cayu.evals import EVAL_SCENARIO_MAX_TOTAL_TEXT_CHARS

EVAL_SCENARIO_MAX_TOTAL_TEXT_CHARS = 1 << 20

Release source · Runtime guarantees · Capability guides

`EVAL_SCENARIO_SCHEMA_VERSION`

from cayu.evals import EVAL_SCENARIO_SCHEMA_VERSION

EVAL_SCENARIO_SCHEMA_VERSION = 2

Release source · Runtime guarantees · Capability guides

`eval_scenario_to_json`

from cayu.evals import eval_scenario_to_json

Return deterministic, human-readable scenario-v2 JSON.

eval_scenario_to_json(scenario: EvalScenarioDocumentV2) -> str

Release source · Runtime guarantees · Capability guides

`EvalScenarioDocumentV2`

from cayu.evals import EvalScenarioDocumentV2

One canonical external-stimulus sequence with no executable authority.

Bases: _SchemaV2PortableModel.

Fields and attributes

Name	Type	Default
`schema_version`	`Literal[2]`	`EVAL_SCENARIO_SCHEMA_VERSION`
`revision`	`StrictStr`	`required`
`id`	`StrictStr`	`required`
`target_key`	`StrictStr`	`required`
`name`	`StrictStr`	`required`
`description`	`StrictStr \| None`	`None`
`source`	`EvaluationSourceIdentityV1 \| None`	`None`
`events`	`tuple[ScenarioEventV2, ...]`	`Field(min_length=1, max_length=EVAL_SCENARIO_MAX_EVENTS)`
`artifact_requirements`	`tuple[ScenarioArtifactRequirementV2, ...]`	`Field(default_factory=tuple, max_length=EVAL_SCENARIO_MAX_ARTIFACT_REQUIREMENTS)`
`secret_requirements`	`tuple[ScenarioSecretRequirementV2, ...]`	`Field(default_factory=tuple, max_length=EVAL_SCENARIO_MAX_SECRET_REQUIREMENTS)`

validate_schema_version_type (method)

EvalScenarioDocumentV2.validate_schema_version_type(value: object) -> object

validate_collections_are_ordered (method)

EvalScenarioDocumentV2.validate_collections_are_ordered(value: object, info) -> object

validate_revision_shape (method)

EvalScenarioDocumentV2.validate_revision_shape(value: str, info) -> str

validate_ids (method)

EvalScenarioDocumentV2.validate_ids(value: str, info) -> str

validate_name (method)

EvalScenarioDocumentV2.validate_name(value: str, info) -> str

validate_description (method)

EvalScenarioDocumentV2.validate_description(value: str | None, info) -> str | None

validate_contract (method)

EvalScenarioDocumentV2.validate_contract() -> EvalScenarioDocumentV2

create (method)

EvalScenarioDocumentV2.create(*, id: str, target_key: str, name: str, events: Sequence[ScenarioEventV2], description: str | None=None, source: EvaluationSourceIdentityV1 | None=None, artifact_requirements: Sequence[ScenarioArtifactRequirementV2]=(), secret_requirements: Sequence[ScenarioSecretRequirementV2]=()) -> EvalScenarioDocumentV2

Release source · Runtime guarantees · Capability guides

`EvalScenarioInspectionV2`

from cayu.evals import EvalScenarioInspectionV2

Bases: _SchemaV2PortableModel.

Fields and attributes

Name	Type	Default
`schema_version`	`Literal[2]`	`EVAL_SCENARIO_SCHEMA_VERSION`
`revision`	`StrictStr`	`required`
`id`	`StrictStr`	`required`
`target_key`	`StrictStr`	`required`
`event_count`	`StrictInt`	`Field(ge=1, le=EVAL_SCENARIO_MAX_EVENTS)`
`input_event_count`	`StrictInt`	`Field(ge=1, le=EVAL_SCENARIO_MAX_EVENTS)`
`approval_checkpoint_count`	`StrictInt`	`Field(ge=0, le=EVAL_SCENARIO_MAX_EVENTS)`
`message_count`	`StrictInt`	`Field(ge=1, le=EVAL_SCENARIO_MAX_EVENTS * EVAL_SCENARIO_MAX_MESSAGES_PER_EVENT)`
`part_count`	`StrictInt`	`Field(ge=1, le=EVAL_SCENARIO_MAX_EVENTS * EVAL_SCENARIO_MAX_MESSAGES_PER_EVENT * EVAL_SCENARIO_MAX_PARTS_PER_MESSAGE)`
`artifact_requirement_count`	`StrictInt`	`Field(ge=0, le=EVAL_SCENARIO_MAX_ARTIFACT_REQUIREMENTS)`
`secret_requirement_count`	`StrictInt`	`Field(ge=0, le=EVAL_SCENARIO_MAX_SECRET_REQUIREMENTS)`

validate_schema_version_type (method)

EvalScenarioInspectionV2.validate_schema_version_type(value: object) -> object

validate_revision (method)

EvalScenarioInspectionV2.validate_revision(value: str, info) -> str

validate_ids (method)

EvalScenarioInspectionV2.validate_ids(value: str, info) -> str

validate_counts (method)

EvalScenarioInspectionV2.validate_counts() -> EvalScenarioInspectionV2

Release source · Runtime guarantees · Capability guides

`inspect_eval_scenario`

from cayu.evals import inspect_eval_scenario

inspect_eval_scenario(scenario: EvalScenarioDocumentV2) -> EvalScenarioInspectionV2

Release source · Runtime guarantees · Capability guides

`load_eval_scenario`

from cayu.evals import load_eval_scenario

Read at most the scenario hard limit before decoding or validating JSON.

load_eval_scenario(path: str | Path) -> EvalScenarioDocumentV2

Release source · Runtime guarantees · Capability guides

`scenario_from_corpus_case`

from cayu.evals import scenario_from_corpus_case

Lift one runnable corpus-v2 case into the scenario-v2 stimulus contract.

scenario_from_corpus_case(corpus: EvalCorpusDocument, case_id: str, *, scenario_id: str | None=None) -> EvalScenarioDocumentV2

Release source · Runtime guarantees · Capability guides

`ScenarioApprovalCheckpointEventV2`

from cayu.evals import ScenarioApprovalCheckpointEventV2

Require a new approval decision for a matching current tool call.

The checkpoint identifies an expected pause but contains no approve/deny choice, approval id, actor, or reusable authorization from the source run.

Bases: _ScenarioEventBaseV2.

Fields and attributes

Name	Type	Default
`sequence`	`StrictInt`	`Field(ge=0, le=EVAL_SCENARIO_MAX_EVENTS - 1)`
`id`	`StrictStr`	`required`
`kind`	`Literal['approval_checkpoint']`	`'approval_checkpoint'`
`tool_name`	`StrictStr`	`required`
`occurrence`	`StrictInt`	`Field(default=1, ge=1, le=EVAL_SCENARIO_MAX_EVENTS)`
`resolution`	`Literal['fresh_decision']`	`'fresh_decision'`

validate_id (method)

ScenarioApprovalCheckpointEventV2.validate_id(value: str, info) -> str

validate_tool_name (method)

ScenarioApprovalCheckpointEventV2.validate_tool_name(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

`ScenarioArtifactRequirementV2`

from cayu.evals import ScenarioArtifactRequirementV2

An immutable file requirement resolved and authorized only at launch.

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`id`	`StrictStr`	`required`
`source`	`Literal['fixture_digest', 'artifact_reference']`	`required`
`reference`	`StrictStr \| None`	`None`
`content_sha256`	`StrictStr`	`required`
`filename`	`StrictStr`	`required`
`content_type`	`StrictStr`	`required`
`size_bytes`	`StrictInt`	`Field(ge=1, le=MAX_DURABLE_JSON_INTEGER)`

validate_id (method)

ScenarioArtifactRequirementV2.validate_id(value: str, info) -> str

validate_content_sha256 (method)

ScenarioArtifactRequirementV2.validate_content_sha256(value: str, info) -> str

validate_reference (method)

ScenarioArtifactRequirementV2.validate_reference(value: str | None, info) -> str | None

validate_filename (method)

ScenarioArtifactRequirementV2.validate_filename(value: str, info) -> str

validate_content_type (method)

ScenarioArtifactRequirementV2.validate_content_type(value: str, info) -> str

validate_source_reference (method)

ScenarioArtifactRequirementV2.validate_source_reference() -> ScenarioArtifactRequirementV2

Release source · Runtime guarantees · Capability guides

`ScenarioEventV2`

from cayu.evals import ScenarioEventV2

ScenarioEventV2: TypeAlias = Annotated[ScenarioInitialInputEventV2 | ScenarioQueuedInputEventV2 | ScenarioResumedInputEventV2 | ScenarioApprovalCheckpointEventV2, Field(discriminator='kind')]

Release source · Runtime guarantees · Capability guides

`ScenarioFilePartV2`

from cayu.evals import ScenarioFilePartV2

A file input by immutable requirement id; it never carries file bytes.

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`type`	`Literal['file']`	`'file'`
`artifact_requirement_id`	`StrictStr`	`required`

validate_requirement_id (method)

ScenarioFilePartV2.validate_requirement_id(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

`ScenarioInitialInputEventV2`

from cayu.evals import ScenarioInitialInputEventV2

Input used to create the fresh scenario session.

Bases: _ScenarioEventBaseV2.

Fields and attributes

Name	Type	Default
`sequence`	`StrictInt`	`Field(ge=0, le=EVAL_SCENARIO_MAX_EVENTS - 1)`
`id`	`StrictStr`	`required`
`kind`	`Literal['initial']`	`'initial'`
`input`	`ScenarioInputV2`	`required`

validate_id (method)

ScenarioInitialInputEventV2.validate_id(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

`ScenarioInputPartV2`

from cayu.evals import ScenarioInputPartV2

ScenarioInputPartV2: TypeAlias = Annotated[ScenarioTextPartV2 | ScenarioJsonPartV2 | ScenarioFilePartV2, Field(discriminator='type')]

Release source · Runtime guarantees · Capability guides

`ScenarioInputV2`

from cayu.evals import ScenarioInputV2

One bounded batch of caller messages delivered at an explicit lifecycle point.

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`messages`	`tuple[ScenarioUserMessageV2, ...]`	`Field(min_length=1, max_length=EVAL_SCENARIO_MAX_MESSAGES_PER_EVENT)`

validate_messages_are_ordered (method)

ScenarioInputV2.validate_messages_are_ordered(value: object, info) -> object

create (method)

ScenarioInputV2.create(messages: Sequence[ScenarioUserMessageV2]) -> ScenarioInputV2

Release source · Runtime guarantees · Capability guides

`ScenarioJsonPartV2`

from cayu.evals import ScenarioJsonPartV2

One structured caller input preserved as durable JSON.

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`type`	`Literal['json']`	`'json'`
`value`	`Any`	`required`

validate_value (method)

ScenarioJsonPartV2.validate_value(value: object) -> object

freeze_value (method)

ScenarioJsonPartV2.freeze_value(value: Any) -> Any

serialize_value (method)

ScenarioJsonPartV2.serialize_value(value: Any) -> Any

Release source · Runtime guarantees · Capability guides

`ScenarioQueuedInputEventV2`

from cayu.evals import ScenarioQueuedInputEventV2

Input queued while the scenario session is active.

Bases: _ScenarioEventBaseV2.

Fields and attributes

Name	Type	Default
`sequence`	`StrictInt`	`Field(ge=0, le=EVAL_SCENARIO_MAX_EVENTS - 1)`
`id`	`StrictStr`	`required`
`kind`	`Literal['queued']`	`'queued'`
`delivery_mode`	`Literal['next_turn', 'on_idle']`	`required`
`input`	`ScenarioInputV2`	`required`

validate_id (method)

ScenarioQueuedInputEventV2.validate_id(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

`ScenarioResumedInputEventV2`

from cayu.evals import ScenarioResumedInputEventV2

Fresh caller input supplied when a scenario session is paused.

manual_recovery is the scenario-v2 wire name for an ordinary explicit CayuApp.resume(...) interaction. It does not assert or synthesize the outcome of a runtime tool call whose external effect is unknown.

Bases: _ScenarioEventBaseV2.

Fields and attributes

Name	Type	Default
`sequence`	`StrictInt`	`Field(ge=0, le=EVAL_SCENARIO_MAX_EVENTS - 1)`
`id`	`StrictStr`	`required`
`kind`	`Literal['resumed']`	`'resumed'`
`resume_kind`	`Literal['user_input', 'manual_recovery']`	`'user_input'`
`input`	`ScenarioInputV2`	`required`

validate_id (method)

ScenarioResumedInputEventV2.validate_id(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

`ScenarioSecretRequirementV2`

from cayu.evals import ScenarioSecretRequirementV2

A named server-side dependency; no secret value or vault handle is portable.

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`id`	`StrictStr`	`required`
`usage`	`Literal['provider', 'tool', 'environment', 'artifact', 'other']`	`required`
`purpose`	`StrictStr`	`required`

validate_id (method)

ScenarioSecretRequirementV2.validate_id(value: str, info) -> str

validate_purpose (method)

ScenarioSecretRequirementV2.validate_purpose(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

`ScenarioTextPartV2`

from cayu.evals import ScenarioTextPartV2

One literal caller-authored text part.

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`type`	`Literal['text']`	`'text'`
`text`	`StrictStr`	`required`

validate_text (method)

ScenarioTextPartV2.validate_text(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

`ScenarioUserMessageV2`

from cayu.evals import ScenarioUserMessageV2

One external user message with ordered text, JSON, or file-reference parts.

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`role`	`Literal['user']`	`'user'`
`content`	`tuple[ScenarioInputPartV2, ...]`	`Field(min_length=1, max_length=EVAL_SCENARIO_MAX_PARTS_PER_MESSAGE)`

validate_content_is_ordered (method)

ScenarioUserMessageV2.validate_content_is_ordered(value: object, info) -> object

create (method)

ScenarioUserMessageV2.create(parts: Sequence[ScenarioInputPartV2]) -> ScenarioUserMessageV2

Release source · Runtime guarantees · Capability guides