cayu.evals.corpus

Public Cayu API. Definitions are grouped by their public source module, and definitions from private implementation modules appear on their package’s page; adding a symbol does not move existing links. The import shown at each entry is a supported public namespace. Read the Python API overview and install the required extra.

ArtifactAssertionSpec

from cayu.evals import ArtifactAssertionSpec

Require bounded structural or explicitly retained public-text artifact evidence.

Bases: _AssertionSpecBase.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
kind	Literal['artifact']	'artifact'
scope	Literal['session', 'environment']	'session'
filename	StrictStr | None	None
content_type	StrictStr | None	None
minimum_bytes	StrictInt | None	Field(default=None, ge=0, le=EVIDENCE_MAX_TOTAL_TOKENS)
maximum_bytes	StrictInt | None	Field(default=None, ge=0, le=EVIDENCE_MAX_TOTAL_TOKENS)
sha256	StrictStr | None	None
text_contains	StrictStr | None	None
min_count	StrictInt	Field(default=1, ge=0, le=EVAL_ARTIFACT_EVIDENCE_MAX_ARTIFACTS)
max_count	StrictInt | None	Field(default=None, ge=0, le=EVAL_ARTIFACT_EVIDENCE_MAX_ARTIFACTS)

validate_id (method)

ArtifactAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

ArtifactAssertionSpec.validate_description(value: str | None, info) -> str | None

validate_filename (method)

ArtifactAssertionSpec.validate_filename(value: str | None, info) -> str | None

validate_content_type (method)

ArtifactAssertionSpec.validate_content_type(value: str | None, info) -> str | None

validate_sha256 (method)

ArtifactAssertionSpec.validate_sha256(value: str | None, info) -> str | None

validate_text_contains (method)

ArtifactAssertionSpec.validate_text_contains(value: str | None, info) -> str | None

validate_contract (method)

ArtifactAssertionSpec.validate_contract() -> ArtifactAssertionSpec

Release source · Runtime guarantees · Capability guides

assertion_spec_revision

from cayu.evals import assertion_spec_revision

assertion_spec_revision(spec: AssertionSpec) -> str

Release source · Runtime guarantees · Capability guides

AssertionSpec

from cayu.evals import AssertionSpec

AssertionSpec: TypeAlias = Annotated[RootStatusAssertionSpec | ChildStatusAssertionSpec | FinalOutputEqualsAssertionSpec | FinalOutputContainsAssertionSpec | ToolCalledAssertionSpec | ToolArgumentsContainAssertionSpec | ToolResultContainsAssertionSpec | ToolsCalledInOrderAssertionSpec | ProcessEventAssertionSpec | ProcessEventsInOrderAssertionSpec | WorkspaceFileAssertionSpec | ArtifactAssertionSpec | MemoryAttributionAssertionSpec | MaxToolCallsAssertionSpec | MaxModelStepsAssertionSpec | UsageRecordedAssertionSpec | MaxTotalTokensAssertionSpec | MaxEstimatedCostAssertionSpec | ModelJudgeAssertionSpec | StructuredModelJudgeAssertionSpec, Field(discriminator='kind')]

Release source · Runtime guarantees · Capability guides

ChildStatusAssertionSpec

from cayu.evals import ChildStatusAssertionSpec

Bases: _AssertionSpecBase.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
kind	Literal['child_status']	'child_status'
expected	Literal['completed', 'failed', 'interrupted']	required
min_count	StrictInt	Field(default=1, ge=0, le=EVIDENCE_MAX_CHILD_SESSIONS)
max_count	StrictInt | None	Field(default=None, ge=0, le=EVIDENCE_MAX_CHILD_SESSIONS)

validate_id (method)

ChildStatusAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

ChildStatusAssertionSpec.validate_description(value: str | None, info) -> str | None

validate_count_range (method)

ChildStatusAssertionSpec.validate_count_range() -> ChildStatusAssertionSpec

Release source · Runtime guarantees · Capability guides

CorpusUserMessageSpec

from cayu.evals import CorpusUserMessageSpec

One portable user-role text message; no structured or executable input.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
role	Literal['user']	'user'
text	StrictStr	required

validate_text (method)

CorpusUserMessageSpec.validate_text(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

eval_corpus_from_json

from cayu.evals import eval_corpus_from_json

Load one bounded corpus v4 JSON document from text.

eval_corpus_from_json(source: str) -> EvalCorpusDocument

Release source · Runtime guarantees · Capability guides

eval_corpus_inspection_to_json

from cayu.evals import eval_corpus_inspection_to_json

Return deterministic JSON for one defensive corpus inspection.

eval_corpus_inspection_to_json(inspection: EvalCorpusInspectionV1) -> str

Release source · Runtime guarantees · Capability guides

EVAL_CORPUS_MAX_ASSERTIONS_PER_CASE

from cayu.evals import EVAL_CORPUS_MAX_ASSERTIONS_PER_CASE

EVAL_CORPUS_MAX_ASSERTIONS_PER_CASE = 64

Release source · Runtime guarantees · Capability guides

EVAL_CORPUS_MAX_BYTES

from cayu.evals import EVAL_CORPUS_MAX_BYTES

EVAL_CORPUS_MAX_BYTES = 8 << 20

Release source · Runtime guarantees · Capability guides

EVAL_CORPUS_MAX_CASES

from cayu.evals import EVAL_CORPUS_MAX_CASES

EVAL_CORPUS_MAX_CASES = 1000

Release source · Runtime guarantees · Capability guides

EVAL_CORPUS_MAX_MERGE_INPUTS

from cayu.evals import EVAL_CORPUS_MAX_MERGE_INPUTS

EVAL_CORPUS_MAX_MERGE_INPUTS = 256

Release source · Runtime guarantees · Capability guides

EVAL_CORPUS_MAX_MESSAGE_CHARS

from cayu.evals import EVAL_CORPUS_MAX_MESSAGE_CHARS

EVAL_CORPUS_MAX_MESSAGE_CHARS = 65536

Release source · Runtime guarantees · Capability guides

EVAL_CORPUS_MAX_MESSAGES_PER_CASE

from cayu.evals import EVAL_CORPUS_MAX_MESSAGES_PER_CASE

EVAL_CORPUS_MAX_MESSAGES_PER_CASE = 16

Release source · Runtime guarantees · Capability guides

EVAL_CORPUS_MAX_PUBLISHED_ASSERTION_RESULTS

from cayu.evals import EVAL_CORPUS_MAX_PUBLISHED_ASSERTION_RESULTS

EVAL_CORPUS_MAX_PUBLISHED_ASSERTION_RESULTS = 10000

Release source · Runtime guarantees · Capability guides

EVAL_CORPUS_MAX_SUITES

from cayu.evals import EVAL_CORPUS_MAX_SUITES

EVAL_CORPUS_MAX_SUITES = 64

Release source · Runtime guarantees · Capability guides

EVAL_CORPUS_MAX_TIMEOUT_SECONDS

from cayu.evals import EVAL_CORPUS_MAX_TIMEOUT_SECONDS

EVAL_CORPUS_MAX_TIMEOUT_SECONDS = 3600

Release source · Runtime guarantees · Capability guides

EVAL_CORPUS_MAX_TOTAL_MESSAGE_CHARS

from cayu.evals import EVAL_CORPUS_MAX_TOTAL_MESSAGE_CHARS

EVAL_CORPUS_MAX_TOTAL_MESSAGE_CHARS = 262144

Release source · Runtime guarantees · Capability guides

EVAL_CORPUS_MAX_TRIALS

from cayu.evals import EVAL_CORPUS_MAX_TRIALS

EVAL_CORPUS_MAX_TRIALS = 100

Release source · Runtime guarantees · Capability guides

EVAL_CORPUS_SCHEMA_VERSION

from cayu.evals import EVAL_CORPUS_SCHEMA_VERSION

EVAL_CORPUS_SCHEMA_VERSION = 4

Release source · Runtime guarantees · Capability guides

eval_corpus_to_json

from cayu.evals import eval_corpus_to_json

Return deterministic, human-readable corpus v4 JSON.

eval_corpus_to_json(corpus: EvalCorpusDocument) -> str

Release source · Runtime guarantees · Capability guides

eval_run_contract_for_corpus

from cayu.evals import eval_run_contract_for_corpus

Freeze the exact portable contract that must precede suite execution.

eval_run_contract_for_corpus(corpus: EvalCorpusDocument, suite_id: str) -> EvalRunContractV2

Release source · Runtime guarantees · Capability guides

EvalCaseSpec

from cayu.evals import EvalCaseSpec

One deterministic expectation set with optional fresh-run input.

Captured-session evaluations intentionally set input to None when the retained evidence cannot be represented as one corpus-v2 invocation. Such cases remain portable historical evaluation contracts, but execution rejects them until a runnable input or scenario is authored.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
revision	StrictStr	required
suite_id	StrictStr	required
name	StrictStr	required
description	StrictStr | None	None
source	EvaluationSourceIdentityV1	required
input	RunInputSpec | None	required
assertions	tuple[AssertionSpec, ...]	Field(min_length=1, max_length=EVAL_CORPUS_MAX_ASSERTIONS_PER_CASE)

validate_assertions_are_ordered (method)

EvalCaseSpec.validate_assertions_are_ordered(value: object, info) -> object

validate_id (method)

EvalCaseSpec.validate_id(value: str, info) -> str

validate_revision_shape (method)

EvalCaseSpec.validate_revision_shape(value: str, info) -> str

validate_name (method)

EvalCaseSpec.validate_name(value: str, info) -> str

validate_description (method)

EvalCaseSpec.validate_description(value: str | None, info) -> str | None

validate_contract (method)

EvalCaseSpec.validate_contract() -> EvalCaseSpec

create (method)

EvalCaseSpec.create(*, id: str, suite_id: str, name: str, source: EvaluationSourceIdentityV1, input: RunInputSpec | None, assertions: Sequence[AssertionSpec], description: str | None=None) -> EvalCaseSpec

Release source · Runtime guarantees · Capability guides

EvalCorpusDocument

from cayu.evals import EvalCorpusDocument

One canonical, authority-free corpus for exactly one trusted target key.

Bases: _SchemaV4PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[4]	EVAL_CORPUS_SCHEMA_VERSION
revision	StrictStr	required
target_key	StrictStr	required
evidence_policy	EvaluationEvidencePolicySpec	required
pricing_profile	PricingProfileIdentityV1 | None	None
suites	tuple[EvalSuiteSpec, ...]	Field(min_length=1, max_length=EVAL_CORPUS_MAX_SUITES)
cases	tuple[EvalCaseSpec, ...]	Field(min_length=1, max_length=EVAL_CORPUS_MAX_CASES)

validate_schema_version_type (method)

EvalCorpusDocument.validate_schema_version_type(value: object) -> object

validate_collections_are_ordered (method)

EvalCorpusDocument.validate_collections_are_ordered(value: object, info) -> object

validate_revision_shape (method)

EvalCorpusDocument.validate_revision_shape(value: str, info) -> str

validate_target_key (method)

EvalCorpusDocument.validate_target_key(value: str, info) -> str

validate_contract (method)

EvalCorpusDocument.validate_contract() -> EvalCorpusDocument

create (method)

EvalCorpusDocument.create(*, target_key: str, evidence_policy: EvaluationEvidencePolicySpec, suites: Sequence[EvalSuiteSpec], cases: Sequence[EvalCaseSpec], pricing_profile: PricingProfileIdentityV1 | None=None) -> EvalCorpusDocument

Release source · Runtime guarantees · Capability guides

EvalCorpusInspectionV1

from cayu.evals import EvalCorpusInspectionV1

Stable authority-free summary returned by corpus validation and inspection.

Bases: _SchemaV1PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[1]	1
revision	StrictStr	required
target_key	StrictStr	required
evidence_policy_revision	StrictStr	required
pricing_profile_fingerprint	StrictStr | None	None
suite_count	StrictInt	Field(ge=1, le=EVAL_CORPUS_MAX_SUITES)
case_count	StrictInt	Field(ge=1, le=EVAL_CORPUS_MAX_CASES)
assertion_count	StrictInt	Field(ge=1)
expanded_assertion_result_count	StrictInt	Field(ge=1, le=EVAL_CORPUS_MAX_SUITES * EVAL_CORPUS_MAX_PUBLISHED_ASSERTION_RESULTS)
suites	tuple[EvalCorpusSuiteInspectionV1, ...]	Field(min_length=1, max_length=EVAL_CORPUS_MAX_SUITES)

validate_schema_version_type (method)

EvalCorpusInspectionV1.validate_schema_version_type(value: object) -> object

validate_revision_fields (method)

EvalCorpusInspectionV1.validate_revision_fields(value: str | None, info) -> str | None

validate_target_key (method)

EvalCorpusInspectionV1.validate_target_key(value: str, info) -> str

validate_suites_are_ordered (method)

EvalCorpusInspectionV1.validate_suites_are_ordered(value: object, info) -> object

validate_summary (method)

EvalCorpusInspectionV1.validate_summary() -> EvalCorpusInspectionV1

Release source · Runtime guarantees · Capability guides

EvalCorpusSuiteInspectionV1

from cayu.evals import EvalCorpusSuiteInspectionV1

Bounded structural summary for one corpus suite.

Bases: _SchemaV1PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[1]	1
id	StrictStr	required
revision	StrictStr	required
name	StrictStr	required
case_count	StrictInt	Field(ge=1, le=EVAL_CORPUS_MAX_CASES)
assertion_count	StrictInt	Field(ge=1)
trials	StrictInt	Field(ge=1, le=EVAL_CORPUS_MAX_TRIALS)
minimum_passed_trials	StrictInt	Field(ge=1, le=EVAL_CORPUS_MAX_TRIALS)
max_concurrency	StrictInt	Field(ge=1)
timeout_seconds	StrictInt	Field(ge=1, le=EVAL_CORPUS_MAX_TIMEOUT_SECONDS)

validate_schema_version_type (method)

EvalCorpusSuiteInspectionV1.validate_schema_version_type(value: object) -> object

validate_id (method)

EvalCorpusSuiteInspectionV1.validate_id(value: str, info) -> str

validate_revision (method)

EvalCorpusSuiteInspectionV1.validate_revision(value: str, info) -> str

validate_name (method)

EvalCorpusSuiteInspectionV1.validate_name(value: str, info) -> str

validate_counts (method)

EvalCorpusSuiteInspectionV1.validate_counts() -> EvalCorpusSuiteInspectionV1

Release source · Runtime guarantees · Capability guides

EvalJudgeEvidenceSelectionV1

from cayu.evals import EvalJudgeEvidenceSelectionV1

Candidate evidence a public judge profile is requested to receive.

Bases: _SchemaV1PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[1]	1
include_final_output	Literal[True]	True
include_transcript	StrictBool	False

validate_schema_version_type (method)

EvalJudgeEvidenceSelectionV1.validate_schema_version_type(value: object) -> object

validate_final_output_type (method)

EvalJudgeEvidenceSelectionV1.validate_final_output_type(value: object) -> object

Release source · Runtime guarantees · Capability guides

EvalProcessEventKind

from cayu.evals import EvalProcessEventKind

EvalProcessEventKind: TypeAlias = Literal['session_started', 'session_resumed', 'session_awaiting_user_input', 'session_completed', 'session_failed', 'session_interrupted', 'session_limit_reached', 'tool_call_started', 'tool_call_completed', 'tool_call_failed', 'tool_call_blocked', 'tool_approval_requested', 'tool_approved', 'tool_approval_denied', 'tool_approval_expired', 'structured_output_validated', 'structured_output_failed', 'budget_limit_reached']

Release source · Runtime guarantees · Capability guides

EvalSuiteSpec

from cayu.evals import EvalSuiteSpec

Reusable execution settings; case membership remains mergeable by suite_id.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
revision	StrictStr	required
name	StrictStr	required
description	StrictStr | None	None
trial_request	TrialRequestSpec	Field(default_factory=TrialRequestSpec)

validate_id (method)

EvalSuiteSpec.validate_id(value: str, info) -> str

validate_revision_shape (method)

EvalSuiteSpec.validate_revision_shape(value: str, info) -> str

validate_name (method)

EvalSuiteSpec.validate_name(value: str, info) -> str

validate_description (method)

EvalSuiteSpec.validate_description(value: str | None, info) -> str | None

validate_revision (method)

EvalSuiteSpec.validate_revision() -> EvalSuiteSpec

create (method)

EvalSuiteSpec.create(*, id: str, name: str, description: str | None=None, trial_request: TrialRequestSpec | None=None, _preserve_missing_trial_policy: bool=False) -> EvalSuiteSpec

Release source · Runtime guarantees · Capability guides

EvaluationEvidencePolicySpec

from cayu.evals import EvaluationEvidencePolicySpec

Complete v1 redaction, tool retention, and cardinality behavior.

Bases: _SchemaV1PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[1]	EVALUATION_EVIDENCE_POLICY_SCHEMA_VERSION
revision	StrictStr	required
event_projection	Literal['runtime_public_alias_free_v1']	'runtime_public_alias_free_v1'
include_event_payloads	Literal[False]	False
include_transcript_text	Literal[False]	False
include_tool_arguments	StrictBool	True
include_tool_results	StrictBool	False
include_artifact_text	StrictBool	False
max_final_output_chars	Literal[65536]	EVIDENCE_MAX_FINAL_OUTPUT_CHARS
max_child_sessions	Literal[500]	EVIDENCE_MAX_CHILD_SESSIONS
max_tool_calls	Literal[4096]	EVIDENCE_MAX_TOOL_CALLS
max_model_steps	Literal[4096]	EVIDENCE_MAX_MODEL_STEPS
max_total_tokens	Literal[9007199254740991]	EVIDENCE_MAX_TOTAL_TOKENS

validate_schema_version_type (method)

EvaluationEvidencePolicySpec.validate_schema_version_type(value: object) -> object

validate_omission_flag_types (method)

EvaluationEvidencePolicySpec.validate_omission_flag_types(value: object, info) -> object

validate_tool_evidence_flag_types (method)

EvaluationEvidencePolicySpec.validate_tool_evidence_flag_types(value: object, info) -> object

validate_revision_shape (method)

EvaluationEvidencePolicySpec.validate_revision_shape(value: str, info) -> str

validate_revision (method)

EvaluationEvidencePolicySpec.validate_revision() -> EvaluationEvidencePolicySpec

create (method)

EvaluationEvidencePolicySpec.create(*, include_tool_arguments: bool=True, include_tool_results: bool=False, include_artifact_text: bool=False) -> EvaluationEvidencePolicySpec

standard (method)

EvaluationEvidencePolicySpec.standard() -> EvaluationEvidencePolicySpec

Default policy: existing public arguments, but no result retention.

supported_revisions (method)

EvaluationEvidencePolicySpec.supported_revisions() -> frozenset[str]

Every fixed policy combination understood by this wire generation.

policy_for_revision (method)

EvaluationEvidencePolicySpec.policy_for_revision(revision: str) -> EvaluationEvidencePolicySpec

Resolve one supported revision to the exact retention policy it identifies.

Release source · Runtime guarantees · Capability guides

EvaluationSourceIdentityV1

from cayu.evals import EvaluationSourceIdentityV1

Diagnostic capture or authored-definition provenance without runtime authority.

Bases: _SchemaV1PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[1]	EVALUATION_SOURCE_IDENTITY_SCHEMA_VERSION
application_release_id	StrictStr	required
app_manifest_schema_version	StrictStr	required
app_manifest_fingerprint	StrictStr	required
evidence_revision	StrictStr	required

validate_schema_version_type (method)

EvaluationSourceIdentityV1.validate_schema_version_type(value: object) -> object

validate_release_id (method)

EvaluationSourceIdentityV1.validate_release_id(value: str, info) -> str

validate_manifest_schema_version (method)

EvaluationSourceIdentityV1.validate_manifest_schema_version(value: str, info) -> str

validate_manifest_fingerprint (method)

EvaluationSourceIdentityV1.validate_manifest_fingerprint(value: str, info) -> str

validate_evidence_revision (method)

EvaluationSourceIdentityV1.validate_evidence_revision(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

FinalOutputContainsAssertionSpec

from cayu.evals import FinalOutputContainsAssertionSpec

Bases: _AssertionSpecBase.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
kind	Literal['final_output_contains']	'final_output_contains'
expected	StrictStr	required

validate_id (method)

FinalOutputContainsAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

FinalOutputContainsAssertionSpec.validate_description(value: str | None, info) -> str | None

validate_expected (method)

FinalOutputContainsAssertionSpec.validate_expected(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

FinalOutputEqualsAssertionSpec

from cayu.evals import FinalOutputEqualsAssertionSpec

Bases: _AssertionSpecBase.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
kind	Literal['final_output_equals']	'final_output_equals'
expected	StrictStr	required
comparison	Literal['text', 'json']	Field(default='text', exclude_if=lambda value: value == 'text')

validate_id (method)

FinalOutputEqualsAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

FinalOutputEqualsAssertionSpec.validate_description(value: str | None, info) -> str | None

validate_comparison (method)

FinalOutputEqualsAssertionSpec.validate_comparison() -> FinalOutputEqualsAssertionSpec

validate_expected (method)

FinalOutputEqualsAssertionSpec.validate_expected(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

inspect_eval_corpus

from cayu.evals import inspect_eval_corpus

Return a defensive bounded structural summary of one corpus.

inspect_eval_corpus(corpus: EvalCorpusDocument) -> EvalCorpusInspectionV1

Release source · Runtime guarantees · Capability guides

JudgePrivacyPolicyV1

from cayu.evals import JudgePrivacyPolicyV1

Public identity of the server policy controlling evaluator-only data.

Bases: _SchemaV1PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[1]	1
key	StrictStr	required
revision	StrictStr	required
allow_transcript	StrictBool	False
allow_public_reference	StrictBool	True
allow_private_reference	StrictBool	False

validate_schema_version_type (method)

JudgePrivacyPolicyV1.validate_schema_version_type(value: object) -> object

validate_key (method)

JudgePrivacyPolicyV1.validate_key(value: str, info) -> str

validate_revision_shape (method)

JudgePrivacyPolicyV1.validate_revision_shape(value: str, info) -> str

validate_contract (method)

JudgePrivacyPolicyV1.validate_contract() -> JudgePrivacyPolicyV1

create (method)

JudgePrivacyPolicyV1.create(*, key: str, allow_transcript: bool=False, allow_public_reference: bool=True, allow_private_reference: bool=False) -> JudgePrivacyPolicyV1

public_only (method)

JudgePrivacyPolicyV1.public_only() -> JudgePrivacyPolicyV1

Release source · Runtime guarantees · Capability guides

JudgeProfileIdentityV1

from cayu.evals import JudgeProfileIdentityV1

Safe, immutable public snapshot of one trusted model-judge route.

Bases: _SchemaV1PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[1]	1
key	StrictStr	required
revision	StrictStr	required
label	StrictStr	required
provider_name	StrictStr	required
model	StrictStr	required
implementation_revision	StrictStr	required
allowed_evidence	tuple[Literal['final_output', 'transcript', 'public_reference', 'private_reference'], ...]	Field(min_length=1, max_length=4)
timeout_seconds	StrictInt	Field(ge=1, le=EVAL_CORPUS_MAX_TIMEOUT_SECONDS)
max_input_tokens	StrictInt	Field(ge=1, le=EVIDENCE_MAX_TOTAL_TOKENS)
max_output_tokens	StrictInt	Field(ge=1, le=EVIDENCE_MAX_TOTAL_TOKENS)
max_total_tokens	StrictInt	Field(ge=1, le=EVIDENCE_MAX_TOTAL_TOKENS)
max_estimated_cost	StrictStr | None	None
cost_currency	StrictStr | None	None
pricing_profile_fingerprint	StrictStr | None	None
privacy_policy_key	StrictStr	required
privacy_policy_revision	StrictStr	required
same_model_use	Literal['forbidden', 'allowed_and_labeled']	'forbidden'

validate_schema_version_type (method)

JudgeProfileIdentityV1.validate_schema_version_type(value: object) -> object

validate_ids (method)

JudgeProfileIdentityV1.validate_ids(value: str, info) -> str

validate_revisions (method)

JudgeProfileIdentityV1.validate_revisions(value: str, info) -> str

validate_public_text (method)

JudgeProfileIdentityV1.validate_public_text(value: str, info) -> str

validate_allowed_evidence_order (method)

JudgeProfileIdentityV1.validate_allowed_evidence_order(value: object, info) -> object

validate_max_estimated_cost (method)

JudgeProfileIdentityV1.validate_max_estimated_cost(value: str | None, info) -> str | None

validate_cost_currency (method)

JudgeProfileIdentityV1.validate_cost_currency(value: str | None, info) -> str | None

validate_pricing_fingerprint (method)

JudgeProfileIdentityV1.validate_pricing_fingerprint(value: str | None, info) -> str | None

validate_contract (method)

JudgeProfileIdentityV1.validate_contract() -> JudgeProfileIdentityV1

comparison_revision (property)

JudgeProfileIdentityV1.comparison_revision() -> str

Return semantic judge-route identity without its presentation label.

Release source · Runtime guarantees · Capability guides

JudgeReferenceV1

from cayu.evals import JudgeReferenceV1

JudgeReferenceV1: TypeAlias = Annotated[PublicJudgeReferenceV1 | PrivateJudgeReferenceV1, Field(discriminator='kind')]

Release source · Runtime guarantees · Capability guides

load_eval_corpus

from cayu.evals import load_eval_corpus

Read at most the corpus hard limit before decoding or validating JSON.

load_eval_corpus(path: str | Path) -> EvalCorpusDocument

Release source · Runtime guarantees · Capability guides

MaxEstimatedCostAssertionSpec

from cayu.evals import MaxEstimatedCostAssertionSpec

Bases: _AssertionSpecBase.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
kind	Literal['max_estimated_cost']	'max_estimated_cost'
maximum	StrictStr	required
currency	StrictStr	'USD'

validate_id (method)

MaxEstimatedCostAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

MaxEstimatedCostAssertionSpec.validate_description(value: str | None, info) -> str | None

validate_maximum (method)

MaxEstimatedCostAssertionSpec.validate_maximum(value: str, info) -> str

validate_currency (method)

MaxEstimatedCostAssertionSpec.validate_currency(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

MaxModelStepsAssertionSpec

from cayu.evals import MaxModelStepsAssertionSpec

Bases: _AssertionSpecBase.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
kind	Literal['max_model_steps']	'max_model_steps'
maximum	StrictInt	Field(ge=0, le=EVIDENCE_MAX_MODEL_STEPS)

validate_id (method)

MaxModelStepsAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

MaxModelStepsAssertionSpec.validate_description(value: str | None, info) -> str | None

Release source · Runtime guarantees · Capability guides

MaxToolCallsAssertionSpec

from cayu.evals import MaxToolCallsAssertionSpec

Bases: _AssertionSpecBase.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
kind	Literal['max_tool_calls']	'max_tool_calls'
maximum	StrictInt	Field(ge=0, le=EVIDENCE_MAX_TOOL_CALLS)

validate_id (method)

MaxToolCallsAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

MaxToolCallsAssertionSpec.validate_description(value: str | None, info) -> str | None

Release source · Runtime guarantees · Capability guides

MaxTotalTokensAssertionSpec

from cayu.evals import MaxTotalTokensAssertionSpec

Bases: _AssertionSpecBase.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
kind	Literal['max_total_tokens']	'max_total_tokens'
maximum	StrictInt	Field(ge=0, le=EVIDENCE_MAX_TOTAL_TOKENS)

validate_id (method)

MaxTotalTokensAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

MaxTotalTokensAssertionSpec.validate_description(value: str | None, info) -> str | None

Release source · Runtime guarantees · Capability guides

MemoryAttributionAssertionSpec

from cayu.evals import MemoryAttributionAssertionSpec

Require complete bounded memory admission and provider-exposure evidence.

Bases: _AssertionSpecBase.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
kind	Literal['memory_attribution']	'memory_attribution'
min_admitted_items	StrictInt	Field(default=1, ge=0, le=EVAL_MEMORY_ATTRIBUTION_MAX_ADMITTED_ITEMS)
max_admitted_items	StrictInt | None	Field(default=None, ge=0, le=EVAL_MEMORY_ATTRIBUTION_MAX_ADMITTED_ITEMS)
min_provider_exposures	StrictInt	Field(default=1, ge=0, le=EVAL_MEMORY_ATTRIBUTION_MAX_PROVIDER_EXPOSURES)
max_provider_exposures	StrictInt | None	Field(default=None, ge=0, le=EVAL_MEMORY_ATTRIBUTION_MAX_PROVIDER_EXPOSURES)

validate_id (method)

MemoryAttributionAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

MemoryAttributionAssertionSpec.validate_description(value: str | None, info) -> str | None

validate_count_ranges (method)

MemoryAttributionAssertionSpec.validate_count_ranges() -> MemoryAttributionAssertionSpec

Release source · Runtime guarantees · Capability guides

merge_eval_corpora

from cayu.evals import merge_eval_corpora

Merge compatible corpora by immutable ID/revision contracts.

merge_eval_corpora(corpora: list[EvalCorpusDocument] | tuple[EvalCorpusDocument, ...], *, replace_conflicts: bool=False) -> EvalCorpusDocument

Release source · Runtime guarantees · Capability guides

merge_eval_corpus_files

from cayu.evals import merge_eval_corpus_files

Validate, merge, and atomically replace one corpus destination.

merge_eval_corpus_files(destination: str | Path, inputs: list[str | Path] | tuple[str | Path, ...], *, replace_conflicts: bool=False) -> EvalCorpusDocument

Release source · Runtime guarantees · Capability guides

ModelJudgeAssertionSpec

from cayu.evals import ModelJudgeAssertionSpec

Authority-free graded evaluation resolved by one trusted target.

Bases: _AssertionSpecBase.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
kind	Literal['model_judge']	'model_judge'
evaluator_key	StrictStr	required
rubric	StrictStr	required
rubric_version	StrictStr	required
threshold	StrictFloat	Field(default=0.5, ge=0.0, le=1.0, allow_inf_nan=False)
include_transcript	StrictBool	False

validate_id (method)

ModelJudgeAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

ModelJudgeAssertionSpec.validate_description(value: str | None, info) -> str | None

validate_evaluator_key (method)

ModelJudgeAssertionSpec.validate_evaluator_key(value: str, info) -> str

validate_rubric (method)

ModelJudgeAssertionSpec.validate_rubric(value: str, info) -> str

validate_rubric_version (method)

ModelJudgeAssertionSpec.validate_rubric_version(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

pricing_profile_identity

from cayu.evals import pricing_profile_identity

Fingerprint one validated PriceBook without carrying its executable pricing data.

pricing_profile_identity(price_book: PriceBook) -> PricingProfileIdentityV1

Release source · Runtime guarantees · Capability guides

PRICING_PROFILE_SEMANTICS_VERSION

from cayu.evals import PRICING_PROFILE_SEMANTICS_VERSION

PRICING_PROFILE_SEMANTICS_VERSION = 1

Release source · Runtime guarantees · Capability guides

PricingProfileIdentityV1

from cayu.evals import PricingProfileIdentityV1

Content identity for trusted pricing, never the actual PriceBook.

Bases: _SchemaV1PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[1]	PRICING_PROFILE_IDENTITY_SCHEMA_VERSION
pricing_semantics_version	Literal[1]	PRICING_PROFILE_SEMANTICS_VERSION
fingerprint	StrictStr	required
price_book_version	StrictStr	required
generated_at	StrictStr	required
currencies	tuple[StrictStr, ...]	Field(min_length=1, max_length=32)

validate_schema_version_type (method)

PricingProfileIdentityV1.validate_schema_version_type(value: object) -> object

validate_pricing_semantics_version_type (method)

PricingProfileIdentityV1.validate_pricing_semantics_version_type(value: object) -> object

validate_fingerprint (method)

PricingProfileIdentityV1.validate_fingerprint(value: str, info) -> str

validate_identity_text (method)

PricingProfileIdentityV1.validate_identity_text(value: str, info) -> str

validate_currencies (method)

PricingProfileIdentityV1.validate_currencies(value: object, info) -> tuple[str, ...]

Release source · Runtime guarantees · Capability guides

PrivateJudgeReferenceV1

from cayu.evals import PrivateJudgeReferenceV1

Authority-free identity for evaluator truth retained only by the server.

Bases: _SchemaV1PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[1]	1
kind	Literal['private_reference']	'private_reference'
key	StrictStr	required
revision	StrictStr	required
privacy_policy_key	StrictStr	required
privacy_policy_revision	StrictStr	required

validate_schema_version_type (method)

PrivateJudgeReferenceV1.validate_schema_version_type(value: object) -> object

validate_ids (method)

PrivateJudgeReferenceV1.validate_ids(value: str, info) -> str

validate_revisions (method)

PrivateJudgeReferenceV1.validate_revisions(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

ProcessEventAssertionSpec

from cayu.evals import ProcessEventAssertionSpec

Require or forbid one closed, payload-free runtime process fact.

Bases: _AssertionSpecBase.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
kind	Literal['process_event']	'process_event'
event	EvalProcessEventKind	required
min_count	StrictInt	Field(default=1, ge=0, le=EVAL_PROCESS_EVENT_EVIDENCE_MAX_EVENTS)
max_count	StrictInt | None	Field(default=None, ge=0, le=EVAL_PROCESS_EVENT_EVIDENCE_MAX_EVENTS)

validate_id (method)

ProcessEventAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

ProcessEventAssertionSpec.validate_description(value: str | None, info) -> str | None

validate_count_range (method)

ProcessEventAssertionSpec.validate_count_range() -> ProcessEventAssertionSpec

Release source · Runtime guarantees · Capability guides

ProcessEventsInOrderAssertionSpec

from cayu.evals import ProcessEventsInOrderAssertionSpec

Require the exact filtered order of selected portable process facts.

Bases: _AssertionSpecBase.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
kind	Literal['process_events_in_order']	'process_events_in_order'
events	tuple[EvalProcessEventKind, ...]	Field(min_length=1, max_length=EVAL_CORPUS_MAX_PROCESS_EVENTS)

validate_id (method)

ProcessEventsInOrderAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

ProcessEventsInOrderAssertionSpec.validate_description(value: str | None, info) -> str | None

validate_events_are_ordered (method)

ProcessEventsInOrderAssertionSpec.validate_events_are_ordered(value: object, info) -> object

Release source · Runtime guarantees · Capability guides

PublicJudgeReferenceV1

from cayu.evals import PublicJudgeReferenceV1

Evaluator-only, deliberately portable reference truth.

Bases: _SchemaV1PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[1]	1
kind	Literal['public_reference']	'public_reference'
id	StrictStr	required
revision	StrictStr	required
expected_answer	StrictStr | None	None
expected_facts	tuple[StrictStr, ...]	Field(default=(), max_length=EVAL_CORPUS_MAX_JUDGE_REFERENCE_FACTS)

validate_schema_version_type (method)

PublicJudgeReferenceV1.validate_schema_version_type(value: object) -> object

validate_id (method)

PublicJudgeReferenceV1.validate_id(value: str, info) -> str

validate_revision_shape (method)

PublicJudgeReferenceV1.validate_revision_shape(value: str, info) -> str

validate_expected_answer (method)

PublicJudgeReferenceV1.validate_expected_answer(value: str | None, info) -> str | None

validate_expected_facts (method)

PublicJudgeReferenceV1.validate_expected_facts(value: object, info) -> tuple[str, ...]

validate_contract (method)

PublicJudgeReferenceV1.validate_contract() -> PublicJudgeReferenceV1

create (method)

PublicJudgeReferenceV1.create(*, id: str, expected_answer: str | None=None, expected_facts: Sequence[str]=()) -> PublicJudgeReferenceV1

Release source · Runtime guarantees · Capability guides

RootStatusAssertionSpec

from cayu.evals import RootStatusAssertionSpec

Bases: _AssertionSpecBase.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
kind	Literal['root_status']	'root_status'
expected	Literal['completed', 'failed']	required

validate_id (method)

RootStatusAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

RootStatusAssertionSpec.validate_description(value: str | None, info) -> str | None

Release source · Runtime guarantees · Capability guides

RunInputSpec

from cayu.evals import RunInputSpec

The authority-free user input supplied after a trusted local bootstrap.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
messages	tuple[CorpusUserMessageSpec, ...]	Field(default=(), max_length=EVAL_CORPUS_MAX_MESSAGES_PER_CASE)
opaque_external_case_ref	OpaqueExternalCaseRefV1 | None	Field(default=None, exclude_if=lambda value: value is None)

validate_messages_are_ordered (method)

RunInputSpec.validate_messages_are_ordered(value: object, info) -> object

copy_opaque_external_case_ref (method)

RunInputSpec.copy_opaque_external_case_ref(value: object) -> object

validate_total_text (method)

RunInputSpec.validate_total_text() -> RunInputSpec

Release source · Runtime guarantees · Capability guides

StructuredModelJudgeAssertionSpec

from cayu.evals import StructuredModelJudgeAssertionSpec

Typed rubric judgment bound to an exact trusted server profile.

Bases: _AssertionSpecBase.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
kind	Literal['structured_model_judge']	'structured_model_judge'
judge_profile_key	StrictStr	required
judge_profile_revision	StrictStr	required
rubric	StructuredRubricV1	required
reference	JudgeReferenceV1 | None	None
threshold	StrictStr	'0.5'
evidence	EvalJudgeEvidenceSelectionV1	Field(default_factory=EvalJudgeEvidenceSelectionV1)

validate_id (method)

StructuredModelJudgeAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

StructuredModelJudgeAssertionSpec.validate_description(value: str | None, info) -> str | None

validate_judge_profile_key (method)

StructuredModelJudgeAssertionSpec.validate_judge_profile_key(value: str, info) -> str

validate_judge_profile_revision (method)

StructuredModelJudgeAssertionSpec.validate_judge_profile_revision(value: str, info) -> str

validate_threshold (method)

StructuredModelJudgeAssertionSpec.validate_threshold(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

StructuredRubricCriterionV1

from cayu.evals import StructuredRubricCriterionV1

One stable, weighted dimension that Cayu—not the judge—aggregates.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
name	StrictStr	required
description	StrictStr	required
weight	StrictStr	required

validate_id (method)

StructuredRubricCriterionV1.validate_id(value: str, info) -> str

validate_name (method)

StructuredRubricCriterionV1.validate_name(value: str, info) -> str

validate_description (method)

StructuredRubricCriterionV1.validate_description(value: str, info) -> str

validate_weight (method)

StructuredRubricCriterionV1.validate_weight(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

StructuredRubricV1

from cayu.evals import StructuredRubricV1

Content-addressed rubric with an exact, deterministic weight partition.

Bases: _SchemaV1PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[1]	1
id	StrictStr	required
revision	StrictStr	required
criteria	tuple[StructuredRubricCriterionV1, ...]	Field(min_length=1, max_length=EVAL_CORPUS_MAX_JUDGE_CRITERIA)

validate_schema_version_type (method)

StructuredRubricV1.validate_schema_version_type(value: object) -> object

validate_id (method)

StructuredRubricV1.validate_id(value: str, info) -> str

validate_revision_shape (method)

StructuredRubricV1.validate_revision_shape(value: str, info) -> str

validate_criteria_are_ordered (method)

StructuredRubricV1.validate_criteria_are_ordered(value: object, info) -> object

validate_contract (method)

StructuredRubricV1.validate_contract() -> StructuredRubricV1

create (method)

StructuredRubricV1.create(*, id: str, criteria: Sequence[StructuredRubricCriterionV1]) -> StructuredRubricV1

Release source · Runtime guarantees · Capability guides

ToolArgumentsContainAssertionSpec

from cayu.evals import ToolArgumentsContainAssertionSpec

Require one started invocation’s public arguments to contain bounded JSON.

Bases: _ToolJsonSubsetAssertionSpec.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
tool_name	StrictStr	required
occurrence	StrictInt	Field(default=1, ge=1, le=EVAL_TOOL_CALL_EVIDENCE_MAX_CALLS)
expected_subset	dict[str, Any]	required
kind	Literal['tool_arguments_contain']	'tool_arguments_contain'

validate_id (method)

ToolArgumentsContainAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

ToolArgumentsContainAssertionSpec.validate_description(value: str | None, info) -> str | None

validate_tool_name (method)

ToolArgumentsContainAssertionSpec.validate_tool_name(value: str, info) -> str

validate_expected_subset (method)

ToolArgumentsContainAssertionSpec.validate_expected_subset(value: object, info) -> dict[str, Any]

Release source · Runtime guarantees · Capability guides

ToolCalledAssertionSpec

from cayu.evals import ToolCalledAssertionSpec

Bases: _AssertionSpecBase.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
kind	Literal['tool_called']	'tool_called'
tool_name	StrictStr	required
min_count	StrictInt	Field(default=1, ge=0, le=EVIDENCE_MAX_TOOL_CALLS)
max_count	StrictInt | None	Field(default=None, ge=0, le=EVIDENCE_MAX_TOOL_CALLS)

validate_id (method)

ToolCalledAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

ToolCalledAssertionSpec.validate_description(value: str | None, info) -> str | None

validate_tool_name (method)

ToolCalledAssertionSpec.validate_tool_name(value: str, info) -> str

validate_count_range (method)

ToolCalledAssertionSpec.validate_count_range() -> ToolCalledAssertionSpec

Release source · Runtime guarantees · Capability guides

ToolResultContainsAssertionSpec

from cayu.evals import ToolResultContainsAssertionSpec

Require one explicitly retained public-safe result to contain bounded JSON.

Bases: _ToolJsonSubsetAssertionSpec.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
tool_name	StrictStr	required
occurrence	StrictInt	Field(default=1, ge=1, le=EVAL_TOOL_CALL_EVIDENCE_MAX_CALLS)
expected_subset	dict[str, Any]	required
kind	Literal['tool_result_contains']	'tool_result_contains'

validate_id (method)

ToolResultContainsAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

ToolResultContainsAssertionSpec.validate_description(value: str | None, info) -> str | None

validate_tool_name (method)

ToolResultContainsAssertionSpec.validate_tool_name(value: str, info) -> str

validate_expected_subset (method)

ToolResultContainsAssertionSpec.validate_expected_subset(value: object, info) -> dict[str, Any]

validate_result_root (method)

ToolResultContainsAssertionSpec.validate_result_root() -> ToolResultContainsAssertionSpec

Release source · Runtime guarantees · Capability guides

ToolsCalledInOrderAssertionSpec

from cayu.evals import ToolsCalledInOrderAssertionSpec

Bases: _AssertionSpecBase.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
kind	Literal['tools_called_in_order']	'tools_called_in_order'
tool_names	tuple[StrictStr, ...]	Field(max_length=EVAL_CORPUS_MAX_TOOL_NAMES)

validate_id (method)

ToolsCalledInOrderAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

ToolsCalledInOrderAssertionSpec.validate_description(value: str | None, info) -> str | None

validate_tool_names (method)

ToolsCalledInOrderAssertionSpec.validate_tool_names(value: object, info) -> tuple[str, ...]

Release source · Runtime guarantees · Capability guides

TrialRequestSpec

from cayu.evals import TrialRequestSpec

Bounded fresh-evaluation settings with an optional versioned policy.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
trials	StrictInt	Field(default=1, ge=1, le=EVAL_CORPUS_MAX_TRIALS)
timeout_seconds	StrictInt	Field(default=300, ge=1, le=EVAL_CORPUS_MAX_TIMEOUT_SECONDS)
trial_policy	EvalSuiteTrialPolicyV1 | None	Field(default=None, exclude_if=lambda value: value is None)

copy_trial_policy (method)

TrialRequestSpec.copy_trial_policy(value: object) -> object

validate_trial_policy (method)

TrialRequestSpec.validate_trial_policy() -> TrialRequestSpec

Release source · Runtime guarantees · Capability guides

UsageRecordedAssertionSpec

from cayu.evals import UsageRecordedAssertionSpec

Bases: _AssertionSpecBase.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
kind	Literal['usage_recorded']	'usage_recorded'
min_total_tokens	StrictInt	Field(default=1, ge=0, le=EVIDENCE_MAX_TOTAL_TOKENS)

validate_id (method)

UsageRecordedAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

UsageRecordedAssertionSpec.validate_description(value: str | None, info) -> str | None

Release source · Runtime guarantees · Capability guides

WorkspaceFileAssertionSpec

from cayu.evals import WorkspaceFileAssertionSpec

Require one declared workspace path to have bounded structural properties.

Bases: _AssertionSpecBase.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
id	StrictStr	required
description	StrictStr | None	None
kind	Literal['workspace_file']	'workspace_file'
path	StrictStr	required
present	StrictBool	True
minimum_bytes	StrictInt | None	Field(default=None, ge=0, le=EVIDENCE_MAX_TOTAL_TOKENS)
maximum_bytes	StrictInt | None	Field(default=None, ge=0, le=EVIDENCE_MAX_TOTAL_TOKENS)
sha256	StrictStr | None	None

validate_id (method)

WorkspaceFileAssertionSpec.validate_id(value: str, info) -> str

validate_description (method)

WorkspaceFileAssertionSpec.validate_description(value: str | None, info) -> str | None

validate_path (method)

WorkspaceFileAssertionSpec.validate_path(value: str, info) -> str

validate_sha256 (method)

WorkspaceFileAssertionSpec.validate_sha256(value: str | None, info) -> str | None

validate_contract (method)

WorkspaceFileAssertionSpec.validate_contract() -> WorkspaceFileAssertionSpec

Release source · Runtime guarantees · Capability guides