cayu.evals.published

Public Cayu API. Definitions are grouped by their public source module, and definitions from private implementation modules appear on their package’s page; adding a symbol does not move existing links. The import shown at each entry is a supported public namespace. Read the Python API overview and install the required extra.

`publish_eval_run`

from cayu.evals import publish_eval_run

Project one lossless internal suite run into the public corpus result graph.

publish_eval_run(corpus: EvalCorpusDocument, run: EvalRun) -> PublishedEvalRun

Release source · Runtime guarantees · Capability guides

`PUBLISHED_EVAL_MAX_BYTES`

from cayu.evals import PUBLISHED_EVAL_MAX_BYTES

PUBLISHED_EVAL_MAX_BYTES = 32 << 20

Release source · Runtime guarantees · Capability guides

`PUBLISHED_EVAL_SCHEMA_VERSION`

from cayu.evals import PUBLISHED_EVAL_SCHEMA_VERSION

PUBLISHED_EVAL_SCHEMA_VERSION = 10

Release source · Runtime guarantees · Capability guides

`PublishedArtifactDetail`

from cayu.evals import PublishedArtifactDetail

Bases: _PublishedAssertionDetail.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
`kind`	`Literal['artifact']`	`'artifact'`
`scope`	`Literal['session', 'environment']`	`required`
`filename`	`StrictStr \| None`	`None`
`content_type`	`StrictStr \| None`	`None`
`minimum_bytes`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVIDENCE_MAX_TOTAL_TOKENS)`
`maximum_bytes`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVIDENCE_MAX_TOTAL_TOKENS)`
`digest_required`	`StrictBool`	`required`
`text_required`	`StrictBool`	`required`
`observation_state`	`PublishedStructuralObservationState`	`required`
`min_count`	`StrictInt`	`Field(ge=0, le=EVAL_ARTIFACT_EVIDENCE_MAX_ARTIFACTS)`
`max_count`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVAL_ARTIFACT_EVIDENCE_MAX_ARTIFACTS)`
`matching_count`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVAL_ARTIFACT_EVIDENCE_MAX_ARTIFACTS)`

validate_filename (method)

PublishedArtifactDetail.validate_filename(value: str | None, info) -> str | None

validate_content_type (method)

PublishedArtifactDetail.validate_content_type(value: str | None, info) -> str | None

validate_contract (method)

PublishedArtifactDetail.validate_contract() -> PublishedArtifactDetail

Release source · Runtime guarantees · Capability guides

`PublishedAssertionDetail`

from cayu.evals import PublishedAssertionDetail

PublishedAssertionDetail: TypeAlias = Annotated[PublishedRootStatusDetail | PublishedChildStatusDetail | PublishedFinalOutputEqualsDetail | PublishedFinalOutputContainsDetail | PublishedToolCalledDetail | PublishedToolArgumentsContainDetail | PublishedToolResultContainsDetail | PublishedToolsCalledInOrderDetail | PublishedProcessEventDetail | PublishedProcessEventsInOrderDetail | PublishedWorkspaceFileDetail | PublishedArtifactDetail | PublishedMemoryAttributionDetail | PublishedMaxToolCallsDetail | PublishedMaxModelStepsDetail | PublishedUsageRecordedDetail | PublishedMaxTotalTokensDetail | PublishedMaxEstimatedCostDetail | PublishedModelJudgeDetail | PublishedStructuredModelJudgeDetail, Field(discriminator='kind')]

Release source · Runtime guarantees · Capability guides

`PublishedAssertionResult`

from cayu.evals import PublishedAssertionResult

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`assertion_id`	`StrictStr`	`required`
`assertion_revision`	`StrictStr`	`required`
`outcome`	`PublishedOutcome`	`required`
`score`	`StrictFloat \| None`	`Field(default=None, ge=0.0, le=1.0)`
`code`	`PublishedOutcome`	`required`
`message`	`StrictStr`	`required`
`detail`	`PublishedAssertionDetail`	`required`

validate_id (method)

PublishedAssertionResult.validate_id(value: str, info) -> str

validate_revision (method)

PublishedAssertionResult.validate_revision(value: str, info) -> str

validate_contract (method)

PublishedAssertionResult.validate_contract() -> PublishedAssertionResult

Release source · Runtime guarantees · Capability guides

`PublishedChildStatusDetail`

from cayu.evals import PublishedChildStatusDetail

Bases: _PublishedAssertionDetail.

Fields and attributes

Name	Type	Default
`kind`	`Literal['child_status']`	`'child_status'`
`expected`	`Literal['completed', 'failed', 'interrupted']`	`required`
`min_count`	`StrictInt`	`Field(ge=0, le=EVIDENCE_MAX_CHILD_SESSIONS)`
`max_count`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVIDENCE_MAX_CHILD_SESSIONS)`
`matching_count`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVIDENCE_MAX_CHILD_SESSIONS)`

validate_count_range (method)

PublishedChildStatusDetail.validate_count_range() -> PublishedChildStatusDetail

Release source · Runtime guarantees · Capability guides

`PublishedEvalCaseResult`

from cayu.evals import PublishedEvalCaseResult

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`case_id`	`StrictStr`	`required`
`case_revision`	`StrictStr`	`required`
`status`	`PublishedStatus`	`required`
`score`	`StrictFloat \| None`	`Field(default=None, ge=0.0, le=1.0)`
`reliability`	`EvalCaseReliabilityV1`	`required`
`trials`	`tuple[PublishedEvalTrialResult, ...]`	`Field(min_length=1, max_length=EVAL_CORPUS_MAX_TRIALS)`
`duration_ms`	`StrictInt`	`Field(ge=0, le=PUBLISHED_EVAL_MAX_DURATION_MS)`

validate_trials_are_ordered (method)

PublishedEvalCaseResult.validate_trials_are_ordered(value: object, info) -> object

validate_id (method)

PublishedEvalCaseResult.validate_id(value: str, info) -> str

validate_revision (method)

PublishedEvalCaseResult.validate_revision(value: str, info) -> str

validate_contract (method)

PublishedEvalCaseResult.validate_contract() -> PublishedEvalCaseResult

Release source · Runtime guarantees · Capability guides

`PublishedEvalRun`

from cayu.evals import PublishedEvalRun

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`schema_version`	`Literal[10]`	`required`
`revision`	`StrictStr`	`required`
`corpus_revision`	`StrictStr`	`required`
`target_key`	`StrictStr`	`required`
`suite_id`	`StrictStr`	`required`
`suite_revision`	`StrictStr`	`required`
`evidence_policy_revision`	`StrictStr`	`required`
`pricing_profile_fingerprint`	`StrictStr \| None`	`None`
`trial_policy`	`EvalSuiteTrialPolicyV1`	`required`
`accepted_exposure`	`EvalSuiteRunExposureV1 \| None`	`Field(default=None, exclude_if=lambda value: value is None)`
`status`	`PublishedStatus`	`required`
`score`	`StrictFloat \| None`	`Field(default=None, ge=0.0, le=1.0)`
`cases`	`tuple[PublishedEvalCaseResult, ...]`	`Field(min_length=1, max_length=EVAL_CORPUS_MAX_CASES)`
`duration_ms`	`StrictInt`	`Field(ge=0, le=PUBLISHED_EVAL_MAX_DURATION_MS)`

validate_cases_are_ordered (method)

PublishedEvalRun.validate_cases_are_ordered(value: object, info) -> object

validate_expanded_result_limit (method)

PublishedEvalRun.validate_expanded_result_limit(value)

Reject oversized raw graphs before constructing their nested models.

validate_revision_fields (method)

PublishedEvalRun.validate_revision_fields(value: str | None, info) -> str | None

validate_id (method)

PublishedEvalRun.validate_id(value: str, info) -> str

validate_contract (method)

PublishedEvalRun.validate_contract() -> PublishedEvalRun

Release source · Runtime guarantees · Capability guides

`PublishedEvalTrialResult`

from cayu.evals import PublishedEvalTrialResult

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`usage_evidence_state`	`Literal['complete', 'partial', 'unavailable'] \| None`	`Field(default=None, exclude_if=lambda value: value is None)`
`operation_outcomes`	`OperationOutcomeSummary \| None`	`Field(default=None, exclude_if=lambda value: value is None)`
`execution_status`	`Literal['completed', 'failed'] \| None`	`Field(default=None, exclude_if=lambda value: value is None)`
`execution_failure_category`	`Literal['provider_failure', 'environment_failure', 'execution_failure'] \| None`	`Field(default=None, exclude_if=lambda value: value is None)`
`failure_evidence`	`FailureEvidence \| None`	`Field(default=None, exclude_if=lambda value: value is None)`
`failure_capture`	`WorkflowFailureCapture \| None`	`Field(default=None, exclude_if=lambda value: value is None)`
`capture_bounds`	`SessionTrajectoryBounds \| None`	`Field(default=None, exclude_if=lambda value: value is None)`
`capture_diagnostic`	`WorkflowCaptureDiagnostic \| None`	`Field(default=None, exclude_if=lambda value: value is None)`
`trial_number`	`StrictInt`	`Field(ge=1, le=EVAL_CORPUS_MAX_TRIALS)`
`source_trial_revision`	`StrictStr`	`Field(min_length=64, max_length=64)`
`status`	`PublishedStatus`	`required`
`score`	`StrictFloat \| None`	`Field(default=None, ge=0.0, le=1.0)`
`assertions`	`tuple[PublishedAssertionResult, ...]`	`Field(min_length=1, max_length=EVAL_CORPUS_MAX_ASSERTIONS_PER_CASE)`
`evidence_complete`	`StrictBool`	`required`
`duration_ms`	`StrictInt`	`Field(ge=0, le=PUBLISHED_EVAL_MAX_DURATION_MS)`
`usage`	`PublishedUsageSummaryV1 \| None`	`None`
`output`	`EvalTrialOutputPreviewV1`	`required`
`memory_attribution`	`EvalMemoryAttributionEvidenceV1`	`required`
`code`	`EvalTrialDiagnosticCode`	`required`
`message`	`StrictStr`	`required`

validate_assertions_are_ordered (method)

PublishedEvalTrialResult.validate_assertions_are_ordered(value: object, info) -> object

validate_source_trial_revision (method)

PublishedEvalTrialResult.validate_source_trial_revision(value: str) -> str

copy_output (method)

PublishedEvalTrialResult.copy_output(value: object) -> object

copy_memory_attribution (method)

PublishedEvalTrialResult.copy_memory_attribution(value: object) -> object

validate_contract (method)

PublishedEvalTrialResult.validate_contract() -> PublishedEvalTrialResult

Release source · Runtime guarantees · Capability guides

`PublishedFinalOutputContainsDetail`

from cayu.evals import PublishedFinalOutputContainsDetail

Bases: _PublishedAssertionDetail.

Fields and attributes

Name	Type	Default
`kind`	`Literal['final_output_contains']`	`'final_output_contains'`
`matched`	`StrictBool \| None`	`None`

Release source · Runtime guarantees · Capability guides

`PublishedFinalOutputEqualsDetail`

from cayu.evals import PublishedFinalOutputEqualsDetail

Bases: _PublishedAssertionDetail.

Fields and attributes

Name	Type	Default
`kind`	`Literal['final_output_equals']`	`'final_output_equals'`
`comparison`	`Literal['text', 'json']`	`Field(default='text', exclude_if=lambda value: value == 'text')`
`matched`	`StrictBool \| None`	`None`

Release source · Runtime guarantees · Capability guides

`PublishedJudgeReferenceIdentityV1`

from cayu.evals import PublishedJudgeReferenceIdentityV1

Safe reference identity; private evaluator truth is never represented.

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`kind`	`Literal['public_reference', 'private_reference']`	`required`
`key`	`StrictStr`	`required`
`revision`	`StrictStr`	`required`
`availability`	`Literal['available']`	`'available'`
`privacy_policy_key`	`StrictStr \| None`	`None`
`privacy_policy_revision`	`StrictStr \| None`	`None`

validate_ids (method)

PublishedJudgeReferenceIdentityV1.validate_ids(value: str | None, info) -> str | None

validate_revisions (method)

PublishedJudgeReferenceIdentityV1.validate_revisions(value: str | None, info) -> str | None

validate_privacy_identity (method)

PublishedJudgeReferenceIdentityV1.validate_privacy_identity() -> PublishedJudgeReferenceIdentityV1

Release source · Runtime guarantees · Capability guides

`PublishedMaxEstimatedCostDetail`

from cayu.evals import PublishedMaxEstimatedCostDetail

Bases: _PublishedAssertionDetail.

Fields and attributes

Name	Type	Default
`kind`	`Literal['max_estimated_cost']`	`'max_estimated_cost'`
`maximum`	`StrictStr`	`required`
`currency`	`StrictStr`	`required`
`estimated_cost`	`StrictStr \| None`	`None`
`priced_model_steps`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVIDENCE_MAX_MODEL_STEPS)`
`unpriced_model_steps`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVIDENCE_MAX_MODEL_STEPS)`

validate_decimal_text (method)

PublishedMaxEstimatedCostDetail.validate_decimal_text(value: str | None, info) -> str | None

validate_currency (method)

PublishedMaxEstimatedCostDetail.validate_currency(value: str, info) -> str

validate_cost_counts (method)

PublishedMaxEstimatedCostDetail.validate_cost_counts() -> PublishedMaxEstimatedCostDetail

Release source · Runtime guarantees · Capability guides

`PublishedMaxModelStepsDetail`

from cayu.evals import PublishedMaxModelStepsDetail

Bases: _PublishedAssertionDetail.

Fields and attributes

Name	Type	Default
`kind`	`Literal['max_model_steps']`	`'max_model_steps'`
`maximum`	`StrictInt`	`Field(ge=0, le=EVIDENCE_MAX_MODEL_STEPS)`
`actual`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVIDENCE_MAX_MODEL_STEPS)`

Release source · Runtime guarantees · Capability guides

`PublishedMaxToolCallsDetail`

from cayu.evals import PublishedMaxToolCallsDetail

Bases: _PublishedAssertionDetail.

Fields and attributes

Name	Type	Default
`kind`	`Literal['max_tool_calls']`	`'max_tool_calls'`
`maximum`	`StrictInt`	`Field(ge=0, le=EVIDENCE_MAX_TOOL_CALLS)`
`actual`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVIDENCE_MAX_TOOL_CALLS)`

Release source · Runtime guarantees · Capability guides

`PublishedMaxTotalTokensDetail`

from cayu.evals import PublishedMaxTotalTokensDetail

Bases: _PublishedAssertionDetail.

Fields and attributes

Name	Type	Default
`kind`	`Literal['max_total_tokens']`	`'max_total_tokens'`
`maximum`	`StrictInt`	`Field(ge=0, le=EVIDENCE_MAX_TOTAL_TOKENS)`
`actual`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVIDENCE_MAX_TOTAL_TOKENS)`

Release source · Runtime guarantees · Capability guides

`PublishedMemoryAttributionDetail`

from cayu.evals import PublishedMemoryAttributionDetail

Bounded structural memory expectation and its conclusive counts, when available.

Bases: _PublishedAssertionDetail.

Fields and attributes

Name	Type	Default
`kind`	`Literal['memory_attribution']`	`'memory_attribution'`
`min_admitted_items`	`StrictInt`	`Field(ge=0, le=EVAL_MEMORY_ATTRIBUTION_MAX_ADMITTED_ITEMS)`
`max_admitted_items`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVAL_MEMORY_ATTRIBUTION_MAX_ADMITTED_ITEMS)`
`min_provider_exposures`	`StrictInt`	`Field(ge=0, le=EVAL_MEMORY_ATTRIBUTION_MAX_PROVIDER_EXPOSURES)`
`max_provider_exposures`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVAL_MEMORY_ATTRIBUTION_MAX_PROVIDER_EXPOSURES)`
`observation_state`	`MemoryAttributionObservationState`	`required`
`evidence_revision`	`StrictStr`	`required`
`limitations`	`tuple[EvalMemoryEvidenceLimitation, ...]`	`()`
`admitted_item_count`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVAL_MEMORY_ATTRIBUTION_MAX_ADMITTED_ITEMS)`
`provider_exposure_count`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVAL_MEMORY_ATTRIBUTION_MAX_PROVIDER_EXPOSURES)`

validate_evidence_revision (method)

PublishedMemoryAttributionDetail.validate_evidence_revision(value: str, info) -> str

validate_limitations_are_ordered (method)

PublishedMemoryAttributionDetail.validate_limitations_are_ordered(value: object, info) -> object

validate_contract (method)

PublishedMemoryAttributionDetail.validate_contract() -> PublishedMemoryAttributionDetail

Release source · Runtime guarantees · Capability guides

`PublishedModelJudgeCostV1`

from cayu.evals import PublishedModelJudgeCostV1

Observed priced cost, or an explicit unpriced state, for one judgment.

Bases: _PublishedJudgeCostV1.

Fields and attributes

Name	Type	Default
`availability`	`Literal['priced', 'unavailable']`	`required`
`currency`	`StrictStr \| None`	`None`
`estimated_cost`	`StrictStr \| None`	`None`
`priced_model_steps`	`StrictInt \| None`	`Field(default=None, ge=0)`
`unpriced_model_steps`	`StrictInt \| None`	`Field(default=None, ge=0)`

validate_currency (method)

PublishedModelJudgeCostV1.validate_currency(value: str | None, info) -> str | None

validate_estimated_cost (method)

PublishedModelJudgeCostV1.validate_estimated_cost(value: str | None, info) -> str | None

validate_availability (method)

PublishedModelJudgeCostV1.validate_availability() -> _PublishedJudgeCostV1

Release source · Runtime guarantees · Capability guides

`PublishedModelJudgeDetail`

from cayu.evals import PublishedModelJudgeDetail

Bounded public contract and safe outcome evidence for one model judgment.

Bases: _PublishedAssertionDetail.

Fields and attributes

Name	Type	Default
`kind`	`Literal['model_judge']`	`'model_judge'`
`evaluator_key`	`StrictStr`	`required`
`evaluator_implementation_revision`	`StrictStr`	`required`
`rubric`	`StrictStr`	`required`
`rubric_version`	`StrictStr`	`required`
`threshold`	`StrictFloat`	`Field(ge=0.0, le=1.0, allow_inf_nan=False)`
`include_transcript`	`StrictBool`	`required`
`diagnostic`	`PublishedModelJudgeDiagnostic`	`required`
`judge_profile`	`JudgeProfileIdentityV1`	`required`
`candidate_route_relation`	`Literal['independent_model', 'same_model', 'unknown']`	`required`
`usage`	`PublishedModelJudgeUsageV1 \| None`	`None`
`cost`	`PublishedModelJudgeCostV1 \| None`	`None`

validate_evaluator_key (method)

PublishedModelJudgeDetail.validate_evaluator_key(value: str, info) -> str

validate_evaluator_implementation_revision (method)

PublishedModelJudgeDetail.validate_evaluator_implementation_revision(value: str, info) -> str

validate_rubric (method)

PublishedModelJudgeDetail.validate_rubric(value: str, info) -> str

validate_rubric_version (method)

PublishedModelJudgeDetail.validate_rubric_version(value: str, info) -> str

validate_judgment (method)

PublishedModelJudgeDetail.validate_judgment() -> PublishedModelJudgeDetail

Release source · Runtime guarantees · Capability guides

`PublishedModelJudgeUsageV1`

from cayu.evals import PublishedModelJudgeUsageV1

Observed usage for one successfully recorded rubric-string judgment.

Bases: _PublishedJudgeUsageV1.

Fields and attributes

Name	Type	Default
`model_steps`	`StrictInt`	`Field(ge=1, le=EVIDENCE_MAX_MODEL_STEPS)`
`input_tokens`	`AggregateCount`	`Field(ge=0)`
`output_tokens`	`AggregateCount`	`Field(ge=0)`
`total_tokens`	`AggregateCount`	`Field(ge=0)`

validate_token_totals (method)

PublishedModelJudgeUsageV1.validate_token_totals() -> _PublishedJudgeUsageV1

Release source · Runtime guarantees · Capability guides

`PublishedProcessEventDetail`

from cayu.evals import PublishedProcessEventDetail

Bases: _PublishedAssertionDetail.

Fields and attributes

Name	Type	Default
`kind`	`Literal['process_event']`	`'process_event'`
`event`	`EvalProcessEventKind`	`required`
`min_count`	`StrictInt`	`Field(ge=0, le=EVAL_PROCESS_EVENT_EVIDENCE_MAX_EVENTS)`
`max_count`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVAL_PROCESS_EVENT_EVIDENCE_MAX_EVENTS)`
`matching_count`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVAL_PROCESS_EVENT_EVIDENCE_MAX_EVENTS)`

validate_count_range (method)

PublishedProcessEventDetail.validate_count_range() -> PublishedProcessEventDetail

Release source · Runtime guarantees · Capability guides

`PublishedProcessEventsInOrderDetail`

from cayu.evals import PublishedProcessEventsInOrderDetail

Bases: _PublishedAssertionDetail.

Fields and attributes

Name	Type	Default
`kind`	`Literal['process_events_in_order']`	`'process_events_in_order'`
`expected`	`tuple[EvalProcessEventKind, ...]`	`Field(min_length=1, max_length=EVAL_CORPUS_MAX_PROCESS_EVENTS)`
`actual_count`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVAL_PROCESS_EVENT_EVIDENCE_MAX_EVENTS)`
`matched`	`StrictBool \| None`	`None`

validate_expected_is_ordered (method)

PublishedProcessEventsInOrderDetail.validate_expected_is_ordered(value: object, info) -> object

validate_observation (method)

PublishedProcessEventsInOrderDetail.validate_observation() -> PublishedProcessEventsInOrderDetail

Release source · Runtime guarantees · Capability guides

`PublishedRootStatusDetail`

from cayu.evals import PublishedRootStatusDetail

Bases: _PublishedAssertionDetail.

Fields and attributes

Name	Type	Default
`kind`	`Literal['root_status']`	`'root_status'`
`expected`	`Literal['completed', 'failed']`	`required`
`actual`	`Literal['completed', 'failed', 'interrupted'] \| None`	`None`

Release source · Runtime guarantees · Capability guides

`PublishedStructuredJudgeCostV1`

from cayu.evals import PublishedStructuredJudgeCostV1

Observed priced cost, or an explicit unpriced state, for one judgment.

Bases: _PublishedJudgeCostV1.

Fields and attributes

Name	Type	Default
`availability`	`Literal['priced', 'unavailable']`	`required`
`currency`	`StrictStr \| None`	`None`
`estimated_cost`	`StrictStr \| None`	`None`
`priced_model_steps`	`StrictInt \| None`	`Field(default=None, ge=0)`
`unpriced_model_steps`	`StrictInt \| None`	`Field(default=None, ge=0)`

validate_currency (method)

PublishedStructuredJudgeCostV1.validate_currency(value: str | None, info) -> str | None

validate_estimated_cost (method)

PublishedStructuredJudgeCostV1.validate_estimated_cost(value: str | None, info) -> str | None

validate_availability (method)

PublishedStructuredJudgeCostV1.validate_availability() -> _PublishedJudgeCostV1

Release source · Runtime guarantees · Capability guides

`PublishedStructuredJudgeCriterionV1`

from cayu.evals import PublishedStructuredJudgeCriterionV1

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`criterion_id`	`StrictStr`	`required`
`weight`	`StrictStr`	`required`
`score`	`StrictStr`	`required`
`explanation`	`StrictStr \| None`	`required`
`explanation_state`	`Literal['available', 'redacted', 'unavailable']`	`required`

validate_id (method)

PublishedStructuredJudgeCriterionV1.validate_id(value: str, info) -> str

validate_unit_decimal (method)

PublishedStructuredJudgeCriterionV1.validate_unit_decimal(value: str, info) -> str

validate_explanation (method)

PublishedStructuredJudgeCriterionV1.validate_explanation(value: str | None, info) -> str | None

validate_explanation_state (method)

PublishedStructuredJudgeCriterionV1.validate_explanation_state() -> PublishedStructuredJudgeCriterionV1

Release source · Runtime guarantees · Capability guides

`PublishedStructuredJudgeUsageV1`

from cayu.evals import PublishedStructuredJudgeUsageV1

Observed usage for one successfully recorded structured judgment.

Bases: _PublishedJudgeUsageV1.

Fields and attributes

Name	Type	Default
`model_steps`	`StrictInt`	`Field(ge=1, le=EVIDENCE_MAX_MODEL_STEPS)`
`input_tokens`	`AggregateCount`	`Field(ge=0)`
`output_tokens`	`AggregateCount`	`Field(ge=0)`
`total_tokens`	`AggregateCount`	`Field(ge=0)`

validate_token_totals (method)

PublishedStructuredJudgeUsageV1.validate_token_totals() -> _PublishedJudgeUsageV1

Release source · Runtime guarantees · Capability guides

`PublishedStructuredModelJudgeDetail`

from cayu.evals import PublishedStructuredModelJudgeDetail

Typed public judgment with no prompt, credentials, options, or private truth.

Bases: _PublishedAssertionDetail.

Fields and attributes

Name	Type	Default
`kind`	`Literal['structured_model_judge']`	`'structured_model_judge'`
`judge_profile`	`JudgeProfileIdentityV1`	`required`
`candidate_route_relation`	`Literal['independent_model', 'same_model', 'unknown']`	`required`
`rubric_id`	`StrictStr`	`required`
`rubric_revision`	`StrictStr`	`required`
`reference`	`PublishedJudgeReferenceIdentityV1 \| None`	`None`
`threshold`	`StrictStr`	`required`
`evidence`	`EvalJudgeEvidenceSelectionV1`	`required`
`diagnostic`	`PublishedModelJudgeDiagnostic`	`required`
`criteria`	`tuple[PublishedStructuredJudgeCriterionV1, ...]`	`Field(default=(), max_length=EVAL_CORPUS_MAX_JUDGE_CRITERIA)`
`aggregate_score`	`StrictStr \| None`	`None`
`usage`	`PublishedStructuredJudgeUsageV1 \| None`	`None`
`cost`	`PublishedStructuredJudgeCostV1 \| None`	`None`

validate_rubric_id (method)

PublishedStructuredModelJudgeDetail.validate_rubric_id(value: str, info) -> str

validate_rubric_revision (method)

PublishedStructuredModelJudgeDetail.validate_rubric_revision(value: str, info) -> str

validate_threshold (method)

PublishedStructuredModelJudgeDetail.validate_threshold(value: str, info) -> str

validate_aggregate_score (method)

PublishedStructuredModelJudgeDetail.validate_aggregate_score(value: str | None, info) -> str | None

validate_criteria_are_ordered (method)

PublishedStructuredModelJudgeDetail.validate_criteria_are_ordered(value: object, info) -> object

validate_judgment (method)

PublishedStructuredModelJudgeDetail.validate_judgment() -> PublishedStructuredModelJudgeDetail

Release source · Runtime guarantees · Capability guides

`PublishedToolArgumentsContainDetail`

from cayu.evals import PublishedToolArgumentsContainDetail

Bases: _PublishedToolJsonSubsetDetail.

Fields and attributes

Name	Type	Default
`kind`	`Literal['tool_arguments_contain']`	`'tool_arguments_contain'`
`tool_name`	`StrictStr`	`required`
`occurrence`	`StrictInt`	`Field(ge=1, le=EVAL_TOOL_CALL_EVIDENCE_MAX_CALLS)`
`expected_subset`	`dict[str, Any]`	`required`
`observation_state`	`ToolJsonObservationState`	`required`
`invocation_index`	`StrictInt \| None`	`Field(default=None, ge=1, le=EVAL_TOOL_CALL_EVIDENCE_MAX_CALLS)`
`invocation_revision`	`StrictStr \| None`	`None`
`actual`	`dict[str, Any] \| None`	`None`
`matched`	`StrictBool \| None`	`None`

validate_tool_name (method)

PublishedToolArgumentsContainDetail.validate_tool_name(value: str, info) -> str

validate_json_object (method)

PublishedToolArgumentsContainDetail.validate_json_object(value: object, info) -> dict[str, Any] | None

validate_invocation_revision (method)

PublishedToolArgumentsContainDetail.validate_invocation_revision(value: str | None, info) -> str | None

validate_observation (method)

PublishedToolArgumentsContainDetail.validate_observation() -> _PublishedToolJsonSubsetDetail

Release source · Runtime guarantees · Capability guides

`PublishedToolCalledDetail`

from cayu.evals import PublishedToolCalledDetail

Bases: _PublishedAssertionDetail.

Fields and attributes

Name	Type	Default
`kind`	`Literal['tool_called']`	`'tool_called'`
`tool_name`	`StrictStr`	`required`
`min_count`	`StrictInt`	`Field(ge=0, le=EVIDENCE_MAX_TOOL_CALLS)`
`max_count`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVIDENCE_MAX_TOOL_CALLS)`
`matching_count`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVIDENCE_MAX_TOOL_CALLS)`

validate_tool_name (method)

PublishedToolCalledDetail.validate_tool_name(value: str, info) -> str

validate_count_range (method)

PublishedToolCalledDetail.validate_count_range() -> PublishedToolCalledDetail

Release source · Runtime guarantees · Capability guides

`PublishedToolResultContainsDetail`

from cayu.evals import PublishedToolResultContainsDetail

Bases: _PublishedToolJsonSubsetDetail.

Fields and attributes

Name	Type	Default
`kind`	`Literal['tool_result_contains']`	`'tool_result_contains'`
`tool_name`	`StrictStr`	`required`
`occurrence`	`StrictInt`	`Field(ge=1, le=EVAL_TOOL_CALL_EVIDENCE_MAX_CALLS)`
`expected_subset`	`dict[str, Any]`	`required`
`observation_state`	`ToolJsonObservationState`	`required`
`invocation_index`	`StrictInt \| None`	`Field(default=None, ge=1, le=EVAL_TOOL_CALL_EVIDENCE_MAX_CALLS)`
`invocation_revision`	`StrictStr \| None`	`None`
`actual`	`dict[str, Any] \| None`	`None`
`matched`	`StrictBool \| None`	`None`

validate_tool_name (method)

PublishedToolResultContainsDetail.validate_tool_name(value: str, info) -> str

validate_json_object (method)

PublishedToolResultContainsDetail.validate_json_object(value: object, info) -> dict[str, Any] | None

validate_invocation_revision (method)

PublishedToolResultContainsDetail.validate_invocation_revision(value: str | None, info) -> str | None

validate_observation (method)

PublishedToolResultContainsDetail.validate_observation() -> _PublishedToolJsonSubsetDetail

Release source · Runtime guarantees · Capability guides

`PublishedToolsCalledInOrderDetail`

from cayu.evals import PublishedToolsCalledInOrderDetail

Bases: _PublishedAssertionDetail.

Fields and attributes

Name	Type	Default
`kind`	`Literal['tools_called_in_order']`	`'tools_called_in_order'`
`expected_count`	`StrictInt`	`Field(ge=0, le=EVAL_CORPUS_MAX_TOOL_NAMES)`
`actual_count`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVIDENCE_MAX_TOOL_CALLS)`
`matched`	`StrictBool \| None`	`None`

validate_observation (method)

PublishedToolsCalledInOrderDetail.validate_observation() -> PublishedToolsCalledInOrderDetail

Release source · Runtime guarantees · Capability guides

`PublishedUsageRecordedDetail`

from cayu.evals import PublishedUsageRecordedDetail

Bases: _PublishedAssertionDetail.

Fields and attributes

Name	Type	Default
`kind`	`Literal['usage_recorded']`	`'usage_recorded'`
`minimum`	`StrictInt`	`Field(ge=0, le=EVIDENCE_MAX_TOTAL_TOKENS)`
`actual`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVIDENCE_MAX_TOTAL_TOKENS)`

Release source · Runtime guarantees · Capability guides

`PublishedUsageSummaryV1`

from cayu.evals import PublishedUsageSummaryV1

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`model_steps`	`StrictInt`	`Field(ge=0, le=MAX_DURABLE_JSON_INTEGER)`
`tool_calls`	`StrictInt`	`Field(ge=0, le=MAX_DURABLE_JSON_INTEGER)`
`total_tokens`	`AggregateCount`	`Field(ge=0)`

Release source · Runtime guarantees · Capability guides

`PublishedWorkspaceFileDetail`

from cayu.evals import PublishedWorkspaceFileDetail

Bases: _PublishedAssertionDetail.

Fields and attributes

Name	Type	Default
`kind`	`Literal['workspace_file']`	`'workspace_file'`
`path`	`StrictStr`	`required`
`expected_present`	`StrictBool`	`required`
`minimum_bytes`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVIDENCE_MAX_TOTAL_TOKENS)`
`maximum_bytes`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVIDENCE_MAX_TOTAL_TOKENS)`
`digest_required`	`StrictBool`	`required`
`observation_state`	`PublishedStructuralObservationState`	`required`
`actual_present`	`StrictBool \| None`	`None`
`actual_size_bytes`	`StrictInt \| None`	`Field(default=None, ge=0, le=EVIDENCE_MAX_TOTAL_TOKENS)`
`digest_matched`	`StrictBool \| None`	`None`

validate_path (method)

PublishedWorkspaceFileDetail.validate_path(value: str, info) -> str

validate_observation (method)

PublishedWorkspaceFileDetail.validate_observation() -> PublishedWorkspaceFileDetail

Release source · Runtime guarantees · Capability guides