cayu.evals.execution_comparison

Public Cayu API. Definitions are grouped by their public source module, and definitions from private implementation modules appear on their package’s page; adding a symbol does not move existing links. The import shown at each entry is a supported public namespace. Read the Python API overview and install the required extra.

compare_corpus_execution_results

from cayu.evals import compare_corpus_execution_results

Compare published fresh results only when their complete contracts match.

compare_corpus_execution_results(baseline: CorpusExecutionResult, current: CorpusExecutionResult, *, score_tolerance: float=0.0) -> CorpusExecutionComparison

Release source · Runtime guarantees · Capability guides

compare_eval_results

from cayu.evals import compare_eval_results

Compare captured and fresh results through their shared public projection.

compare_eval_results(baseline: CorpusExecutionResult | CapturedEvaluationResultV1 | EvalResultProjectionV1 | EvalResultProjectionV2, current: CorpusExecutionResult | CapturedEvaluationResultV1 | EvalResultProjectionV1 | EvalResultProjectionV2, *, score_tolerance: float=0.0) -> CorpusExecutionComparison

Release source · Runtime guarantees · Capability guides

CORPUS_EXECUTION_COMPARISON_MAX_BYTES

from cayu.evals import CORPUS_EXECUTION_COMPARISON_MAX_BYTES

CORPUS_EXECUTION_COMPARISON_MAX_BYTES = 96 << 20

Release source · Runtime guarantees · Capability guides

corpus_execution_compatibility

from cayu.evals import corpus_execution_compatibility

Check evaluation-contract compatibility without comparing target releases.

corpus_execution_compatibility(baseline: CorpusExecutionResult, current: CorpusExecutionResult) -> CorpusComparisonCompatibility

Release source · Runtime guarantees · Capability guides

CorpusCaseComparison

from cayu.evals import CorpusCaseComparison

Aggregate outcomes for one case in two contract-compatible results.

Bases: BaseModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
case_id	StrictStr	Field(min_length=1, max_length=128)
baseline_status	PublishedStatus	required
current_status	PublishedStatus	required
baseline_score	StrictFloat | None	Field(default=None, ge=0.0, le=1.0)
current_score	StrictFloat | None	Field(default=None, ge=0.0, le=1.0)
baseline_trial_diagnostic_codes	tuple[EvalTrialDiagnosticCode, ...]	Field(default=(), max_length=EVAL_CORPUS_MAX_TRIALS, exclude_if=lambda value: not value)
current_trial_diagnostic_codes	tuple[EvalTrialDiagnosticCode, ...]	Field(default=(), max_length=EVAL_CORPUS_MAX_TRIALS, exclude_if=lambda value: not value)
baseline_reliability	CorpusReliabilityDistributionV1	required
current_reliability	CorpusReliabilityDistributionV1	required
reliability_change	CorpusReliabilityChange	required

validate_case_id (method)

CorpusCaseComparison.validate_case_id(value: str) -> str

validate_diagnostic_sequence (method)

CorpusCaseComparison.validate_diagnostic_sequence(value: object) -> object

copy_reliability (method)

CorpusCaseComparison.copy_reliability(value: object) -> object

validate_scores (method)

CorpusCaseComparison.validate_scores() -> CorpusCaseComparison

Release source · Runtime guarantees · Capability guides

CorpusComparisonCompatibility

from cayu.evals import CorpusComparisonCompatibility

Typed precondition result for later regression comparison and UI adapters.

Bases: BaseModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[1]	1
baseline_result_revision	StrictStr	required
current_result_revision	StrictStr	required
comparable	StrictBool	required
reasons	tuple[CorpusComparisonReason, ...]	()

validate_schema_version_type (method)

CorpusComparisonCompatibility.validate_schema_version_type(value: object) -> object

validate_revisions (method)

CorpusComparisonCompatibility.validate_revisions(value: str, info) -> str

validate_reason_sequence (method)

CorpusComparisonCompatibility.validate_reason_sequence(value: object) -> object

validate_contract (method)

CorpusComparisonCompatibility.validate_contract() -> CorpusComparisonCompatibility

Release source · Runtime guarantees · Capability guides

CorpusComparisonReason

from cayu.evals import CorpusComparisonReason

Stable reason that two published executions cannot be compared as one contract.

Bases: StrEnum.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
TARGET_KEY_MISMATCH	class attribute	'target_key_mismatch'
EXTERNAL_TARGET_REVISION_MISMATCH	class attribute	'external_target_revision_mismatch'
CORPUS_REVISION_MISMATCH	class attribute	'corpus_revision_mismatch'
SUITE_ID_MISMATCH	class attribute	'suite_id_mismatch'
SUITE_REVISION_MISMATCH	class attribute	'suite_revision_mismatch'
EVIDENCE_POLICY_REVISION_MISMATCH	class attribute	'evidence_policy_revision_mismatch'
PRICING_PROFILE_FINGERPRINT_MISMATCH	class attribute	'pricing_profile_fingerprint_mismatch'
TRIAL_POLICY_REVISION_MISMATCH	class attribute	'trial_policy_revision_mismatch'
ACCEPTED_EXPOSURE_CONTRACT_MISMATCH	class attribute	'accepted_exposure_contract_mismatch'
CASE_CONTRACT_MISMATCH	class attribute	'case_contract_mismatch'
ASSERTION_CONTRACT_MISMATCH	class attribute	'assertion_contract_mismatch'

Release source · Runtime guarantees · Capability guides

CorpusComparisonResultSummary

from cayu.evals import CorpusComparisonResultSummary

Bounded public identity and aggregate outcome for one compared result.

Bases: BaseModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
result_revision	StrictStr	required
target_key	StrictStr	required
application_release_id	StrictStr	Field(min_length=1, max_length=256)
app_manifest_fingerprint	StrictStr	Field(min_length=64, max_length=64)
external_target_revision	StrictStr | None	required
corpus_revision	StrictStr	required
suite_id	StrictStr	required
suite_revision	StrictStr	required
evidence_policy_revision	StrictStr	required
pricing_profile_fingerprint	StrictStr | None	required
trial_policy_revision	StrictStr	required
accepted_exposure_revision	StrictStr | None	required
accepted_exposure_comparison_revision	StrictStr | None	required
memory_attribution_support	Literal['unsupported']	'unsupported'
status	PublishedStatus	required
score	StrictFloat | None	Field(default=None, ge=0.0, le=1.0)

validate_revisions (method)

CorpusComparisonResultSummary.validate_revisions(value: str | None, info) -> str | None

validate_ids (method)

CorpusComparisonResultSummary.validate_ids(value: str, info) -> str

validate_release_id (method)

CorpusComparisonResultSummary.validate_release_id(value: str) -> str

validate_manifest_fingerprint (method)

CorpusComparisonResultSummary.validate_manifest_fingerprint(value: str) -> str

validate_exposure_revisions (method)

CorpusComparisonResultSummary.validate_exposure_revisions() -> CorpusComparisonResultSummary

validate_score (method)

CorpusComparisonResultSummary.validate_score() -> CorpusComparisonResultSummary

Release source · Runtime guarantees · Capability guides

CorpusExecutionComparison

from cayu.evals import CorpusExecutionComparison

Deterministic contract-aware comparison of two published corpus executions.

Bases: BaseModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[4]	4
compatibility	CorpusComparisonCompatibility	required
score_tolerance	StrictFloat	Field(default=0.0, ge=0.0, le=1.0)
baseline	CorpusComparisonResultSummary	required
current	CorpusComparisonResultSummary	required
cases	tuple[CorpusCaseComparison, ...]	Field(default_factory=tuple, max_length=1000)
regressions	tuple[CorpusExecutionRegression, ...]	Field(default_factory=tuple, max_length=3002)
structured_judge_comparison_state	EvalStructuredJudgeComparisonState	required
structured_judgments	tuple[EvalStructuredJudgeComparisonV1, ...]	Field(default=(), max_length=EVAL_STRUCTURED_JUDGE_COMPARISON_MAX_ITEMS)
structured_judge_observation_mismatches	tuple[EvalStructuredJudgeObservationMismatchV1, ...]	Field(default=(), max_length=EVAL_STRUCTURED_JUDGE_COMPARISON_MAX_ITEMS * 2)
tool_json_comparison_state	EvalToolJsonComparisonState	required
tool_json_assertions	tuple[EvalToolJsonAssertionComparisonV1, ...]	Field(default=(), max_length=EVAL_STRUCTURED_JUDGE_COMPARISON_MAX_ITEMS)
tool_json_observation_mismatches	tuple[EvalToolJsonObservationMismatchV1, ...]	Field(default=(), max_length=EVAL_STRUCTURED_JUDGE_COMPARISON_MAX_ITEMS * 2)

validate_schema_version_type (method)

CorpusExecutionComparison.validate_schema_version_type(value: object) -> object

validate_structured_judgments_order (method)

CorpusExecutionComparison.validate_structured_judgments_order(value: object, info) -> object

validate_derived_contract (method)

CorpusExecutionComparison.validate_derived_contract() -> CorpusExecutionComparison

Release source · Runtime guarantees · Capability guides

CorpusExecutionRegression

from cayu.evals import CorpusExecutionRegression

One typed regression derived from compatible published aggregates.

Bases: BaseModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
scope	CorpusRegressionScope	required
kind	CorpusRegressionKind	required
case_id	StrictStr | None	Field(default=None, min_length=1, max_length=128)
baseline_status	PublishedStatus | None	None
current_status	PublishedStatus | None	None
baseline_score	StrictFloat | None	Field(default=None, ge=0.0, le=1.0)
current_score	StrictFloat | None	Field(default=None, ge=0.0, le=1.0)
baseline_reliability	CorpusReliabilityDistributionV1 | None	None
current_reliability	CorpusReliabilityDistributionV1 | None	None

validate_case_id (method)

CorpusExecutionRegression.validate_case_id(value: str | None) -> str | None

copy_reliability (method)

CorpusExecutionRegression.copy_reliability(value: object) -> object

validate_shape (method)

CorpusExecutionRegression.validate_shape() -> CorpusExecutionRegression

Release source · Runtime guarantees · Capability guides

CorpusRegressionKind

from cayu.evals import CorpusRegressionKind

Stable dimension on which one compatible result regressed.

Bases: StrEnum.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
STATUS	class attribute	'status'
SCORE	class attribute	'score'
RELIABILITY	class attribute	'reliability'

Release source · Runtime guarantees · Capability guides

CorpusRegressionScope

from cayu.evals import CorpusRegressionScope

Stable location of one compatible-result regression.

Bases: StrEnum.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
RUN	class attribute	'run'
CASE	class attribute	'case'

Release source · Runtime guarantees · Capability guides

CorpusReliabilityDistributionV1

from cayu.evals import CorpusReliabilityDistributionV1

Lossless trial-outcome counts used for cross-release reliability decisions.

Bases: BaseModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[1]	1
total_trials	StrictInt	Field(ge=1, le=EVAL_CORPUS_MAX_TRIALS)
passed_trials	StrictInt	Field(ge=0, le=EVAL_CORPUS_MAX_TRIALS)
candidate_failed_trials	StrictInt	Field(ge=0, le=EVAL_CORPUS_MAX_TRIALS)
runtime_error_trials	StrictInt	Field(ge=0, le=EVAL_CORPUS_MAX_TRIALS)
evaluator_error_trials	StrictInt	Field(ge=0, le=EVAL_CORPUS_MAX_TRIALS)
unavailable_trials	StrictInt	Field(ge=0, le=EVAL_CORPUS_MAX_TRIALS)
cancelled_trials	StrictInt	Field(ge=0, le=EVAL_CORPUS_MAX_TRIALS)

validate_schema_version_type (method)

CorpusReliabilityDistributionV1.validate_schema_version_type(value: object) -> object

validate_distribution (method)

CorpusReliabilityDistributionV1.validate_distribution() -> CorpusReliabilityDistributionV1

from_diagnostic_codes (method)

CorpusReliabilityDistributionV1.from_diagnostic_codes(codes: tuple[EvalTrialDiagnosticCode, ...]) -> CorpusReliabilityDistributionV1

Release source · Runtime guarantees · Capability guides

eval_result_compatibility

from cayu.evals import eval_result_compatibility

Check compatibility across captured and fresh immutable result origins.

eval_result_compatibility(baseline: CorpusExecutionResult | CapturedEvaluationResultV1 | EvalResultProjectionV1 | EvalResultProjectionV2, current: CorpusExecutionResult | CapturedEvaluationResultV1 | EvalResultProjectionV1 | EvalResultProjectionV2) -> CorpusComparisonCompatibility

Release source · Runtime guarantees · Capability guides

EvalStructuredJudgeComparisonV1

from cayu.evals import EvalStructuredJudgeComparisonV1

Explainable comparison of one exact case/trial/assertion observation.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
case_id	StrictStr	required
trial_number	StrictInt | None	Field(default=None, ge=1, le=EVAL_CORPUS_MAX_TRIALS)
assertion_id	StrictStr	required
baseline_outcome	PublishedOutcome	required
current_outcome	PublishedOutcome	required
baseline	EvalStructuredJudgePresentationV1	required
current	EvalStructuredJudgePresentationV1	required
evaluator_change	EvalStructuredJudgeEvaluatorChange	required
aggregate_change	EvalStructuredJudgeAggregateChange	required
aggregate_delta	StrictStr | None	required
regressed	StrictBool	required
criteria	tuple[EvalStructuredJudgeCriterionComparisonV1, ...]	Field(default=(), max_length=EVAL_CORPUS_MAX_JUDGE_CRITERIA)

validate_ids (method)

EvalStructuredJudgeComparisonV1.validate_ids(value: str, info) -> str

copy_judgment (method)

EvalStructuredJudgeComparisonV1.copy_judgment(value: object) -> object

validate_criteria_order (method)

EvalStructuredJudgeComparisonV1.validate_criteria_order(value: object, info) -> object

validate_aggregate_delta (method)

EvalStructuredJudgeComparisonV1.validate_aggregate_delta(value: str | None, info) -> str | None

validate_comparison (method)

EvalStructuredJudgeComparisonV1.validate_comparison() -> EvalStructuredJudgeComparisonV1

Release source · Runtime guarantees · Capability guides

EvalStructuredJudgeCriterionComparisonV1

from cayu.evals import EvalStructuredJudgeCriterionComparisonV1

Exact criterion delta for one identity-matched structured judgment.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
criterion_id	StrictStr	required
weight	StrictStr	required
baseline_score	StrictStr	required
current_score	StrictStr	required
score_delta	StrictStr	required
baseline_explanation_state	Literal['available', 'redacted', 'unavailable']	required
current_explanation_state	Literal['available', 'redacted', 'unavailable']	required

validate_criterion_id (method)

EvalStructuredJudgeCriterionComparisonV1.validate_criterion_id(value: str, info) -> str

validate_unit_decimal (method)

EvalStructuredJudgeCriterionComparisonV1.validate_unit_decimal(value: str, info) -> str

validate_score_delta (method)

EvalStructuredJudgeCriterionComparisonV1.validate_score_delta(value: str, info) -> str

validate_delta (method)

EvalStructuredJudgeCriterionComparisonV1.validate_delta() -> EvalStructuredJudgeCriterionComparisonV1

Release source · Runtime guarantees · Capability guides

EvalStructuredJudgeObservationMismatchV1

from cayu.evals import EvalStructuredJudgeObservationMismatchV1

One structured observation present on only one side of a comparison.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
case_id	StrictStr	required
trial_number	StrictInt | None	Field(default=None, ge=1, le=EVAL_CORPUS_MAX_TRIALS)
assertion_id	StrictStr	required
availability	Literal['baseline_only', 'current_only']	required

validate_ids (method)

EvalStructuredJudgeObservationMismatchV1.validate_ids(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

EvalToolJsonAssertionComparisonV1

from cayu.evals import EvalToolJsonAssertionComparisonV1

Exact comparison of one safe retained tool-JSON observation.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
case_id	StrictStr	required
trial_number	StrictInt | None	Field(default=None, ge=1, le=EVAL_CORPUS_MAX_TRIALS)
assertion_id	StrictStr	required
baseline_outcome	PublishedOutcome	required
current_outcome	PublishedOutcome	required
baseline	PublishedToolArgumentsContainDetail | PublishedToolResultContainsDetail	required
current	PublishedToolArgumentsContainDetail | PublishedToolResultContainsDetail	required
evidence_state_changed	StrictBool	required
observed_value_change	EvalToolJsonObservedValueChange	required
outcome_changed	StrictBool	required
regressed	StrictBool	required

validate_ids (method)

EvalToolJsonAssertionComparisonV1.validate_ids(value: str, info) -> str

copy_detail (method)

EvalToolJsonAssertionComparisonV1.copy_detail(value: object) -> object

validate_comparison (method)

EvalToolJsonAssertionComparisonV1.validate_comparison() -> EvalToolJsonAssertionComparisonV1

Release source · Runtime guarantees · Capability guides

EvalToolJsonObservationMismatchV1

from cayu.evals import EvalToolJsonObservationMismatchV1

One safe tool-JSON observation present on only one comparison side.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
case_id	StrictStr	required
trial_number	StrictInt | None	Field(default=None, ge=1, le=EVAL_CORPUS_MAX_TRIALS)
assertion_id	StrictStr	required
availability	Literal['baseline_only', 'current_only']	required

validate_ids (method)

EvalToolJsonObservationMismatchV1.validate_ids(value: str, info) -> str

Release source · Runtime guarantees · Capability guides