cayu.evals.calibration

Public Cayu API. Definitions are grouped by their public source module, and definitions from private implementation modules appear on their package’s page; adding a symbol does not move existing links. The import shown at each entry is a supported public namespace. Read the Python API overview and install the required extra.

`compile_eval_judge_calibration_draft`

from cayu.evals import compile_eval_judge_calibration_draft

Canonicalize editor material into one immutable fixed-evidence contract.

compile_eval_judge_calibration_draft(draft: EvalJudgeCalibrationDraftV1) -> EvalJudgeCalibrationDefinitionV1

Release source · Runtime guarantees · Capability guides

`EVAL_JUDGE_CALIBRATION_MAX_BYTES`

from cayu.evals import EVAL_JUDGE_CALIBRATION_MAX_BYTES

EVAL_JUDGE_CALIBRATION_MAX_BYTES = 2 << 20

Release source · Runtime guarantees · Capability guides

`EVAL_JUDGE_CALIBRATION_MAX_TRIALS`

from cayu.evals import EVAL_JUDGE_CALIBRATION_MAX_TRIALS

EVAL_JUDGE_CALIBRATION_MAX_TRIALS = 10

Release source · Runtime guarantees · Capability guides

`eval_judge_calibration_report_from_json`

from cayu.evals import eval_judge_calibration_report_from_json

Load one bounded report without accepting an unknown wire shape.

eval_judge_calibration_report_from_json(source: str) -> EvalJudgeCalibrationReportV1

Release source · Runtime guarantees · Capability guides

`eval_judge_calibration_report_to_json`

from cayu.evals import eval_judge_calibration_report_to_json

Serialize one validated report into deterministic bounded storage JSON.

eval_judge_calibration_report_to_json(report: EvalJudgeCalibrationReportV1) -> str

Release source · Runtime guarantees · Capability guides

`EVAL_JUDGE_CALIBRATION_SCHEMA_VERSION`

from cayu.evals import EVAL_JUDGE_CALIBRATION_SCHEMA_VERSION

EVAL_JUDGE_CALIBRATION_SCHEMA_VERSION = 1

Release source · Runtime guarantees · Capability guides

`EvalJudgeCalibrationCriterionLabelV1`

from cayu.evals import EvalJudgeCalibrationCriterionLabelV1

One human score for an exact rubric criterion.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
`criterion_id`	`StrictStr`	`required`
`score`	`StrictStr`	`required`

validate_criterion_id (method)

EvalJudgeCalibrationCriterionLabelV1.validate_criterion_id(value: str, info) -> str

validate_score (method)

EvalJudgeCalibrationCriterionLabelV1.validate_score(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

`EvalJudgeCalibrationDefinitionV1`

from cayu.evals import EvalJudgeCalibrationDefinitionV1

Immutable calibration inputs; invoking it can only execute the judge.

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`schema_version`	`Literal[1]`	`EVAL_JUDGE_CALIBRATION_SCHEMA_VERSION`
`revision`	`StrictStr`	`required`
`id`	`StrictStr`	`required`
`target_key`	`StrictStr`	`required`
`assertion`	`StructuredModelJudgeAssertionSpec`	`required`
`evidence`	`EvalJudgeCalibrationEvidenceV1`	`required`
`human_label`	`EvalJudgeCalibrationHumanLabelV1`	`required`
`trials`	`StrictInt`	`Field(ge=1, le=EVAL_JUDGE_CALIBRATION_MAX_TRIALS)`

validate_revision_shape (method)

EvalJudgeCalibrationDefinitionV1.validate_revision_shape(value: str, info) -> str

validate_ids (method)

EvalJudgeCalibrationDefinitionV1.validate_ids(value: str, info) -> str

validate_contract (method)

EvalJudgeCalibrationDefinitionV1.validate_contract() -> EvalJudgeCalibrationDefinitionV1

Release source · Runtime guarantees · Capability guides

`EvalJudgeCalibrationDraftV1`

from cayu.evals import EvalJudgeCalibrationDraftV1

Revision-free browser material for one calibration run.

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`schema_version`	`Literal[1]`	`EVAL_JUDGE_CALIBRATION_SCHEMA_VERSION`
`id`	`StrictStr`	`required`
`target_key`	`StrictStr`	`required`
`assertion`	`StructuredModelJudgeAssertionSpec`	`required`
`evidence_source_id`	`StrictStr`	`required`
`task`	`StrictStr`	`required`
`final_output`	`StrictStr`	`required`
`transcript`	`StrictStr \| None`	`None`
`human_criteria`	`tuple[EvalJudgeCalibrationCriterionLabelV1, ...]`	`Field(min_length=1, max_length=EVAL_CORPUS_MAX_JUDGE_CRITERIA)`
`trials`	`StrictInt`	`Field(default=1, ge=1, le=EVAL_JUDGE_CALIBRATION_MAX_TRIALS)`

validate_ids (method)

EvalJudgeCalibrationDraftV1.validate_ids(value: str, info) -> str

validate_evidence_source_id (method)

EvalJudgeCalibrationDraftV1.validate_evidence_source_id(value: str, info) -> str

validate_task (method)

EvalJudgeCalibrationDraftV1.validate_task(value: str, info) -> str

validate_final_output (method)

EvalJudgeCalibrationDraftV1.validate_final_output(value: str, info) -> str

validate_transcript (method)

EvalJudgeCalibrationDraftV1.validate_transcript(value: str | None, info) -> str | None

validate_human_criteria_order (method)

EvalJudgeCalibrationDraftV1.validate_human_criteria_order(value: object, info) -> object

Release source · Runtime guarantees · Capability guides

`EvalJudgeCalibrationEvidenceProvenanceV1`

from cayu.evals import EvalJudgeCalibrationEvidenceProvenanceV1

Operator-declared origin for fixed evidence, without implied verification.

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`schema_version`	`Literal[1]`	`EVAL_JUDGE_CALIBRATION_SCHEMA_VERSION`
`kind`	`Literal['operator_supplied']`	`'operator_supplied'`
`source_id`	`StrictStr`	`required`

validate_source_id (method)

EvalJudgeCalibrationEvidenceProvenanceV1.validate_source_id(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

`EvalJudgeCalibrationEvidenceV1`

from cayu.evals import EvalJudgeCalibrationEvidenceV1

Content-addressed, public-safe candidate evidence that is never executed.

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`schema_version`	`Literal[1]`	`EVAL_JUDGE_CALIBRATION_SCHEMA_VERSION`
`revision`	`StrictStr`	`required`
`provenance`	`EvalJudgeCalibrationEvidenceProvenanceV1`	`required`
`task`	`StrictStr`	`required`
`final_output`	`StrictStr`	`required`
`transcript`	`StrictStr \| None`	`None`

validate_revision_shape (method)

EvalJudgeCalibrationEvidenceV1.validate_revision_shape(value: str, info) -> str

validate_task (method)

EvalJudgeCalibrationEvidenceV1.validate_task(value: str, info) -> str

validate_final_output (method)

EvalJudgeCalibrationEvidenceV1.validate_final_output(value: str, info) -> str

validate_transcript (method)

EvalJudgeCalibrationEvidenceV1.validate_transcript(value: str | None, info) -> str | None

validate_revision (method)

EvalJudgeCalibrationEvidenceV1.validate_revision() -> EvalJudgeCalibrationEvidenceV1

create (method)

EvalJudgeCalibrationEvidenceV1.create(*, source_id: str, task: str, final_output: str, transcript: str | None) -> EvalJudgeCalibrationEvidenceV1

Release source · Runtime guarantees · Capability guides

`EvalJudgeCalibrationHumanLabelV1`

from cayu.evals import EvalJudgeCalibrationHumanLabelV1

Human ground truth bound to one exact structured rubric.

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`schema_version`	`Literal[1]`	`EVAL_JUDGE_CALIBRATION_SCHEMA_VERSION`
`rubric_revision`	`StrictStr`	`required`
`criteria`	`tuple[EvalJudgeCalibrationCriterionLabelV1, ...]`	`Field(min_length=1, max_length=EVAL_CORPUS_MAX_JUDGE_CRITERIA)`
`aggregate_score`	`StrictStr`	`required`

validate_rubric_revision (method)

EvalJudgeCalibrationHumanLabelV1.validate_rubric_revision(value: str, info) -> str

validate_criteria_order (method)

EvalJudgeCalibrationHumanLabelV1.validate_criteria_order(value: object, info) -> object

validate_aggregate_score (method)

EvalJudgeCalibrationHumanLabelV1.validate_aggregate_score(value: str, info) -> str

Release source · Runtime guarantees · Capability guides

`EvalJudgeCalibrationReportV1`

from cayu.evals import EvalJudgeCalibrationReportV1

Immutable completed calibration with every fixed-evidence judge trial.

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`schema_version`	`Literal[1]`	`EVAL_JUDGE_CALIBRATION_SCHEMA_VERSION`
`revision`	`StrictStr`	`required`
`run_id`	`StrictStr`	`required`
`definition`	`EvalJudgeCalibrationDefinitionV1`	`required`
`judge_profile`	`JudgeProfileIdentityV1`	`required`
`candidate_route_relation`	`Literal['independent_model', 'same_model']`	`required`
`trials`	`tuple[EvalJudgeCalibrationTrialV1, ...]`	`Field(min_length=1, max_length=EVAL_JUDGE_CALIBRATION_MAX_TRIALS)`

validate_revision_shape (method)

EvalJudgeCalibrationReportV1.validate_revision_shape(value: str, info) -> str

validate_run_id (method)

EvalJudgeCalibrationReportV1.validate_run_id(value: str, info) -> str

validate_trial_order (method)

EvalJudgeCalibrationReportV1.validate_trial_order(value: object, info) -> object

validate_contract (method)

EvalJudgeCalibrationReportV1.validate_contract() -> EvalJudgeCalibrationReportV1

create (method)

EvalJudgeCalibrationReportV1.create(*, run_id: str, prepared: PreparedEvalJudgeCalibration, trials: tuple[EvalJudgeCalibrationTrialV1, ...]) -> EvalJudgeCalibrationReportV1

Release source · Runtime guarantees · Capability guides

`EvalJudgeCalibrationTrialV1`

from cayu.evals import EvalJudgeCalibrationTrialV1

One lossless judge observation compared with the fixed human label.

Bases: _PortableModel.

Fields and attributes

Name	Type	Default
`schema_version`	`Literal[1]`	`EVAL_JUDGE_CALIBRATION_SCHEMA_VERSION`
`revision`	`StrictStr`	`required`
`sequence`	`StrictInt`	`Field(ge=1, le=EVAL_JUDGE_CALIBRATION_MAX_TRIALS)`
`judgment`	`PublishedAssertionResult`	`required`
`aggregate_absolute_error`	`StrictStr \| None`	`None`
`pass_agreement`	`StrictBool \| None`	`None`

validate_revision_shape (method)

EvalJudgeCalibrationTrialV1.validate_revision_shape(value: str, info) -> str

validate_error (method)

EvalJudgeCalibrationTrialV1.validate_error(value: str | None, info) -> str | None

validate_contract (method)

EvalJudgeCalibrationTrialV1.validate_contract() -> EvalJudgeCalibrationTrialV1

create (method)

EvalJudgeCalibrationTrialV1.create(*, sequence: int, judgment: PublishedAssertionResult, human_aggregate_score: str, threshold: str) -> EvalJudgeCalibrationTrialV1

Release source · Runtime guarantees · Capability guides

`prepare_eval_judge_calibration`

from cayu.evals import prepare_eval_judge_calibration

Resolve exact current authority without compiling any candidate request.

prepare_eval_judge_calibration(definition: EvalJudgeCalibrationDefinitionV1, target: CorpusTarget) -> PreparedEvalJudgeCalibration

Release source · Runtime guarantees · Capability guides

`PreparedEvalJudgeCalibration`

from cayu.evals import PreparedEvalJudgeCalibration

Process-local judge authority paired with immutable browser-reviewed input.

Decorators: @dataclass(frozen=True, slots=True).

Fields and attributes

Name	Type	Default
`definition`	`EvalJudgeCalibrationDefinitionV1`	`required`
`judge_profile`	`JudgeProfileIdentityV1`	`required`
`candidate_route_relation`	`Literal['independent_model', 'same_model']`	`required`
`evaluator`	`_CompiledStructuredModelJudgeAssertion`	`required`

Release source · Runtime guarantees · Capability guides

`run_eval_judge_calibration_trial`

from cayu.evals import run_eval_judge_calibration_trial

Execute exactly one judge call over retained text; no candidate path exists.

async run_eval_judge_calibration_trial(prepared: PreparedEvalJudgeCalibration, *, sequence: int) -> EvalJudgeCalibrationTrialV1

Release source · Runtime guarantees · Capability guides