cayu.evals.benchmark_package

Public Cayu API. Definitions are grouped by their public source module, and definitions from private implementation modules appear on their package’s page; adding a symbol does not move existing links. The import shown at each entry is a supported public namespace. Read the Python API overview and install the required extra.

benchmark_package_from_json

from cayu.evals import benchmark_package_from_json

benchmark_package_from_json(source: str) -> BenchmarkPackageV1

Release source · Runtime guarantees · Capability guides

benchmark_package_scenarios

from cayu.evals import benchmark_package_scenarios

Resolve scenarios with the same package provenance as their authored cases.

benchmark_package_scenarios(package: BenchmarkPackageV1) -> tuple[EvalScenarioDocumentV2, ...]

Release source · Runtime guarantees · Capability guides

benchmark_package_to_json

from cayu.evals import benchmark_package_to_json

benchmark_package_to_json(package: BenchmarkPackageV1) -> str

Release source · Runtime guarantees · Capability guides

benchmark_suite_selection

from cayu.evals import benchmark_suite_selection

Bind package identity into the existing comparable case and cohort identities.

benchmark_suite_selection(package: BenchmarkPackageV1, case_ids: Sequence[str] | None=None) -> tuple[EvalSuiteDocumentV3, EvalSuiteSelectionV1]

Release source · Runtime guarantees · Capability guides

BenchmarkFileV1

from cayu.evals import BenchmarkFileV1

A package-relative source for one existing immutable scenario requirement.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
scenario_revision	StrictStr	required
requirement_id	StrictStr	required
path	StrictStr	required

validate_revision (method)

BenchmarkFileV1.validate_revision(value: str, info) -> str

validate_id (method)

BenchmarkFileV1.validate_id(value: str, info) -> str

validate_path (method)

BenchmarkFileV1.validate_path(value: str) -> str

Release source · Runtime guarantees · Capability guides

BenchmarkPackageV1

from cayu.evals import BenchmarkPackageV1

A versioned benchmark over the canonical authored-suite/case contracts.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
schema_version	Literal[1]	1
revision	StrictStr	required
id	StrictStr	required
version	StrictStr	required
scorer_id	StrictStr	required
scorer_version	StrictStr	required
suite	EvalSuiteDocumentV3	required
scenarios	tuple[EvalScenarioDocumentV2, ...]	Field(default=(), max_length=1000)
files	tuple[BenchmarkFileV1, ...]	Field(default=(), max_length=1000)
requirements	BenchmarkRequirementsV1	Field(default_factory=BenchmarkRequirementsV1)

validate_schema_version (method)

BenchmarkPackageV1.validate_schema_version(value: object) -> object

validate_revision (method)

BenchmarkPackageV1.validate_revision(value: str, info) -> str

validate_ids (method)

BenchmarkPackageV1.validate_ids(value: str, info) -> str

validate_versions (method)

BenchmarkPackageV1.validate_versions(value: str, info) -> str

validate_order (method)

BenchmarkPackageV1.validate_order(value: object, info) -> object

validate_contract (method)

BenchmarkPackageV1.validate_contract() -> BenchmarkPackageV1

create (method)

BenchmarkPackageV1.create(*, id: str, version: str, scorer_id: str, scorer_version: str, suite: EvalSuiteDocumentV3, scenarios: Sequence[EvalScenarioDocumentV2]=(), files: Sequence[BenchmarkFileV1]=(), requirements: BenchmarkRequirementsV1 | None=None) -> BenchmarkPackageV1

Release source · Runtime guarantees · Capability guides

BenchmarkRequirementsV1

from cayu.evals import BenchmarkRequirementsV1

Names which must already exist on the explicitly selected trusted target.

Bases: _PortableModel.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
environments	tuple[StrictStr, ...]	Field(default=(), max_length=64)
tools	tuple[StrictStr, ...]	Field(default=(), max_length=256)

validate_order (method)

BenchmarkRequirementsV1.validate_order(value: object, info) -> object

validate_names (method)

BenchmarkRequirementsV1.validate_names(value: tuple[str, ...], info) -> tuple[str, ...]

Release source · Runtime guarantees · Capability guides

load_benchmark_package

from cayu.evals import load_benchmark_package

Read a manifest (or directory/benchmark.json) and verify every named input file.

load_benchmark_package(path: str | Path) -> LoadedBenchmarkPackage

Release source · Runtime guarantees · Capability guides

LoadedBenchmarkPackage

from cayu.evals import LoadedBenchmarkPackage

Decorators: @dataclass(frozen=True).

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
package	BenchmarkPackageV1	required
root	Path	required
file_contents	tuple[tuple[BenchmarkFileV1, bytes], ...]	required
manifest_path	Path | None	None

Release source · Runtime guarantees · Capability guides