cayu.evals.testing

Public Cayu API. Definitions are grouped by their public source module, and definitions from private implementation modules appear on their package’s page; adding a symbol does not move existing links. The import shown at each entry is a supported public namespace. Read the Python API overview and install the required extra.

model_facing_text

from cayu.evals import model_facing_text

Render a model request as the text its model can read, one line per part.

Text, peer content (with the attribution header bundled providers add), tool calls (name and arguments) and tool results (content only) are shown; structured tool data, provider state and reasoning are not, because custom tool results reach the model only through content. Attachments appear as [attachment]. Use it with ScriptedModelProvider.requests to check what a real model would have seen at each step.

model_facing_text(request: ModelRequest) -> str

Release source · Runtime guarantees · Capability guides

model_facing_tool_result

from cayu.evals import model_facing_tool_result

Return the text a model receives for one tool result.

Custom tool results reach the model only through content (plus any file attachments); their structured data is for dashboards, workflows and downstream tools. A tool that puts its findings only in structured leaves the model with nothing to read. Assert on this in scripted-provider tests instead of inspecting structured.

model_facing_tool_result(result: ToolResult | ToolResultPart) -> str

Release source · Runtime guarantees · Capability guides

scripted_structured_output

from cayu.evals import scripted_structured_output

Build one complete provider script that submits structured JSON output.

This is the public test/eval seam for Cayu’s provider-neutral structured-output tool protocol. Callers supply only the JSON value; Cayu owns the reserved tool name and argument envelope.

scripted_structured_output(output: Any, *, id: str | None=None) -> tuple[ModelStreamEvent, ModelStreamEvent]

Release source · Runtime guarantees · Capability guides

ScriptedModelProvider

from cayu.evals import ScriptedModelProvider

Deterministic provider for local evals and tests.

Pass positional event batches for sequential scripts, or a synchronous response_factory selecting one complete batch from a detached ModelRequest. Shared concurrent evals require request-aware selection; per-trial workflow factories may instead construct independent positional providers. Native structured-output support is explicitly opt-in.

Bases: ModelProvider.

Fields and attributes

Defaults below are source expressions; Field(...) includes schema/validation settings and default_factory creates a fresh value. Inherited declarations are included. Constructor behavior follows the model or dataclass shown in the bases/decorators.

Name	Type	Default
name	class attribute	'scripted'
billing_provider_name	str | None	None
usage_dialect	UsageDialect	UsageDialect.AUTO
supports_native_structured_output	class attribute	False

prepare_auxiliary_request (method)

ScriptedModelProvider.prepare_auxiliary_request(request: ModelRequest, *, max_output_tokens: int) -> ModelRequest

stream_deadlines (property)

ScriptedModelProvider.stream_deadlines() -> ProviderStreamDeadlines

Return immutable dispatch deadlines for this provider.

runtime_stream (method)

ScriptedModelProvider.runtime_stream(request: ModelRequest) -> AsyncIterator[ModelStreamEvent]

Open the governed model-attempt stream.

Runtime, application, example, and conformance callers must use this entrance so normalized semantic and absolute deadlines also guard opaque custom providers. Transparent provider wrappers must override this entrance and delegate to the wrapped provider’s runtime_stream; forwarding only the raw stream hook would add an opaque outer guard that cannot trust terminal evidence accepted by the wrapped adapter.

execution_profile_identity (property)

ScriptedModelProvider.execution_profile_identity() -> ExecutionProfileBehaviorIdentity | None

Optional application-versioned identity for opaque adapter behavior.

provider_operation_mode (property)

ScriptedModelProvider.provider_operation_mode() -> ProviderOperationMode

provider_operations (property)

ScriptedModelProvider.provider_operations() -> ProviderOperationAdapter | None

context_pressure_profile (property)

ScriptedModelProvider.context_pressure_profile() -> ModelContextPressureProfile

request_cache_policy (method)

ScriptedModelProvider.request_cache_policy(request: ModelRequest) -> CachePolicy | None

Describe the effective provider-visible cache breakpoints, when known.

Implementations must be deterministic, side-effect free, and perform no I/O.

request_cache_projection (method)

ScriptedModelProvider.request_cache_projection(request: ModelRequest) -> RequestCacheProjection | None

Project the effective cache policy and provider-selected prefix.

Providers whose message preparation can filter or normalize content should override this hook and return the exact ephemeral prefix through the marker they applied. The default preserves compatibility with policy-only provider extensions and lets the runtime derive a provider-neutral prefix. Implementations must be deterministic, side-effect free, and perform no I/O.

request_footprint_options (method)

ScriptedModelProvider.request_footprint_options(request: ModelRequest) -> dict[str, Any]

Project privacy-safe provider-visible options for local measurement.

Implementations must return copied JSON, omit arbitrary metadata and secret-bearing extension fields, and perform no I/O.

request_fingerprint_options (method)

ScriptedModelProvider.request_fingerprint_options(request: ModelRequest) -> dict[str, Any]

Project complete effective provider-visible options for local analysis.

The runtime keeps this copied value ephemeral: it supplies keyed HMAC input when configured and identifies which non-allowlisted option categories are active, but the value and its field names are never persisted. Built-in adapters override this hook so ignored namespaces, defaults, and normalized controls follow their payload-building semantics. The default conservatively treats every non-runtime request option as provider-visible. Implementations must be deterministic, side-effect free, and perform no I/O.

preflight_model_target (method)

ScriptedModelProvider.preflight_model_target(*, model: str) -> None

Reject an unavailable provider/model target before runtime mutation.

This side-effect-free hook is called during initial-run admission before the session is created and again immediately before provider dispatch. Ordinary providers need no override. Applications may use it for credential-free inspection placeholders whose live execution must fail with a concrete setup diagnostic.

preflight_thinking (method)

ScriptedModelProvider.preflight_thinking(*, model: str, thinking: ThinkingConfig | None) -> None

preflight_portable_messages (method)

ScriptedModelProvider.preflight_portable_messages(*, model: str, messages: list[Message], tools: list[dict[str, Any]]) -> None

preflight_hosted_tools (method)

ScriptedModelProvider.preflight_hosted_tools(*, model: str, hosted_tools: tuple[OpenAIWebSearch, ...], options: dict[str, Any]) -> None

Reject provider-hosted authority this adapter cannot honor.

This hook is side-effect free and is called before a new session is created. The conservative default rejects every hosted tool so custom providers cannot silently omit registered execution authority.

supports_targeted_tool_projection (method)

ScriptedModelProvider.supports_targeted_tool_projection(*, model: str, protocol: str) -> bool

Report an explicitly established provider/model projection capability.

The conservative default is unsupported. Implementations must not infer support from a model-name pattern because provider aliases and compatible endpoints can expose different wire contracts under the same model name.

preflight_targeted_tool_projection (method)

ScriptedModelProvider.preflight_targeted_tool_projection(*, model: str, protocol: str) -> None

Reject targeted-tool authority that this provider cannot project.

supports_tool_discovery_projection (method)

ScriptedModelProvider.supports_tool_discovery_projection(*, model: str, protocol: str) -> bool

Report an explicitly established provider/model discovery capability.

preflight_tool_discovery_projection (method)

ScriptedModelProvider.preflight_tool_discovery_projection(*, model: str, protocol: str) -> None

Reject discovery authority that this provider cannot project.

billing_identity_for_request (method)

async ScriptedModelProvider.billing_identity_for_request(request: ModelRequest) -> BillingIdentity | None

Resolve optional request context needed for commercial accounting.

billing_identity_for_completion (method)

ScriptedModelProvider.billing_identity_for_completion(identity: BillingIdentity | None, payload: dict[str, Any]) -> BillingIdentity | None

Merge provider-reported completion facts into a request identity.

preflight_native_structured_output_schema (method)

ScriptedModelProvider.preflight_native_structured_output_schema(json_schema: dict[str, Any]) -> None

Optionally reject a NATIVE structured-output schema this adapter’s provider API would refuse, before any model request is made.

The runtime calls this at every entrance (after the supports_native_structured_output check, before any session is created or transitioned) with a caller-owned copy of the schema; implementations must not mutate it. Adapters should raise NativeStructuredOutputSchemaInvalid with a path-specific message. The default accepts everything: adapters without provider-specific schema rules stay source-compatible.

count_input_tokens (method)

async ScriptedModelProvider.count_input_tokens(request: ModelRequest) -> InputTokenCountResult | None

Optionally count the input tokens for one request before submission.

Providers that need to call a remote counting endpoint should do so here. Remote counters are opt-in observability/calibration hooks, not default context-overflow enforcement. The default implementation is intentionally unavailable so existing providers remain source-compatible.

stream (method)

async ScriptedModelProvider.stream(request: ModelRequest) -> AsyncIterator[ModelStreamEvent]

init (method)

ScriptedModelProvider.__init__(events: Iterable[ModelStreamEvent] | Iterable[Iterable[ModelStreamEvent]] | None=None, *, name: str='scripted', background: bool=False, response_factory: Callable[[ModelRequest], Iterable[ModelStreamEvent]] | None=None, supports_native_structured_output: bool=False) -> None

background_operation_ids (property)

ScriptedModelProvider.background_operation_ids() -> tuple[str, ...]

Return deterministic operation ids created by this scripted provider.

complete_background_operation (method)

ScriptedModelProvider.complete_background_operation(operation_id: str | None=None) -> str

Simulate one scripted provider operation finishing while Cayu is offline.

Release source · Runtime guarantees · Capability guides