cayu.evals.judges

Public Cayu API. Definitions are grouped by their public source module, and definitions from private implementation modules appear on their package’s page; adding a symbol does not move existing links. The import shown at each entry is a supported public namespace. Read the Python API overview and install the required extra.

LLMJudge

from cayu.evals import LLMJudge

Graded assertion: a model scores the run’s output 0..1 against a rubric.

The judge runs its OWN agent — configured by the caller on app, typically a stronger or different model than the agent under test — so judging is an explicit, separate dependency (not the live-handle coupling assertions otherwise avoid) and is deterministically testable with a scripted provider. The continuous score flows into the case/run score via the score-first format. The judgment is auditable: metadata records the judge’s provider/model, the rubric (and version), the exact prompt, the raw output, and the parsed score/rationale.

Every judge session is created with a durable zero-application-tool capability ceiling. The registered judge agent may have tools for its ordinary workloads, but their definitions are absent from the judge request and returned calls cannot reach authorization or execution. Each evaluation opens a new session on the judge app and deletes it (best-effort) once the judgment is captured, so large suites don’t accumulate orphan judge sessions; stores that don’t support delete_session simply retain them.

The graded material (task, final output, transcript) is delimited as untrusted data in the judge prompt, and the score is only accepted as a well-formed JSON object — a run under test cannot smuggle instructions or a fake score past the rubric. Judge configuration, execution, safety, and response-decoding failures are evaluator errors, never evidence that the candidate failed its rubric.

Bases: EvalAssertion.

name (property)

LLMJudge.name() -> str

assertion_revision (property)

LLMJudge.assertion_revision() -> str | None

Stable definition revision carried by results when one exists.

evaluates_failed_session (property)

LLMJudge.evaluates_failed_session() -> bool

Whether a failed root session is assertion evidence instead of a runner error.

reads_final_output_only (property)

LLMJudge.reads_final_output_only() -> bool

Whether this assertion reads nothing but the run’s final output.

Override to return True only when evaluate reads context.final_output (and, for workflows, the projected structured output) and no events, transcript, usage, children, probes, or memory evidence. Such assertions still score a completed workflow whose child evidence exceeded its capture bounds; undeclared assertions are unavailable on that partial evidence.

evaluate (method)

async LLMJudge.evaluate(context: EvalContext) -> EvalAssertionResult

required_probes (method)

LLMJudge.required_probes() -> ProbeRequirements

Live-environment data this assertion needs captured into the Trajectory.

Default: none. Workspace/artifact assertions override this so the runner snapshots exactly what they read while the environment is still live, keeping evaluation offline.

passed (method)

LLMJudge.passed(message: str='', *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

failed (method)

LLMJudge.failed(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

unavailable (method)

LLMJudge.unavailable(message: str, *, metadata: dict[str, Any] | None=None, cost_summary: SessionCostSummary | None=None) -> EvalAssertionResult

error (method)

LLMJudge.error(message: str, *, metadata: dict[str, Any] | None=None) -> EvalAssertionResult

score_result (method)

LLMJudge.score_result(score: float, *, threshold: float=1.0, message: str='', metadata: dict[str, Any] | None=None) -> EvalAssertionResult

init (method)

LLMJudge.__init__(app: CayuApp, *, agent_name: str, rubric: str, threshold: float=0.5, rubric_version: str | None=None, include_transcript: bool=False, name: str | None=None, timeout_seconds: int | None=None, max_input_tokens: int | None=None, max_output_tokens: int | None=None, max_total_tokens: int | None=None, max_estimated_cost: str | None=None, cost_currency: str | None=None, price_book: PriceBook | None=None) -> None

Release source · Runtime guarantees · Capability guides