Skip to content

cayu eval

Generated from the released command parser. These commands show help only; operations run when you invoke the command without --help.

Terminal window
usage: cayu eval [-h]
                 {run,resume,cancel,rescore,retry,package,status,failures,export,report,memory-report,compare,validate,inspect,merge} ...


Run and report Cayu runtime-native evals. Start with `cayu eval run` for the
project-configured hermetic proof.


positional arguments:
  {run,resume,cancel,rescore,retry,package,status,failures,export,report,memory-report,compare,validate,inspect,merge}
    run                 Run a configured or explicit eval plan.
    resume              Resume a native benchmark campaign using its saved
                        admission.
    cancel              Request native cancellation for a benchmark campaign.
    rescore             Evaluate a new static scorer using saved output,
                        without candidate or judge calls.
    retry               Execute a bounded, explicitly selected successor
                        trial.
    package             Create, validate, and inspect portable benchmark
                        packages.
    status              Inspect process-run progress.
    failures            Inspect recorded case failures and session
                        diagnostics.
    export              Export validated process receipts and results to a new
                        private ZIP.
    report              Render a JSON or HTML report from eval results.
    memory-report       Build a paired repeated-trial memory experiment
                        report.
    compare             Compare baseline and current eval results.
    validate            Validate a portable eval corpus.
    inspect             Inspect a validated portable eval corpus.
    merge               Atomically merge portable eval corpora.


options:
  -h, --help            show this help message and exit

cayu eval run

Section titled “cayu eval run”
Terminal window
usage: cayu eval run [-h] [--format {json} | --json] [--output FILE]
                     [--html-output FILE] [--corpus FILE] [--suite SUITE_ID]
                     [--max-concurrency COUNT] [--stagger-seconds SECONDS]
                     [--processes COUNT] [--process-directory DIRECTORY]
                     [--case-timeout-seconds SECONDS] [--package PATH]
                     [--case CASE_ID] [--campaign-directory NEW_DIRECTORY]
                     [--admit-only] [--trials TRIALS]
                     [--minimum-passed-trials MINIMUM_PASSED_TRIALS]
                     [--max-steps MAX_STEPS]
                     [--max-total-tokens MAX_TOTAL_TOKENS]
                     [--max-tool-calls MAX_TOOL_CALLS]
                     [--max-estimated-cost MAX_ESTIMATED_COST]
                     [--currency CURRENCY] [--provider PROVIDER]
                     [--model MODEL] [--environment ENVIRONMENT]
                     [--max-execution-attempts MAX_EXECUTION_ATTEMPTS]
                     [--max-retry-attempts MAX_RETRY_ATTEMPTS]
                     [--retry-category RETRY_CATEGORIES]
                     [--retry-backoff-seconds RETRY_BACKOFF_SECONDS]
                     [target]


Run a configured or explicit eval plan and emit a stable JSON result. Use
`--output FILE` to save it.


positional arguments:
  target                Python target that returns a direct-suite or corpus-
                        target EvalPlan. Defaults to [tool.cayu].eval_target.


options:
  -h, --help            show this help message and exit
  --format {json}       Output format (default: json).
  --json                Emit JSON (shorthand for `--format json`).
  --output, -o FILE     Write output to FILE instead of stdout.
  --html-output FILE    Also write an HTML report to FILE.
  --corpus FILE         Run a portable corpus through the target's trusted
                        CorpusTarget.
  --suite SUITE_ID      Corpus suite to run (optional only when the corpus has
                        one suite).
  --max-concurrency COUNT
                        Maximum concurrently executing cases (default: 1).
  --stagger-seconds SECONDS
                        Minimum interval between case admissions (default: 0);
                        2 minutes = 120 seconds.
  --processes COUNT     Fresh Python worker processes for native direct suites
                        (default: 1).
  --process-directory DIRECTORY
                        New private directory for process admission, logs, and
                        worker results.
  --case-timeout-seconds SECONDS
                        Limit each direct-suite case to SECONDS; corpus
                        timeouts are declared in JSON.
  --package PATH        Run a portable benchmark package through native
                        durable Evals.
  --case CASE_ID        Explicit packaged-benchmark case selection; repeat to
                        select multiple cases.
  --campaign-directory NEW_DIRECTORY
                        New private directory for native campaign admission,
                        catalog, and inspection.
  --admit-only          Persist a packaged campaign without dispatching it;
                        use eval resume to start it.
  --trials TRIALS       Independent trials per package case.
  --minimum-passed-trials MINIMUM_PASSED_TRIALS
                        Passed trials needed by the package's native trial
                        policy.
  --max-steps MAX_STEPS
                        Narrow the packaged target's per-trial model-step
                        limit.
  --max-total-tokens MAX_TOTAL_TOKENS
                        Per-trial observed-token stop threshold; in-flight
                        usage can overshoot.
  --max-tool-calls MAX_TOOL_CALLS
                        Narrow the packaged target's per-trial tool-call
                        allowance.
  --max-estimated-cost MAX_ESTIMATED_COST
                        Per-trial priced budget; requires the target's trusted
                        PriceBook.
  --currency CURRENCY   Currency for --max-estimated-cost (default: USD).
  --provider PROVIDER   Registered provider for the packaged run; requires
                        --model.
  --model MODEL         Model on the explicitly selected --provider.
  --environment ENVIRONMENT
                        Existing target environment for the packaged run.
  --max-execution-attempts MAX_EXECUTION_ATTEMPTS
                        Explicitly authorize up to this many full execution
                        attempts after interruption (1-10).
  --max-retry-attempts MAX_RETRY_ATTEMPTS
                        Reserve up to 3 selective successor trials per
                        original slot; default 0.
  --retry-category RETRY_CATEGORIES
                        Eligible typed failure category; repeat. Wrong answers
                        are excluded by default.
  --retry-backoff-seconds RETRY_BACKOFF_SECONDS
                        Minimum elapsed time after the preceding attempt,
                        0-3600 seconds.

cayu eval resume

Section titled “cayu eval resume”
Terminal window
usage: cayu eval resume [-h] directory [target]


Resume saved native work under its admitted recovery policy. Inspect it with
`cayu eval status DIRECTORY`.


positional arguments:
  directory
  target


options:
  -h, --help  show this help message and exit

cayu eval cancel

Section titled “cayu eval cancel”
Terminal window
usage: cayu eval cancel [-h] directory


Request cancellation of admitted campaign runs. Inspect settlement with `cayu
eval status DIRECTORY`.


positional arguments:
  directory


options:
  -h, --help  show this help message and exit

cayu eval rescore

Section titled “cayu eval rescore”
Terminal window
usage: cayu eval rescore [-h] --package SCORER_PACKAGE --output OUTPUT
                         directory


Apply a new static scorer package to retained output. Review the new receipt
at --output; original results remain available through eval status.


positional arguments:
  directory


options:
  -h, --help            show this help message and exit
  --package SCORER_PACKAGE
  --output OUTPUT       New scorer receipt outside the original campaign.

cayu eval retry

Section titled “cayu eval retry”
Terminal window
usage: cayu eval retry [-h] --case CASE_ID --trial TRIAL [--attempt ATTEMPT]
                       [--allow-reexecution]
                       directory [target]


Retry one eligible original trial within its reserved allowance. Inspect
original and successor evidence with `cayu eval status DIRECTORY`.


positional arguments:
  directory
  target


options:
  -h, --help           show this help message and exit
  --case CASE_ID
  --trial TRIAL
  --attempt ATTEMPT
  --allow-reexecution  Explicit decision to replay when the target has no
                       application reset contract.

cayu eval package

Section titled “cayu eval package”
Terminal window
usage: cayu eval package [-h] {init,validate,inspect} ...


Manage portable native benchmark definitions. Start with `cayu eval package
init DIRECTORY`, then validate the package.


positional arguments:
  {init,validate,inspect}
    init                Create the installed synthetic benchmark package in a
                        new directory.
    validate            Validate a package without importing any execution
                        target.
    inspect             Inspect a package without importing any execution
                        target.


options:
  -h, --help            show this help message and exit

cayu eval package init

Section titled “cayu eval package init”
Terminal window
usage: cayu eval package init [-h] [--failure-modes] [--interruption-case]
                              [--scorer-version SCORER_VERSION]
                              directory


Create a synthetic benchmark package without external calls. Next use `cayu
eval package validate DIRECTORY` or launch with `cayu eval run --package
DIRECTORY`.


positional arguments:
  directory


options:
  -h, --help            show this help message and exit
  --failure-modes       Include synthetic provider and scoring failures.
  --interruption-case   Include a synthetic pending call for explicit process-
                        loss qualification.
  --scorer-version SCORER_VERSION
                        Explicit synthetic scorer version for saved-output
                        qualification.

cayu eval package validate

Section titled “cayu eval package validate”
Terminal window
usage: cayu eval package validate [-h] [--case CASE_IDS] path


Validate exact package material and cohort identity. Launch a validated
package with `cayu eval run TARGET --package PATH`.


positional arguments:
  path


options:
  -h, --help       show this help message and exit
  --case CASE_IDS

cayu eval package inspect

Section titled “cayu eval package inspect”
Terminal window
usage: cayu eval package inspect [-h] [--case CASE_IDS] path


Inspect exact package material and cohort identity. Launch a validated package
with `cayu eval run TARGET --package PATH`.


positional arguments:
  path


options:
  -h, --help       show this help message and exit
  --case CASE_IDS

cayu eval status

Section titled “cayu eval status”
Terminal window
usage: cayu eval status [-h] [--sessions | --no-sessions]
                        [--session-evidence FILE]
                        [--max-sessions MAX_SESSIONS]
                        [--max-diagnostics MAX_DIAGNOSTICS] [--case CASE_ID]
                        [--format {json,table} | --json | --table]
                        [--output FILE]
                        PROCESS_DIRECTORY


Read native process receipts without importing the target or starting work.
Session observations use explicit read-only store links. Admission and recent
events do not prove worker liveness.


positional arguments:
  PROCESS_DIRECTORY


options:
  -h, --help            show this help message and exit
  --sessions, --no-sessions
                        Include bounded live session observations (default:
                        enabled for failures).
  --session-evidence FILE
                        Explicit launch- and case-bound session locators for
                        an older run.
  --max-sessions MAX_SESSIONS
  --max-diagnostics MAX_DIAGNOSTICS
  --case CASE_ID
  --format {json,table}
                        Output format (default: table).
  --json                Emit JSON (shorthand for `--format json`).
  --table               Emit human-readable output (shorthand for `--format
                        table`).
  --output, -o FILE     Write output to FILE instead of stdout.

cayu eval failures

Section titled “cayu eval failures”
Terminal window
usage: cayu eval failures [-h] [--sessions | --no-sessions]
                          [--session-evidence FILE]
                          [--max-sessions MAX_SESSIONS]
                          [--max-diagnostics MAX_DIAGNOSTICS] [--case CASE_ID]
                          [--format {json,table} | --json | --table]
                          [--output FILE]
                          PROCESS_DIRECTORY


Read native process receipts without importing the target or starting work.
Session observations use explicit read-only store links. Admission and recent
events do not prove worker liveness.


positional arguments:
  PROCESS_DIRECTORY


options:
  -h, --help            show this help message and exit
  --sessions, --no-sessions
                        Include bounded live session observations (default:
                        enabled for failures).
  --session-evidence FILE
                        Explicit launch- and case-bound session locators for
                        an older run.
  --max-sessions MAX_SESSIONS
  --max-diagnostics MAX_DIAGNOSTICS
  --case CASE_ID
  --format {json,table}
                        Output format (default: table).
  --json                Emit JSON (shorthand for `--format json`).
  --table               Emit human-readable output (shorthand for `--format
                        table`).
  --output, -o FILE     Write output to FILE instead of stdout.

cayu eval export

Section titled “cayu eval export”
Terminal window
usage: cayu eval export [-h] --output NEW_ZIP PROCESS_DIRECTORY


Export exact inspected receipt and result bytes with hashes. This does not
snapshot session stores or include application files and logs.


positional arguments:
  PROCESS_DIRECTORY


options:
  -h, --help            show this help message and exit
  --output, -o NEW_ZIP

cayu eval report

Section titled “cayu eval report”
Terminal window
usage: cayu eval report [-h] [--format {html,json} | --json | --html]
                        [--output FILE]
                        RESULTS_JSON


Render saved eval results as HTML by default or JSON explicitly. Use `--output
FILE` to save the report.


positional arguments:
  RESULTS_JSON          Eval JSON results file.


options:
  -h, --help            show this help message and exit
  --format {html,json}  Output format (default: html).
  --json                Emit JSON (shorthand for `--format json`).
  --html                Emit HTML (shorthand for `--format html`).
  --output, -o FILE     Write output to FILE instead of stdout.

cayu eval memory-report

Section titled “cayu eval memory-report”
Terminal window
usage: cayu eval memory-report [-h] [--format {html,json} | --json | --html]
                               [--output FILE]
                               REQUEST_JSON


Validate an exact memory experiment request and build its deterministic fixed-
candidate report without launching additional work.


positional arguments:
  REQUEST_JSON          Memory experiment report request JSON file.


options:
  -h, --help            show this help message and exit
  --format {html,json}  Output format (default: json).
  --json                Emit JSON (shorthand for `--format json`).
  --html                Emit HTML (shorthand for `--format html`).
  --output, -o FILE     Write output to FILE instead of stdout.

cayu eval compare

Section titled “cayu eval compare”
Terminal window
usage: cayu eval compare [-h] [--format {html,json} | --json | --html]
                         [--output FILE] [--score-tolerance DELTA]
                         BASELINE_JSON CURRENT_JSON


Compare baseline and current eval results. JSON is the default; a nonzero exit
reports regressions.


positional arguments:
  BASELINE_JSON
  CURRENT_JSON


options:
  -h, --help            show this help message and exit
  --format {html,json}  Output format (default: json).
  --json                Emit JSON (shorthand for `--format json`).
  --html                Emit HTML (shorthand for `--format html`).
  --output, -o FILE     Write output to FILE instead of stdout.
  --score-tolerance DELTA
                        Allowed score drop before a regression is flagged
                        (default: 0.0).

cayu eval validate

Section titled “cayu eval validate”
Terminal window
usage: cayu eval validate [-h] [--format {json,table} | --json | --table]
                          [--output FILE]
                          CORPUS_JSON


Validate one bounded portable eval corpus without loading an application
target.


positional arguments:
  CORPUS_JSON


options:
  -h, --help            show this help message and exit
  --format {json,table}
                        Output format (default: table).
  --json                Emit JSON (shorthand for `--format json`).
  --table               Emit human-readable output (shorthand for `--format
                        table`).
  --output, -o FILE     Write output to FILE instead of stdout.

cayu eval inspect

Section titled “cayu eval inspect”
Terminal window
usage: cayu eval inspect [-h] [--format {json,table} | --json | --table]
                         [--output FILE]
                         CORPUS_JSON


Show the target, suites, cases, assertions, and expanded result count for one
validated corpus.


positional arguments:
  CORPUS_JSON


options:
  -h, --help            show this help message and exit
  --format {json,table}
                        Output format (default: table).
  --json                Emit JSON (shorthand for `--format json`).
  --table               Emit human-readable output (shorthand for `--format
                        table`).
  --output, -o FILE     Write output to FILE instead of stdout.

cayu eval merge

Section titled “cayu eval merge”
Terminal window
usage: cayu eval merge [-h] [--replace-conflicts] [--format {json,table} |
                       --json | --table] [--output FILE]
                       DESTINATION_JSON CORPUS_JSON [CORPUS_JSON ...]


Validate and merge compatible corpus files, then atomically replace the
destination.


positional arguments:
  DESTINATION_JSON
  CORPUS_JSON


options:
  -h, --help            show this help message and exit
  --replace-conflicts   Replace same-ID content conflicts in command-line
                        order.
  --format {json,table}
                        Output format (default: table).
  --json                Emit JSON (shorthand for `--format json`).
  --table               Emit human-readable output (shorthand for `--format
                        table`).
  --output, -o FILE     Write output to FILE instead of stdout.
Section titled “Related guides”

Getting started · Operations · Configuration