orchestrata_ultimate.harness
Reference › orchestrata_ultimate › harness
Volk Test Harness — scenario-driven evaluation system for autonomous agents.
The Volk Test Harness (SPEC-102) evaluates autonomous agents by scenario instead of by function return value. A scenario, written in YAML, names a volk, an input, a mode and a list of weighted evaluators; the harness runs it for several trials in isolated environments, judges the output, compares it with a stored snapshot and records the outcome. The curated narrative gives the story and diagrams.
The parts, by role. Definition and configuration: models, config and scenario. Orchestration: the runner. Running a trial: isolation (git worktree or Incus container), executor (CXP packet or direct call, plus the command whitelist) and the evaluators. Outcomes: snapshots, history (JSON and graph), reporting (HTML, JUnit, trend queries) and the self-healing loop.
The entry point is the cli, started by main, with subcommands run, list, snapshot, report, trends and config. Control flows from the cli to the runner, which for each trial creates isolation, runs the setup hook, executes, evaluates, snapshots, optionally heals and cleans up, then records the aggregated result. To read the code, start at the runner.
Curated narrative
The hand-written story of this package: orchestrata_ultimate.harness.
20 modules, 4,131 lines of code.
Subpackages
| Package | Modules | LOC |
|---|---|---|
evaluators |
7 | 715 |
Modules
| Module | Summary | LOC |
|---|---|---|
__init__ |
Volk Test Harness — scenario-driven evaluation system for autonomous agents. | 27 |
__main__ |
Entry point for python -m orchestrata_ultimate.harness. |
6 |
cli |
CLI interface for the Volk Test Harness — argparse-based, minimal dependencies. | 422 |
config |
Configuration loader for the Volk Test Harness. | 133 |
executor |
Scenario execution engines — CXP packet injection and direct method calls. | 197 |
healing |
Self-healing loop for the Volk Test Harness. | 482 |
history |
History tracking for Volk Test Harness — JSON and graph backends. | 351 |
isolation |
Trial isolation via git worktrees and Incus containers. | 257 |
models |
Core dataclasses and Pydantic models for the Volk Test Harness. | 213 |
reporting |
HTML report generation and JUnit XML export for Volk Test Harness. | 419 |
runner |
HarnessRunner — orchestrates scenario execution, evaluation, and reporting. | 480 |
scenario |
Scenario loading, validation, and setup hook execution. | 240 |
snapshots |
Snapshot serialization and comparison for the Volk Test Harness. | 189 |
Dependency diagram
