harness package

Reviewed against commit 8a2cb42ad on 2026-10-23.

Reference: package page (20 modules, 4,131 lines; seven of them in evaluators) · key modules: runner, executor, healing, history, isolation, cli, main. Where the harness sits among orchestrata's tooling is in the overview.

The story

Orchestrata's volks are autonomous agents, and autonomous agents are notoriously hard to test with ordinary unit tests: their behavior depends on prompts, live infrastructure, and multi-step message flows, and it drifts as the models and prompts behind them change. orchestrata_ultimate.harness — the Volk Test Harness — exists to close that gap. Instead of asserting on a function's return value, you describe a scenario in YAML: which volk to exercise, what input to send, how many trials to run, and which evaluators should judge the output. The harness then executes that scenario against the real system, scores every trial, and records the outcome so the same scenario can be re-run later and compared. It is runnable directly via python -m orchestrata_ultimate.harness, whose CLI (run, list, snapshot, report, trends, config) is the usual entry point.

A run flows through a small pipeline. config.load_config() builds a HarnessConfig from YAML plus environment-variable overrides, and scenario.py loads and validates Scenario models (Pydantic, from YAML), optionally executing a setup hook first. HarnessRunner then orchestrates each scenario: for every trial it mints a test_id, acquires an isolated sandbox, assembles a HarnessContext, hands the scenario to an executor, runs the configured evaluators concurrently, computes a weighted composite score, diffs the output against a stored snapshot, and — on failure — invokes the self-healing loop before aggregating the per-trial TrialResults into a ScenarioResult. Execution comes in two flavors behind the Executor abstract base. CXPacketExecutor builds a real CXPPacket and dispatches it through the kernel's MessageDispatcher to the target volk, exercising the genuine agent-runtime path, while DirectCallExecutor dynamically imports a volk class and calls one of its methods in-process for fast, infrastructure-light trials. Any subprocess a scenario wants to run is gated by a command whitelist (enforce_whitelist, raising SecurityError), so scenarios can't smuggle in arbitrary shell access.

Judging is pluggable. Evaluators register themselves at import time via the @register_evaluator decorator, and scenario YAML refers to them by name — file_exists, syntax_check, lint_check, trace_validator, llm_judge. Each subclass of Evaluator returns an EvaluatorResult, so a trial's verdict is the composition of deterministic checks (AST parse, ruff lint, expected files, @traces requirement annotations) and — where quality is subjective — an LLM-as-judge score obtained through the project's call_llm client. Because trials mutate real state, isolation.py gives each trial its own sandbox: a git worktree under .harness/worktrees/ (WorktreeIsolation) or an ephemeral Incus container (IncusIsolation — this project runs Incus, not Docker), plus per-trial Redis database numbers and graph resource prefixes so parallel trials never collide. When a trial fails, SelfHealingLoop (Spec 102, Workstream F) can attempt an autonomous 3-hop repair: it dispatches diagnostic → repair → escalation CXP packets to the volk, extracts a repair strategy from the response, and records each hop as a HopDiagnostic inside a HealingResult. When no dispatcher is available it degrades to a simpler internal retry.

The harness collaborates with its neighbors rather than reimplementing them: it speaks to volks through the cxp package (packet construction, endpoint_config) and the kernel's MessageDispatcher, resolves volk identities through volk_registry, judges output through the llm client, and persists outcomes through history.py — either JSONHistoryTracker (atomic monthly JSON files) or GraphHistoryTracker (MERGE-ing Scenario/Volk/Run/Regression nodes into the operational knowledge graph, orchkg; the class was named for Neo4j until the 2026-07 migration). reporting.py turns recorded history into HTML reports and JUnit XML with per-volk and per-scenario stats and regression counts, and exposes Cypher through TrendQueries for pass-rate and score trends; snapshots.py's SnapshotManager provides versioned, text-based snapshot comparison across runs so a regression surfaces as a unified diff. Together these make the harness the project's answer to the one question ordinary test suites cannot answer: did the agents get better or worse?

Key responsibilities

• Scenario-driven evaluation — load YAML Scenario definitions, run each for N independent trials, and aggregate results into a scored ScenarioResult.
• Two execution modes — inject a real CXPPacket into a live volk (CXPacketExecutor) or call a volk method in-process (DirectCallExecutor), behind the Executor ABC.
• Pluggable evaluators — deterministic checks (file_exists, syntax_check, lint_check, trace_validator) plus subjective llm_judge, each registered by name and returning an EvaluatorResult.
• Trial isolation — per-trial git worktree or ephemeral Incus container, plus isolated Redis DB numbers and graph prefixes, so trials can't contaminate each other or standing infrastructure.
• Self-healing — a bounded 3-hop CXP repair loop (SelfHealingLoop) that diagnoses and retries failed trials, with a dispatcher-free fallback.
• History, snapshots & reporting — durable JSON/graph history, versioned snapshot diffing for regression detection, and HTML/JUnit report generation with trend queries.
• Command safety — a whitelist gate (enforce_whitelist → SecurityError) on any subprocess a scenario tries to run.

Class diagram

Scenario definitions, runtime context, results, and the runner that ties them together. A HarnessRunner loads a Scenario (which owns a list of EvaluatorConfig), builds a fresh HarnessContext per trial, and produces a ScenarioResult composed of TrialResults, each holding its EvaluatorResults (generated from the source, render-verified):

Execution engines and evaluators. Two Executor implementations run a scenario (CXP injection vs. in-process call); five Evaluator subclasses score the output — each registered by its name classvar. SecurityError guards the subprocess whitelist:

The concrete evaluator name classvars — the strings used in scenario YAML — are: FileExistsEvaluator → file_exists, SyntaxEvaluator → syntax_check, LintEvaluator → lint_check, TraceEvaluator → trace_validator, LLMJudgeEvaluator → llm_judge.

Isolation, history persistence, snapshots, and self-healing. Two IsolationManager strategies sandbox each trial; two HistoryTracker backends persist results; SnapshotManager diffs outputs; SelfHealingLoop returns a HealingResult of HopDiagnostic records:

Notes

• Spec: SPEC-102 (Volk Test Harness). The self-healing loop is Workstream F; evaluators are US-005; isolation and executors are US-006; snapshots are US-004; history is US-002.
• Entry point: python -m orchestrata_ultimate.harness → harness/cli.py (subcommands run, list, snapshot, report, trends, config).
• Reporting/trends: reporting.py exposes generate_report() (HTML), generate_junit_xml(), and the TrendQueries Cypher helpers (pass_rate_last_30_days, regression_count, avg_score_trend) — module-level functions/helpers, not shown in the diagrams above.
• Isolation default: HarnessConfig.isolation_mode defaults to "worktree"; set "incus" to sandbox trials in ephemeral oh-trial-<id> containers.
• Source: src/orchestrata_ultimate/harness/.

Class map (generated)

Generated from the live source AST by scripts/gen-module-class-diagrams.py — do not hand-edit. Abstract classes have a green border; members listed are those defined on the class itself (inherited members excluded). Solid empty-triangle arrows = in-package inheritance; dashed arrows into dashed boxes = external/stdlib/cross-package bases; filled diamond = composition (constructed in __init__); open diamond = aggregation (annotated/injected reference); diamond edge labels name the owning attribute.