Orchestrata Ultimate / Specs / 102-volk-test-harness

102-volk-test-harness: Spec 102 — Individual Volk Test Harness

The Individual Volk Test Harness is a scenario-driven evaluation system that exercises each volk (agent) individually through real LLM calls with real Redis/Neo4j infrastructure, operating on the orchestrata-ultimate codebase itself (dogfooding). It is an on-demand quality assurance tool, not part of the standard CI/CD test suite.

At a glance

Reading this spec

This spec defines the Individual Volk Test Harness, a scenario-driven evaluation system that exercises each volk (agent) on its own, using real LLM calls and real Redis and Neo4j infrastructure, against the orchestrata-ultimate codebase itself. It is an on-demand quality assurance tool and is not part of the standard CI/CD test suite. The reason is that volks act autonomously and their output varies from run to run, so quality has to be measured statistically, with real infrastructure, and attributed to a specific agent rather than inferred from end-to-end tests. The harness also catches regressions by comparing snapshots across runs, and can run a self-healing loop of at most 3 hops when a scenario fails.

The method: each volk has a directory of YAML scenarios (harness/scenarios/<volk_name>/) that describe tasks, expected outcomes and evaluation criteria; Python setup hooks prepare the environment; each scenario runs three independent trials, isolated in git worktrees or Incus containers; evaluators judge the result. The twenty functional requirements (FR-001 to FR-020) cover the scenario format, setup hooks, a dual LLM model strategy, CXP packet injection and direct method call modes, three trials per scenario, snapshot comparison, deterministic evaluators and an LLM-as-judge evaluator, volk-specific quality criteria, the self-healing loop, a command whitelist for safety, history in JSON and Neo4j, GitLab CI integration, a CLI, resource isolation, timeouts and cancellation, a configuration file and parallel execution. The 131 requirement IDs (REQ-102-NNN) pin down details: trials use separate Redis databases per worker; results sync to Neo4j after each scenario; healing triggers only on failure, not timeout; a trial passes only when all evaluators pass and the composite score meets the threshold; a setup hook times out after 30 seconds; and a rate-limited LLM call retries with exponential backoff, 5 retries with a 2 second base.

The code is spread across the harness package, with the most citations in the runner (26 requirements) and the healing loop (16), followed by the models, executor, CLI and history modules, plus the evaluators. The requirements traceability matrix lists each ID. The status line in the spec reads Draft and it tracks 3 files. Measured coverage is 104 of 131 requirements cited by at least one @traces in src/ (79.4%), spread over 16 modules; the module citing the most of them is orchestrata_ultimate/harness/runner (26 requirements). In tests/, 0 requirements are cited by a test-side @traces. The 27 requirements with no citing symbol are mainly the GitLab CI requirements, some volk-specific quality criteria, several error-condition handlers and the harness testing requirements; the behaviour may still exist in code that was never annotated. The spec also leaves five open questions, such as whether snapshot updates are automated and whether healing may change a volk's own source.

Requirements

Checkbox is the state recorded in the requirements checklist; Traced by is the count of src/ symbols whose @traces names the ID.

Code that traces this spec

Checklists

checklists/constitutional.md

SPEC-102: Constitutional Compliance Checklist — Individual Volk Test Harness

Article I: Human Authority

• [ ] Harness execution requires explicit human invocation (harness run)
• [ ] Snapshot updates require explicit human approval (harness snapshot update)
• [ ] Quality thresholds (90% pass rate) configured by humans, not auto-adjusted
• [ ] CI pipeline failure alerts human; does not auto-revert or auto-fix
• [ ] Self-healing loop never modifies volk source code in main repo
• [ ] Humans review diagnostic traces before acting on healing recommendations
• [ ] Harness results are advisory; human makes final deploy decision
• [ ] No autonomous harness scheduling without human-configured CI pipeline
• [ ] Harness config changes require human edit of harness/config.yaml
• [ ] All harness runs tagged with human session context when available

Article II: Real Infrastructure Only

• [ ] Production harness scenarios use real LLM calls (DeepSeek v4-pro/v4-flash)
• [ ] Production harness scenarios use real Redis/Neo4j connections
• [ ] Production harness scenarios use real git worktrees (not temp mocks)
• [ ] Deterministic evaluators use real ast.parse, ruff check, mypy binaries
• [ ] No unittest.mock, MagicMock, patch, or monkeypatch in production runs
• [ ] Mock LLM mode exists only for harness infrastructure testing
• [ ] All external service calls validated against real endpoints before counting as "pass"
• [ ] Fallback mode (no dispatcher) logs diagnostics but does not claim success

Article III: Trial Isolation

• [ ] Each trial uses a separate git worktree or Incus container
• [ ] Trials never share Redis DBs (worker-to-DB mapping enforced)
• [ ] Trials never share KG namespaces (UUID test_id prefix enforced)
• [ ] Trial cleanup runs regardless of pass/fail/timeout status
• [ ] Worktree path validated to be inside repository before use
• [ ] Path traversal outside worktree blocked via validation (worktree) or chroot (Incus)
• [ ] No global mutable state in harness code (no module-level caches, registries, counters)
• [ ] Parallel execution safe: python scripts/run_tests.py passes without xdist_group or sequential ordering (pytest-xdist -n/--dist is BANNED)

Article IV: Safety and Sandboxing

• [ ] Command whitelist enforced before every subprocess invocation
• [ ] Whitelist violations raise SecurityError and mark trial FAILED
• [ ] git push and force-push excluded from whitelist
• [ ] API keys read from .env only; never hardcoded, never logged in full
• [ ] Secret scanning runs on snapshots before git commit
• [ ] LLM judge prompts use delimiters to prevent output hijacking
• [ ] Incus containers have no outbound network except API/Redis/KG
• [ ] Worktree mode blocks access to /etc, ~/.ssh, parent directories
• [ ] Diagnostics writes restricted to context.config.diagnostics_dir under worktree

Article V: Determinism Where Possible

• [ ] Syntax evaluator: deterministic ast.parse — no LLM involved
• [ ] Lint evaluator: deterministic ruff check — no LLM involved
• [ ] Trace evaluator: deterministic regex/AST trace parsing — no LLM involved
• [ ] File-exists evaluator: deterministic Path.exists() — no LLM involved
• [ ] LLM-as-judge used only for subjective quality (style, architecture, design)
• [ ] Snapshot serialization deterministic: sort_keys, ISO datetimes, base64 bytes
• [ ] Composite scoring formula documented and fixed (weighted average)

Article VI: Statistical Significance

• [ ] Minimum 3 trials per scenario for stochastic assessment
• [ ] Single-trial results never used for production go/no-go decisions
• [ ] Pass/fail decision considers consistency across trials (not just aggregate)
• [ ] Snapshot regression detection compares all trials, not just one
• [ ] Historical trends computed over minimum 10 runs before significance claimed

Article VII: Transparency and Traceability

• [ ] Every harness class traceable to at least one requirement via @traces
• [ ] Every scenario result includes: scenario_id, volk, git_commit, timestamp
• [ ] Diagnostic traces saved for all failed scenarios (including healing hops)
• [ ] Neo4j graph links :Run → :Scenario → :Volk for full provenance
• [ ] JSON history retained for 12 months minimum
• [ ] All evaluator results stored with name, score, pass/fail, details
• [ ] Self-healing loop produces auditable hop-by-hop record

Article VIII: Graceful Degradation

• [ ] Neo4j unavailable → JSON history still saved; harness continues
• [ ] Redis unavailable → scenarios requiring Redis fail gracefully; others continue
• [ ] Incus unavailable → automatic fallback to worktree mode
• [ ] Worktree creation fails → fallback to temp directory clone
• [ ] LLM API rate-limited → exponential backoff with max retries
• [ ] LLM API unavailable → judge evaluator returns failure; deterministic evaluators continue
• [ ] CXP dispatcher unavailable → healing loop uses fallback mode (logs + simulated retry)
• [ ] Partial service degradation visible in report and logs

Article IX: Dogfooding

• [ ] Harness evaluates volks on orchestrata-ultimate codebase itself
• [ ] Scenario inputs reference real code files in the repo
• [ ] Snapshot baselines derived from actual volk outputs on actual code
• [ ] Quality thresholds calibrated against real codebase standards
• [ ] Harness code subject to same lint/type-check standards as rest of codebase

Article X: No Main Working Directory Mutation

• [ ] Harness never writes to repository root outside harness/ and tests/volk_harness/
• [ ] Volk operations restricted to worktree directory
• [ ] Snapshot updates write to tests/volk_harness/snapshots/ (test directory)
• [ ] History writes to harness/history/ (harness directory)
• [ ] Diagnostics writes to harness/history/diagnostics/ (harness directory)
• [ ] Git worktree configured with remote.origin.push=NOTHING

checklists/requirements.md

The requirement list above comes from this file; 0 of 131 checkbox items are ticked.

Full spec text

Verbatim from specs/102-volk-test-harness/spec.md; headings are pushed down two levels.

Spec 102 — Individual Volk Test Harness

Metadata

Overview

What

The Individual Volk Test Harness is a scenario-driven evaluation system that exercises each volk (agent) individually through real LLM calls with real Redis/Neo4j infrastructure, operating on the orchestrata-ultimate codebase itself (dogfooding). It is an on-demand quality assurance tool, not part of the standard CI/CD test suite.

Why

Individual volks operate autonomously with significant decision-making latitude. Unlike traditional unit tests that validate deterministic functions, volk evaluation requires:

• Stochastic assessment: LLM outputs vary across runs; quality must be measured statistically
• Real infrastructure: Redis queues, Neo4j graph state, and LLM API interactions cannot be meaningfully mocked for agent evaluation
• Self-referential validation: The harness evaluates volks on their ability to improve the very codebase that defines them
• Regression detection: Snapshot comparisons across runs catch subtle degradation in agent quality
• Self-healing: When a volk fails a scenario, the harness can trigger a healing loop (max 3-hop depth) to autonomously diagnose and fix the root cause

Without this harness, volk quality is only measured indirectly through end-to-end system tests, making it impossible to attribute failures to specific agents.

How

1. Scenario Definition: Each volk has a directory of YAML scenario files (harness/scenarios/<volk_name>/) describing tasks, expected outcomes, and evaluation criteria
2. Setup Hooks: Python functions in harness/scenarios/<volk_name>/setup.py prepare the runtime environment (e.g., create temp files, seed Neo4j state)
3. Execution: For each scenario, the harness runs 3 independent trials using git worktrees for isolation (or Incus containers for full sandboxing)
4. Dual LLM Strategy:
   • DeepSeek v4-pro (deepseek-v4-pro) for reasoning-heavy scenarios (architecture review, code quality judgment, judge role)
   • DeepSeek v4-flash (deepseek-v4-flash) for fast execution scenarios (syntax checks, trace analysis, bulk operations)
5. Packet Injection: Scenarios can inject CXP packets into the kernel dispatcher or invoke volk methods directly
6. Evaluation Pipeline:
   • Deterministic evaluators: Syntax validation, lint checks (ruff), trace analysis, file existence checks
   • LLM-as-judge: For subjective quality (code style, architectural conformance, test quality)
7. Snapshot Comparison: Results compared against stored snapshots in tests/volk_harness/snapshots/; mismatches flagged as regressions
8. Self-Healing Loop: On failure, trigger up to 3 repair hops using CXP routing (derived from kernel/dispatcher.py)
9. History Tracking: Results stored in JSON (committed monthly) and Neo4j (updated every run) for trend analysis
10. CI Integration: GitLab CI pipeline (not GitHub Actions) for nightly batch runs

User Stories

• As a core developer, I want to run harness run qa so that I can validate QA volk's code review quality on a sample PR before deploying it to production.
• As an architect, I want to see historical trend graphs in Neo4j showing QA volk's false-positive rate over time, so that I can identify when a model update degraded its precision.
• As a CodeDirector, I want the harness to automatically retry with a healing loop when my generated code fails syntax checks, so that I don't have to manually intervene on transient issues.
• As a release manager, I want the harness to block releases when any volk's scenario pass rate drops below 90%, so that we don't ship regressions.
• As a QA engineer, I want deterministic evaluators to catch 100% of syntax and lint errors without LLM calls, so that expensive LLM evaluation is only used for subjective quality.
• As an infrastructure engineer, I want scenarios to run in git worktrees (or Incus containers) with real Redis/Neo4j, so that tests don't pollute the main working directory or share mutable state.

Terminology

Functional Requirements

FR-001 — Scenario Definition Format

Scenarios SHALL be defined in YAML files under harness/scenarios/<volk_name>/ with the following schema:

• id: Unique scenario identifier (e.g., qa-001)
• volk: Target volk name (e.g., qa, codedirector, architect)
• description: Human-readable scenario purpose
• mode: cxp_inject or direct_call
• input: Scenario-specific input data (structure varies by volk)
• setup_hook: Optional Python function path (e.g., setup:seed_neo4j_graph)
• evaluation: List of evaluator configurations
• llm_model: Override default LLM (v4-pro or v4-flash)
• timeout_seconds: Max execution time (default: 300)

FR-002 — Python Setup Hooks

Setup hooks SHALL be defined in harness/scenarios/<volk_name>/setup.py as async functions accepting a HarnessContext parameter. Hooks MAY create temporary files, seed Neo4j data, or configure Redis state. All resources created by hooks SHALL be automatically cleaned up via the isolation fixture mechanism.

FR-003 — Dual LLM Model Strategy

The harness SHALL support two LLM configurations:

• Default for reasoning/judge scenarios: deepseek-v4-pro with thinking mode enabled
• Default for execution/fast scenarios: deepseek-v4-flash with up to 2500 concurrency Scenario YAML MAY override the default via the llm_model field.

FR-004 — CXP Packet Injection Mode

In cxp_inject mode, the harness SHALL construct a CXP packet and enqueue it via the kernel dispatcher. The packet SHALL contain:

• intent: The operation to perform (e.g., review_code, generate_scaffold)
• payload: Volk-specific input data
• context: Session context including test_id for resource isolation The harness SHALL wait for the CXP response packet or timeout.

FR-005 — Direct Method Call Mode

In direct_call mode, the harness SHALL instantiate the target volk class and invoke the specified method directly. This mode SHALL bypass CXP routing and is intended for fast, deterministic evaluation of specific volk methods.

FR-006 — Three Trials Per Scenario

Each scenario SHALL execute exactly 3 independent trials. Trials SHALL use separate git worktrees (or Incus containers) to ensure no shared mutable state. Results SHALL be compared for consistency across trials.

FR-007 — Snapshot Comparison

After each scenario run, the harness SHALL serialize the output (file contents, traces, evaluation scores) to a text-based snapshot in tests/volk_harness/snapshots/<scenario_id>.snap. On subsequent runs, the harness SHALL compare outputs against stored snapshots and flag any differences as potential regressions. Snapshots SHALL be stored in git.

FR-008 — Deterministic Evaluators

The harness SHALL implement the following deterministic evaluators (no LLM calls):

• syntax_check: Parse Python files with ast.parse; fail on SyntaxError
• lint_check: Run ruff check on generated/modified files; fail on ERROR severity
• trace_validator: Parse execution traces for exception patterns; fail on unhandled exceptions
• file_exists: Verify expected output files were created
• type_check: Run mypy on generated code (optional, configurable)

FR-009 — LLM-as-Judge Evaluator

For subjective quality assessment, the harness SHALL support an llm_judge evaluator that:

• Sends the scenario output to DeepSeek v4-pro with a structured evaluation prompt
• Returns a score (0-100) and categorical feedback
• Uses a temperature of 0.0 for consistency
• Includes the scenario's rubric in the prompt

FR-010 — Volk-Specific Quality Criteria

Each volk SHALL have defined quality thresholds:

• QA Volk: Zero lint/syntax errors in reviewed code; test files conform to test review checklist (T1-T9); false-positive rate < 5%
• CodeDirector Volk: Generated code uses proper patterns; no try/except/pass blocks; proper exception handling and logging; functional style preferred (map/filter/reduce over imperative loops); all tests pass
• Architect Volk: Generated diagrams render without errors; architecture conforms to architecture/ standards; dependency rules respected

FR-011 — Self-Healing Loop

On scenario failure, the harness SHALL enter a self-healing loop:

1. Hop 1: Capture failure details (trace, evaluator output, snapshot diff) into a CXP diagnostic packet
2. Hop 2: Route diagnostic packet to the responsible volk (or a repair volk) via kernel/dispatcher.py routing logic
3. Hop 3: Apply the repair (code patch, config change, or retry with modified parameters)
4. Max Depth: 3 hops. If unresolved after 3 hops, mark scenario as FAILED and emit detailed diagnostics. The healing loop SHALL NOT modify the main working directory (only the worktree/container).

FR-012 — Command Whitelist for Safety

Volk executions within the harness SHALL be restricted to the following commands:

• ruff, python, git (no push), orchestrata-tools, black, isort, uv, pytest Any attempt to execute commands outside this whitelist SHALL fail immediately with a security error.

FR-013 — History Tracking (JSON)

After each run, the harness SHALL append results to harness/history/runs/<YYYY-MM>.json. Each entry SHALL include: scenario ID, volk name, trial number, pass/fail status, evaluator scores, LLM judge score, execution time, git commit hash, and timestamp.

FR-014 — History Tracking (Neo4j)

After each run, the harness SHALL sync results to Neo4j using Neo4jSyncer. The graph SHALL include:

• :Scenario nodes linked to :Volk nodes
• :Run nodes with properties for scores and timestamps
• :Regression nodes when snapshot diffs are detected
• Trend queries SHALL be available via Cypher (e.g., "Show QA volk pass rate over last 30 days")

FR-015 — GitLab CI Integration

The harness SHALL provide a .gitlab-ci.yml job (not GitHub Actions) for nightly batch execution:

• Trigger: Scheduled pipeline at 02:00 UTC
• Parallelism: Scenarios distributed across CI runners
• Artifacts: HTML report + JSON history + snapshot diffs
• Failure handling: Pipeline fails if any volk's overall pass rate < 90%

FR-016 — Harness CLI Interface

The harness SHALL expose a CLI via python -m orchestrata_ultimate.harness with subcommands:

• run <volk_name>: Run all scenarios for a volk
• run --scenario <scenario_id>: Run a specific scenario
• run --trials <n>: Override default 3 trials
• list: List all scenarios with last-run status
• snapshot update: Update stored snapshots to current outputs
• report: Generate HTML report from history
• trends: Query Neo4j for historical trend data

FR-017 — Resource Isolation

Each trial SHALL run in an isolated environment:

• Git worktree (default): Create a temporary worktree; clean up after trial
• Incus container (optional): Full sandboxing for scenarios that modify system state Redis DB isolation SHALL follow the existing worker-to-DB mapping (gw0→db0, gw1→db1). Neo4j resources SHALL be prefixed with test_id and cleaned up in teardown.

FR-018 — Timeout and Cancellation

Each scenario trial SHALL have a configurable timeout (default: 300s). On timeout:

1. Cancel the LLM call (if in progress)
2. Kill the worktree/container process
3. Mark trial as TIMEOUT
4. Do NOT enter self-healing loop for timeouts

FR-019 — Configuration File

The harness SHALL read configuration from harness/config.yaml (or harness/config.json) with:

• Default LLM model assignments per volk
• KG/Redis connection parameters (resolved from the active env profile via the RUL-003 raising resolvers — no localhost fallback)
• Snapshot directory path
• History directory path
• Command whitelist
• Self-healing loop max depth
• Timeout defaults

FR-020 — Parallel Execution

Multiple scenarios SHALL be runnable in parallel via the threadpool runner python scripts/run_tests.py (pytest-xdist -n/--dist is banned — it deadlocks this suite). Parallel execution SHALL respect Redis DB isolation and KG test_id prefixes.

Error Conditions

ERR-001 — Scenario YAML Schema Violation

Trigger: Scenario file missing required fields (id, volk, mode, evaluation). Effect: Scenario skipped; error logged; harness continues with remaining scenarios. Resolution: Fix YAML schema; re-run.

ERR-002 — Setup Hook Import Failure

Trigger: setup_hook references a non-existent function or module. Effect: Scenario marked FAILED; stack trace captured; no self-healing attempted for import errors. Resolution: Fix setup.py module path or function name.

ERR-003 — LLM API Rate Limit

Trigger: DeepSeek API returns 429 (rate limit exceeded). Effect: Trial paused with exponential backoff (max 5 retries, base 2s). If all retries exhausted, trial marked FAILED. Resolution: Reduce concurrency or upgrade API tier; retry manually.

ERR-004 — LLM API Authentication Failure

Trigger: Invalid or expired API key (401/403). Effect: All trials for this scenario fail immediately. Harness logs secure error (key masked). Resolution: Update DEEPSEEK_API_KEY in .env.

ERR-005 — Redis Connection Failure

Trigger: Redis container unreachable or misconfigured. Effect: Scenarios requiring Redis fail. Harness attempts auto-start via scripts/start_test_services.py once. Resolution: Verify the active env profile's Redis host/port; start containers via incus/redis-launch.sh.

ERR-006 — Neo4j Connection Failure

Trigger: Neo4j container unreachable or auth failure. Effect: History sync to Neo4j skipped (JSON history still saved). Real-time scenarios using Neo4j fail. Resolution: Verify the active env profile's KG_ROUTING_URI/KG_DATABASE; deploy the test node via incus/orchgraph-test-node-deploy.sh.

ERR-007 — Git Worktree Creation Failure

Trigger: Insufficient disk space, permission denied, or existing worktree conflict. Effect: Trial falls back to temporary directory clone. If fallback fails, trial marked FAILED. Resolution: Free disk space; clean old worktrees via git worktree prune.

ERR-008 — Snapshot Mismatch (Regression)

Trigger: Scenario output differs from stored snapshot. Effect: Trial marked FAILED with detailed diff. Regression node created in Neo4j. Resolution: Investigate diff. If intentional improvement, run harness snapshot update.

ERR-009 — Self-Healing Loop Exhaustion

Trigger: Scenario fails and self-healing loop reaches max 3 hops without resolution. Effect: Scenario marked FAILED. Full diagnostic trace (all 3 hops) saved to harness/history/diagnostics/. Resolution: Manual investigation of diagnostic trace; fix root cause in volk or scenario.

ERR-010 — Command Whitelist Violation

Trigger: Volk execution attempts to run a command outside the whitelist (e.g., rm -rf /, curl to external URL). Effect: Command blocked immediately. Trial marked FAILED with security flag. Incident logged. Resolution: Review volk prompt/instructions; whitelist violation indicates prompt injection or misalignment.

ERR-011 — Timeout During LLM Call

Trigger: LLM response exceeds timeout_seconds. Effect: LLM call cancelled. Trial marked TIMEOUT. Partial response (if any) discarded. Resolution: Increase timeout in scenario YAML; investigate LLM latency (v4-pro is slower than v4-flash).

ERR-012 — Incus Container Launch Failure

Trigger: Incus daemon not running, bridge misconfigured, or image unavailable. Effect: Falls back to git worktree mode. If worktree also fails, trial marked FAILED. Resolution: Start Incus daemon; verify bridge configuration in incus/README.md.

ERR-013 — Deterministic Evaluator Crash

Trigger: Bug in evaluator code (e.g., ast.parse on non-Python file, ruff config missing). Effect: Evaluator score marked as ERROR. Other evaluators continue. Trial may still pass if remaining evaluators succeed. Resolution: Fix evaluator code or scenario input.

ERR-014 — Snapshot File Corruption

Trigger: Stored snapshot file is not valid UTF-8 or has been manually edited to invalid format. Effect: Snapshot comparison skipped for this scenario; warning logged. Resolution: Delete corrupted snapshot; re-run to regenerate.

ERR-015 — Parallel Execution Resource Conflict

Trigger: Two trials share a Redis DB or Neo4j namespace due to test_id collision or worker misconfiguration. Effect: Non-deterministic failures in both trials. Harness detects collision via pre-flight check and retries with new test_id. Resolution: Verify worker-to-DB mapping; ensure UUID-based test_id generation.

API Contracts

HarnessContext

@dataclass
class HarnessContext:
    """Runtime context passed to setup hooks and evaluators."""
    scenario_id: str
    volk_name: str
    trial_number: int
    test_id: str  # UUID for resource isolation
    worktree_path: Path
    redis_db: int  # Worker-isolated Redis DB number
    neo4j_prefix: str  # test_id prefix for Neo4j resources
    config: HarnessConfig
    logger: logging.Logger

HarnessConfig

@dataclass
class HarnessConfig:
    """Harness configuration loaded from harness/config.yaml."""
    default_llm: str = "deepseek-v4-flash"
    reasoning_llm: str = "deepseek-v4-pro"
    snapshot_dir: Path = Path("tests/volk_harness/snapshots")
    history_dir: Path = Path("harness/history/runs")
    diagnostics_dir: Path = Path("harness/history/diagnostics")
    command_whitelist: list[str] = field(default_factory=lambda: [
        "ruff", "python", "git", "orchestrata-tools", "black", "isort", "uv", "pytest"
    ])
    max_healing_hops: int = 3
    default_timeout: int = 300
    trials_per_scenario: int = 3
    redis_host: str = "localhost"
    redis_port: int = 6379
    neo4j_uri: str = "bolt://localhost:7687"

Scenario Definition (Pydantic Model)

class Scenario(BaseModel):
    id: str
    volk: str
    description: str
    mode: Literal["cxp_inject", "direct_call"]
    input: dict[str, Any]
    setup_hook: str | None = None
    evaluation: list[EvaluatorConfig]
    llm_model: str | None = None
    timeout_seconds: int = 300

class EvaluatorConfig(BaseModel):
    name: str  # e.g., "syntax_check", "llm_judge"
    params: dict[str, Any] = Field(default_factory=dict)
    weight: float = 1.0  # For composite scoring
    threshold: float = 0.0  # Minimum score to pass

HarnessRunner

class HarnessRunner:
    """Main entry point for executing scenarios."""

    def __init__(self, config: HarnessConfig) -> None:
        ...

    async def run_scenario(
        self,
        scenario: Scenario,
        trials: int = 3,
    ) -> ScenarioResult:
        """Run a scenario for N trials and return aggregated results."""
        ...

    async def run_volk(
        self,
        volk_name: str,
        trials: int = 3,
    ) -> list[ScenarioResult]:
        """Run all scenarios for a given volk."""
        ...

    def list_scenarios(self, volk_name: str | None = None) -> list[Scenario]:
        """List all loaded scenarios, optionally filtered by volk."""
        ...

ScenarioResult

@dataclass
class ScenarioResult:
    scenario_id: str
    volk_name: str
    trials: list[TrialResult]
    aggregate_score: float  # Weighted average across evaluators
    passed: bool  # True if all trials pass and no snapshot regressions
    execution_time_ms: int
    git_commit: str
    timestamp: datetime

@dataclass
class TrialResult:
    trial_number: int
    passed: bool
    evaluator_results: list[EvaluatorResult]
    llm_judge_score: float | None
    snapshot_diff: str | None  # None if no diff or no snapshot
    healing_hops_used: int
    execution_time_ms: int
    error: str | None

@dataclass
class EvaluatorResult:
    name: str
    passed: bool
    score: float
    details: str

Evaluator Interface

class Evaluator(ABC):
    """Base class for deterministic and LLM evaluators."""

    @abstractmethod
    async def evaluate(
        self,
        output: Any,
        context: HarnessContext,
        config: EvaluatorConfig,
    ) -> EvaluatorResult:
        ...

class SyntaxEvaluator(Evaluator): ...
class LintEvaluator(Evaluator): ...
class TraceEvaluator(Evaluator): ...
class LLMJudgeEvaluator(Evaluator): ...

SelfHealingLoop

class SelfHealingLoop:
    """Manages autonomous repair attempts on scenario failure."""

    def __init__(
        self,
        dispatcher: CXP_Dispatcher,  # From kernel/dispatcher.py
        max_hops: int = 3,
    ) -> None:
        ...

    async def heal(
        self,
        failure: TrialResult,
        context: HarnessContext,
    ) -> HealingResult:
        """Attempt to heal a failed trial. Returns result after max_hops or success."""
        ...

@dataclass
class HealingResult:
    success: bool
    hops_used: int
    diagnostics: list[HopDiagnostic]
    final_trial_result: TrialResult | None

@dataclass
class HopDiagnostic:
    hop_number: int
    action: str
    packet_sent: CXPPacket
    response: CXPPacket | None
    error: str | None

Security Requirements

SEC-001 — Command Execution Whitelist

The harness SHALL enforce a strict command whitelist (FR-012). Any subprocess invocation by a volk during harness execution MUST be validated against this list. Violations SHALL be logged as security incidents.

SEC-002 — API Key Isolation

DeepSeek API keys SHALL be read from environment variables (.env) only. Keys SHALL never be logged, stored in scenario files, or committed to git. The harness SHALL mask API keys in all log output.

SEC-003 — Network Isolation (Incus Mode)

When running in Incus container mode, containers SHALL have no outbound network access except to:

• DeepSeek API endpoint
• Redis container (internal bridge)
• Neo4j container (internal bridge) No access to external git repos, package registries (unless explicitly mirrored), or the host filesystem outside the worktree.

SEC-004 — Git Push Prevention

The git command whitelist SHALL exclude git push, git push --force, and any git operation that modifies remote state. Harness worktrees SHALL be configured with remote.origin.push= NOTHING.

SEC-005 — File System Sandboxing

Volk operations in the harness SHALL be restricted to the worktree directory. Access to paths outside the worktree (e.g., /etc, ~/.ssh, parent directories) SHALL be blocked via chroot (Incus) or path validation (worktree mode).

SEC-006 — Secret Scanning in Snapshots

Before committing snapshots to git, the harness SHALL scan for potential secrets (API keys, passwords, tokens) using git-secrets or equivalent. Matches SHALL block the commit and alert the operator.

SEC-007 — LLM Prompt Injection Mitigation

LLM-as-judge prompts SHALL be constructed with clear delimiters and explicit instructions to prevent the evaluated output from hijacking the judge's reasoning. System prompts SHALL be templated and validated.

Dependencies

DEP-001 — DeepSeek API

Requires active DeepSeek API account with access to deepseek-v4-pro and deepseek-v4-flash models. API key via DEEPSEEK_API_KEY environment variable.

DEP-002 — Redis (Incus Container)

Redis container must be running for scenarios requiring queue/state operations. Auto-start attempted via scripts/start_test_services.py. Configuration from the active env profile.

DEP-003 — Operational KG (Incus Memgraph node)

The operational KG (Memgraph, orchkg database) must be running for history sync and graph-based scenarios. Configuration from the active env profile.

DEP-004 — Git Worktree Support

Git version >= 2.25 required for worktree operations. git worktree command must be available.

DEP-005 — Incus (Optional)

Incus daemon required only for full container sandboxing. Worktree mode does not require Incus.

DEP-006 — Python 3.14+

Harness code uses Python 3.14 features (e.g., improved typing syntax, free-threading support).

DEP-007 — Existing Volk Implementations

The harness evaluates volks in src/orchestrata_ultimate/agents/volks/. Each volk must implement the expected interface for its scenarios.

DEP-008 — CXP Kernel Dispatcher

src/orchestrata_ultimate/kernel/dispatcher.py must be operational for cxp_inject mode and self-healing loop routing.

DEP-009 — SessionController

src/orchestrata_ultimate/session/session.py for session tracking and changelog generation during harness runs.

DEP-010 — ruff, black, mypy

Required for deterministic evaluators. Must be installed in the virtual environment (uv sync --dev).

Testing Requirements

TST-001 — Harness Self-Tests

The harness itself SHALL have tests in tests/harness/ covering:

• Scenario YAML parsing and validation
• Setup hook execution and cleanup
• Evaluator correctness (e.g., syntax evaluator on valid/invalid Python)
• Snapshot serialization and comparison
• Self-healing loop behavior (mocked CXP dispatcher)
• Command whitelist enforcement
• Timeout handling

TST-002 — Mock LLM Mode

The harness SHALL support a --mock-llm flag (or config option) that replaces real LLM calls with deterministic responses for fast testing of harness infrastructure. Mock responses SHALL be defined in tests/harness/fixtures/mock_llm_responses.yaml.

TST-003 — Integration Tests

Integration tests SHALL verify end-to-end harness execution with:

• A minimal "echo volk" that returns its input
• Scenarios for all execution modes (cxp_inject, direct_call)
• All evaluator types
• Snapshot creation and regression detection

TST-004 — Parallel Safety

Harness tests SHALL pass under the threadpool runner python scripts/run_tests.py (pytest-xdist -n/--dist is banned). No session-scope fixtures. No global mutable state. Use test_id for all resource names.

TST-005 — Incus Fallback Test

Test that worktree mode is correctly used when Incus is unavailable.

TST-006 — Deterministic Evaluator Accuracy

Each deterministic evaluator SHALL be tested with:

• 100% true-positive rate (catches all actual errors)
• Zero false-positives on valid input
• Edge cases (empty files, binary files, non-Python files)

TST-007 — Snapshot Format Stability

Snapshots SHALL use a stable, versioned text format. Tests SHALL verify backward compatibility with existing snapshot files.

TST-008 — Performance Benchmarks

The harness SHALL include performance tests measuring:

• Scenario execution time (p50, p95, p99)
• LLM API latency (per model)
• Worktree creation/teardown time
• Neo4j sync throughput Benchmarks SHALL fail if p95 exceeds 2x baseline.

TST-009 — Security Test Suite

Security tests SHALL verify:

• Command whitelist blocks unauthorized commands
• API keys are masked in logs
• Path traversal attempts are blocked
• LLM judge prompts cannot be hijacked by malicious output

TST-010 — Failure Injection Tests

The harness SHALL be tested with simulated failures:

• Redis connection drop mid-scenario
• Neo4j write failure during history sync
• LLM API 500 error
• Git worktree corruption
• Timeout during long-running LLM call

Open Questions

1. Should the harness support evaluating volk-to-volk interactions (multi-agent scenarios), or strictly individual volks?
2. Should snapshot updates be fully automated (on CI pass), or require human approval?
3. What is the retention policy for JSON history files? (Propose: 12 months local, indefinite in Neo4j)
4. Should the self-healing loop be able to modify the volk's own source code, or only the worktree?
5. Do we need a dedicated "Harness Volk" to manage scenario execution, or should it be imperative code?

Related

• RTM page for 102-volk-test-harness
• All specs
• RTM overview
• Gates