Orchestrata Ultimate › Plans › spec-audit › reverify-2026-10-18 › K-harness-newer

Cluster K — harness/newer REVERIFY (SPEC-102/103/104/105)

Source: plans/spec-audit/reverify-2026-10-18/K-harness-newer.md at 8a2cb42ad · last changed 2026-10-18 · first committed 2026-10-18 · 277 lines.

Prose pending.

Plan text

Cluster K — harness/newer REVERIFY (SPEC-102/103/104/105)

Re-verification against current components branch (commit de9e996f2, 337 commits ahead of the session-195 audits). All evidence regenerated fresh via the AST toolkit (scripts/serialize_codebase_ast.py {symbols,refs,source}); no stale verdict trusted. Prior audit baseline: plans/spec-audit/K-harness-greenfield.md (session 195) and 00-MASTER-PLAN.md §2/§6 (S7 Medic).

Key deltas vs. the 3-month-old audit (high-level):

• SPEC-102 — healing loop was structurally expanded (482 lines, 3-hop CXP dispatch, diagnostics persistence, worktree path-safety) but the two defects the old audit flagged are both still present (payload-overwrite + fabricated-pass _repaired_trial). Scenarios now live at repo-root harness/ (not inside the package). History moved Neo4j→graph.
• SPEC-103 — still DONE and still fully wired (FR-009 pipeline confirmed at new line numbers; enable_tool_refactor is now a config field, not a module constant).
• SPEC-104 — still DONE; gained a requirements_parser.py helper (Prolog facts) and grew to 38 tests.
• SPEC-105 — still DONE; the relationships.py divergence is now resolved in the spec text (REQ-105-005), and FR-008 SHACL is now implemented + specced.

SPEC-102 Individual Volk Test Harness — PARTIAL

Specifies: A scenario-driven, on-demand (not CI) harness that exercises each volk through real LLM + Redis/graph infra, with deterministic + LLM-judge evaluators, git-worktree/Incus isolation, 3-trial runs, snapshot regression, a 3-hop self-healing loop, JSON+graph history, and a CLI. Target src/orchestrata_ultimate/harness/ (+ repo-root harness/ scenarios).

Verified:

• ✅ FR-001/002 scenario model + setup hooks — harness/models.py:42 Scenario (pydantic); harness/scenario.py:182 run_setup_hook, :99 load_scenarios_for_volk. Shipped scenarios present at repo-root harness/scenarios/{qa,architect,codedirector}/scenario_001.yaml + setup.py each (loader default Path("harness/scenarios") resolves there from repo root).
• ✅ FR-003 dual LLM — harness/models.py HarnessConfig.default_llm/reasoning_llm.
• ✅ FR-004/005 executors — harness/executor.py:40 CXPacketExecutor, :89 DirectCallExecutor.
• ✅ FR-006 3 trials — harness/runner.py:106 for trial_number in range(1, trials + 1).
• ✅ FR-007 snapshots — harness/snapshots.py SnapshotManager; runner.py:331 _compare_snapshot.
• ⚠️ FR-008 deterministic evaluators — registry (harness/evaluators/__init__.py:57-78) has 5: SyntaxEvaluator, LintEvaluator, TraceEvaluator, FileExistsEvaluator, LLMJudgeEvaluator. The optional type_check/mypy evaluator is still absent (no type_check symbol/module).
• ✅ FR-009 LLM-judge — harness/evaluators/llm_judge.py LLMJudgeEvaluator.
• ❌ FR-011 self-healing loop is still non-operational. Structural work landed (3-hop CXP dispatch via _run_hop healing.py:222, _has_valid_strategy :306, diagnostics persistence :386, worktree path-safety :443/:463), but the two defects persist:
  • healing.py:274 still overwrites the volk's reply with the outgoing packet payload (response_dict["payload"] = packet.payload) before _extract_strategy (:55) reads it. The outgoing payload for hops 1/2/3 contains action ∈ {diagnostic, apply_repair, escalate}, never "retry"/"patch", so _extract_strategy can never match → the CXP path always exhausts to success=False (3 hops). The comment at :272-273 rationalizes it ("the receiving volk is expected to echo back") but the code deletes that echo.
  • healing.py:313-323 _repaired_trial still fabricates success: docstring literally says """Return a new TrialResult simulating a successful repair.""" and returns passed=True carrying the original failing evaluator_results (:319) and snapshot_diff (:321).
  • healing.py:374 _heal_fallback still fabricates success ("we optimistically assume the retry succeeds") after hop 2. In the runner path it is only reachable via direct dispatcher=None construction (the runner's lazy dispatcher property runner.py:73-77 always returns a real MessageDispatcher), but the fabricated-pass helper is shared.
  • Net: a repair is never re-executed + re-evaluated; "healed" success is never verified.
• ✅ FR-012 whitelist — harness/executor.py:180 enforce_whitelist, :26 SecurityError.
• ⚠️ FR-013/014 history — harness/history.py:113 JSONHistoryTracker + :211 GraphHistoryTracker (graph failures non-fatal, runner.py:60-66). Divergence: spec FR-014 + API-contract still say Neo4jSyncer/neo4j_uri, code targets the orchgraph (Memgraph) via core.graph_factory.resolve_uri.
• ✅ FR-015 GitLab CI — .gitlab-ci.yml present at repo root.
• ✅ FR-016 CLI — harness/cli.py 6 subparsers (run/list/snapshot/report/trends/config, :354-389); __main__.py present.
• ✅ FR-017 isolation — harness/isolation.py:34 WorktreeIsolation, :137 IncusIsolation.
• ✅ FR-018 timeouts — asyncio.wait_for on setup/execute/evaluators; _run_hop HOP_TIMEOUT_SECONDS=30.
• ✅ FR-019 config — harness/config.py:23 load_config (YAML + env overlay, RUL-003 raising resolvers, no localhost fallback). Repo-root harness/config.yaml present.
• ⚠️ FR-020 parallel — runner.py:447-456 run_volk still runs scenarios sequentially; but the spec text was already re-worded (spec.md:210-211) to require the test suite be parallel via the threadpool runner (xdist banned). Reconciled — no code gap.
• ❌ TST-002 --mock-llm still dead config. harness/config.yaml:21 declares mock_llm: false; HarnessConfig (models.py:116-158) has no mock_llm field; _apply_yaml (config.py:67) logs "Unknown config key: mock_llm" and drops it; no --mock-llm CLI flag; no mock_llm_responses.yaml fixture. (symbols --pattern mock_llm → 0.)
• ⚠️ SPEC-102 still not registered in specs/SPEC_INVENTORY.md (103/104/105 are, at lines 28-30; 102 absent).
• ⚠️ Spec-text divergences (stale API-contract block): spec.md:316-333 shows redis_host: str = "localhost"/neo4j_uri: bolt://…; code uses redis_host = resolve_redis_host(), graph_uri = resolve_uri() (models.py:150-153). Spec HarnessContext.neo4j_prefix → code graph_prefix (models.py:110). FR-019 prose already matches code (raising resolvers); the embedded dataclass blocks are the stale part.

Remaining work / plan:

• Scope: (1) Make self-healing actually heal — after a repair hop, re-run the scenario stage + evaluators in the same worktree and mark passed=True only when re-evaluation passes; fix healing.py:274 so _extract_strategy reads the volk's reply payload, not the outgoing one; stop _repaired_trial/_heal_fallback from fabricating success. (2) Wire-or-delete the dead mock_llm config (add mock_llm to HarnessConfig + CLI flag + fixture, or drop the YAML key and TST-002). (3) Optional: add the type_check/mypy evaluator. (4) Register SPEC-102 in SPEC_INVENTORY.md. (5) S8-Scribe doc updates: FR-014/API-contract Neo4j→graph + localhost→resolvers.
• Target files: src/orchestrata_ultimate/harness/healing.py, harness/runner.py (re-verify hook), harness/models.py + harness/cli.py + harness/config.py (mock-llm), new harness/evaluators/type_check.py, specs/SPEC_INVENTORY.md, specs/102-volk-test-harness/spec.md (API-contract block).
• Approach: Thread the Scenario+Executor into SelfHealingLoop.heal (or return a "re-run requested" signal to HarnessRunner) so a hop claiming repair triggers a genuine re-execution + evaluator pass before success is declared. The mock-llm and type_check items are small and self-contained.
• Dependencies: couples to kernel MessageDispatcher/CXP (SPEC-004) and to repair-capable volks (agents/volks/); graph history already wired to orchgraph. Open spec question #4 (may a repair volk edit source, or only the worktree?) still unresolved.
• Effort: M (0.5–2d) — healing re-verify is the bulk; the rest is S. Risk: med — real self-healing needs a live repair-capable volk + real CXP round-trip to test meaningfully; without it the loop stays a stub and must be honestly documented, not report fake successes.

Evidence log:

• symbols --pattern SelfHealingLoop → 10 symbols (heal :126, _run_hop :222, _has_valid_strategy :306, _repaired_trial :313, _heal_fallback :331, _save_diagnostics :386, _resolve_safe_path :443, _assert_under_worktree :463).
• source harness.healing --no-daemon → full 482-line module; pinned response_dict["payload"] = packet.payload at :274, """...simulating a successful repair.""" :315, passed=True :318, evaluator_results=failure.evaluator_results :319, snapshot_diff=failure.snapshot_diff :321, "optimistically assume the retry succeeds" :374.
• symbols --pattern mock_llm → 0 symbol(s).; grep -n mock harness/config.yaml → 21:mock_llm: false; source harness.config --no-daemon → _apply_yaml logs "Unknown config key: %s" (:75).
• source harness.models --no-daemon → HarnessConfig (:116-158) fields — no mock_llm; HarnessContext.graph_prefix (:110).
• ls harness/evaluators/ → base, file_exists, lint, llm_judge, syntax, trace, __init__ (no type_check).
• source harness.history --no-daemon → JSONHistoryTracker :113, GraphHistoryTracker :211.
• source harness.runner --no-daemon → 3-trial loop :106, run_volk sequential :447-456, _maybe_self_heal :351 (constructs SelfHealingLoop(dispatcher=self.dispatcher)), graph non-fatal :60-66.
• find harness -type f → repo-root harness/{config.yaml, scenarios/{qa,architect,codedirector}/ {scenario_001.yaml,setup.py}}.
• grep -n '102' specs/SPEC_INVENTORY.md → absent (103/104/105 at :28-30).
• ls src/orchestrata_ultimate/harness/__main__.py → present (FR-016 python -m …).

SPEC-103 Tool-Calling Code Refactoring — DONE

Specifies: An LLM-driven (DeepSeek deepseek-v4-pro) tool-calling layer that refactors an existing codebase in place — OBSERVE → MUTATE → VERIFY via a sandboxed tool registry, a tool-use loop, a refactoring system prompt, and refactor()/arefactor(); plus provider-protocol tool fields, DeepSeek adapter tool support, and FR-009 REFACTOR-intent pipeline routing.

Verified:

• ✅ FR-001 protocol fields — core/protocols/providers.py:81 Message (tool_calls, tool_call_id, name), :96 ProviderRequest (tools, tool_choice), :118 ProviderResponse (tool_calls), :200 RawProviderResponse (tool_calls); all default None/empty (backward-compatible).
• ✅ FR-002 DeepSeek adapter — providers/adapters/deepseek_adapter.py:56-57 serialises assistant tool_calls into the API payload, :88-94 parses message.tool_calls into the raw response; tools/tool_choice forwarded via parameters.
• ✅ FR-003/004 sandboxed registry + 7 tools — refactor/tools.py:38 RefactorTools; _resolve (:49) rejects path escape; read_file/list_dir/grep/ast_outline/edit_file/write_file/run_check (:56-174); dispatch (:182) never raises (returns ERROR: …); TOOL_SCHEMAS (:212); _CHECK_WHITELIST (:25).
• ✅ FR-005/006/007 loop + prompt + entry point — refactor/agent.py:28 REFACTOR_SYSTEM_PROMPT, :56 RefactorResult, :73 async arefactor, :158 sync refactor (never raises → status="error"); loop for i in range(max_iterations) :105, dispatches via tools.dispatch :129, echoes reasoning_content :119.
• ✅ FR-008 observability — agent.py logs each tool call/turn via core.logging.
• ✅ FR-009 pipeline wiring — DONE. orchestrator/intent_broker.py:82 enable_tool_refactor: bool = field(default=True) (config field, hence symbols shows 0 top-level hits), routing guard :187-194 (self._enable_tool_refactor and … IntentType.REFACTOR → _handle_refactor), _handle_refactor :381 imports+awaits arefactor :408. refs arefactor → 1 def, 3 imports, 10 uses incl. intent_broker.py:407.
• ✅ call_with_tools — llm/client.py:1030 (used by agent.py and agents/volks/base.py).
• ✅ CLI — scripts/refactor.py present.

Remaining work / plan: None required for the capability. Optional (both S, low risk, not blocking): (a) flip SPEC-103 status Draft→Implemented + check DoD; (b) add the on-demand real-DeepSeek integration test (TST-103-004; needs a live DEEPSEEK_API_KEY).

Evidence log:

• symbols --pattern arefactor → refactor/agent.py:73 + intent_broker/tests uses.
• refs arefactor → DEF agent.py:73, IMP intent_broker.py:391, USE intent_broker.py:407.
• symbols --pattern _handle_refactor → intent_broker.py:380.
• source orchestrator.intent_broker --no-daemon → enable_tool_refactor :82, routing :187-194, _handle_refactor :381, await arefactor :408.
• source core.protocols.providers --no-daemon → Message :81, ProviderRequest :96, ProviderResponse :118, RawProviderResponse :200 (all tool fields + @traces REQ-103-001/002/003).
• source providers.adapters.deepseek_adapter --no-daemon → tool_calls serialise :56-57 / parse :88-94.
• source refactor.tools --no-daemon → 7 tools + dispatch (never raises) + TOOL_SCHEMAS + whitelist.
• source refactor.agent --no-daemon → prompt :28, result :56, loop :105, dispatch :129.

SPEC-104 OpenSpec Format Support — DONE

Specifies: A self-contained orchestrata_ultimate.openspec module that detects a project's spec folder + classifies its format (openspec/orchestrata/unknown), parses/renders/validates OpenSpec markdown, edits specs, and scaffolds change proposals — all writes sandboxed to the detected root.

Verified:

• ✅ FR-001/002 detect + recognize — openspec/detect.py:50 detect_spec_folder (never raises), :27 is_openspec (case-sensitive markers; delta or Requirement+Scenario).
• ✅ FR-003 parse — openspec/parse.py:81 parse_spec, delta parse, :109 validate (reports violations, no raise).
• ✅ FR-004/005 render/edit — openspec/render.py:28 render_spec (+ delta/proposal/tasks renderers); openspec/repo.py:29 OpenSpecRepo with capabilities (:50), read_capability (:61), write_capability (:68), add_requirement (:76), create_change (:84, refuses overwrite).
• ✅ FR-006/007 validate + sandbox — validate() returns violation list; repo._safe (:41) mirrors the SPEC-103 path-traversal sandbox.
• ✅ Data models — openspec/models.py (OpenSpecDoc/Requirement/Scenario/Delta/SpecFolder); public re-exports in __init__.py.
• ✅ openspec/requirements_parser.py (new helper: spec checklists → Prolog facts) + 38 tests (tests/openspec/test_openspec.py = 19, test_requirements_parser.py = 19).
• refs detect_spec_folder / refs OpenSpecRepo → 1 def + __init__ import + tests only, 0 production src uses — expected: library capability, pipeline use is optional per spec ("the refactor agent MAY invoke these"). Not orphaned.

Remaining work / plan: None required. Optional (S, low risk): flip status Draft→Implemented.

Evidence log:

• symbols --pattern detect_spec_folder|is_openspec|OpenSpecRepo|parse_spec|render_spec|validate → detect.py:50/:27, repo.py:29, parse.py:81/:109, render.py:28.
• refs detect_spec_folder → DEF + IMP __init__.py + tests USE (0 src uses).
• refs OpenSpecRepo --no-daemon → 1 def, 2 imports, 1 test use (0 src uses).
• source openspec.requirements_parser --no-daemon → parse_requirements :68, run :117 (Prolog facts).
• grep -rc 'def test_' tests/openspec/ → 19 + 19.

SPEC-105 Ontology-Driven Domain Model Generation — DONE

Specifies: A skill that reads an OWL/RDFS Turtle ontology → canonical DomainModel IR → deterministic Jinja2 codegen (screaming-architecture / modular monolith) + bidirectional IR↔ontology round-trip, generating a temporal, insert-only (bitemporal, RDF-star) Stardog repository + SHACL constraints. Target src/orchestrata_ultimate/domaingen/.

Verified:

• ✅ FR-001 ontology→IR — domaingen/reader.py:190 parse_ontology (rdflib; classes→Entity, datatype/object props→Field/Relationship, subClassOf→parent, functional→single, minCardinality→required).
• ✅ FR-002/003 IR→ontology + round-trip — domaingen/ttl.py:101 render_ontology (deterministic).
• ✅ FR-004/006 IR→code deterministic — domaingen/generator.py:90 generate_code; _topo_sort :60; _safe sandbox :126; templates entities.py.j2/init.py.j2/repository.py.j2.
• ✅ FR-005 centralized logging/exceptions — repository.py.j2 imports core.logging.get_logger (:19-21) + RepositoryError(BaseError) (:32).
• ✅ FR-006 temporal insert-only Stardog repo — repository.py.j2 build_insert (:53), build_tombstone (:69), build_as_of (:81), build_as_recorded (:94), build_as_valid (:104), supersede (:154), PREDICATES map (:131), _iri injection guard (:50). domaingen/stardog.py:27 StardogConnection (SPARQL-over-HTTP).
• ✅ FR-007 add-a-node bidirectional regen — generate_code + render_ontology share the IR.
• ✅ FR-008 SHACL — now implemented + specced. domaingen/constraints.py:26 generate_shacl; generator.py:118-119 emits constraints.ttl alongside the domain model.
• ✅ SEC-105-003 — generator.py:30-39 _FORBIDDEN_SPARQL (DELETE/DROP/CLEAR variants) generation-time guard.
• ✅ CLI — scripts/domaingen.py (parse → generate / --render-ttl).
• ✅ Divergence resolved. The prior audit's "relationships.py divergence" is now in the spec: REQ-105-005 (spec.md:138) states relationships are attributes + repository PREDICATES map, "not a separate relationships.py". Templates confirm no relationships.py.j2. → no remaining divergence.
• Tests: tests/domaingen/test_domaingen.py = 25, test_reader.py = 15 (40 total). DoD nearly all [x]; only the live-Stardog round-trip is [ ] (explicitly deferred — no Stardog server in this env).

Remaining work / plan: No code work required. The only open item is the on-demand live-Stardog integration test (already scoped as deferred in the DoD; unit tests assert the generated SPARQL/HTTP shape). Optional (S): flip status In Review→Implemented.

Evidence log:

• symbols --pattern parse_ontology|render_ontology|generate_code|generate_shacl|StardogConnection → reader.py:190, ttl.py:101, generator.py:90, constraints.py:26, stardog.py:27.
• source domaingen.generator --no-daemon → _FORBIDDEN_SPARQL :30-39, _topo_sort :60, generate_code :90, constraints.ttl :118-119, _safe :126.
• grep -n 'build_insert|supersede|build_as_of|build_tombstone|txFrom|PREDICATES|get_logger|RepositoryError' templates/repository.py.j2 → all present; INSERT DATA only, no UPDATE/DELETE.
• ls domaingen/templates/ → entities.py.j2, init.py.j2, repository.py.j2 (no relationships.py.j2).
• grep -rc 'def test_' tests/domaingen/ → 25 + 15.
• Read specs/105-ontology-domain-model/spec.md:138 (REQ-105-005 no-relationships.py text) + DoD [x] rows (:215-221).

Cluster summary table

Only SPEC-102 has real remaining code work (self-healing re-verify + mock-llm wire-or-delete; type_check evaluator optional). 103/104/105 are complete; 105's former relationships.py divergence is now resolved in the spec text, leaving only the deferred live-Stardog integration test.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks