Orchestrata Ultimate › Plans › spec-audit › K-harness-greenfield

Cluster K — Test harness & greenfield capabilities (SPEC-102/103/104/105)

Source: plans/spec-audit/K-harness-greenfield.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-04 · 218 lines.

Prose pending.

Plan text

Cluster K — Test harness & greenfield capabilities (SPEC-102/103/104/105)

Investigation agent K. READ-ONLY audit against actual code in src/orchestrata_ultimate/, per plans/spec-audit/_METHOD.md. Checklist - [ ] boxes ignored (stale); judged by code. These four specs are NOT in docs/spec-gap-register.md (which covers 20 core specs), so this is a fresh audit. Two design docs exist and were cross-checked against code: docs/capabilities-refactor-openspec.md (103/104) and docs/ontology-and-codegen.md (105).

SPEC-102 Individual Volk Test Harness — PARTIAL

• Specifies: A scenario-driven, on-demand (not CI) evaluation harness that exercises each volk through real LLM + Redis/Neo4j infra, with deterministic + LLM-judge evaluators, git-worktree/ Incus isolation, 3-trial runs, snapshot regression, a 3-hop self-healing loop, JSON+Neo4j history, and a CLI. Target src/orchestrata_ultimate/harness/.
• Verified: The infrastructure is heavily and genuinely built (218 test functions in tests/harness/, 20 test modules). Component-by-component:
  • ✅ FR-001 scenario YAML model — harness/models.py:30 Scenario (pydantic); real scenarios at harness/scenarios/{qa,codedirector,architect}/scenario_001.yaml (one per volk).
  • ✅ FR-002 setup hooks — harness/scenario.py:run_setup_hook; per-volk setup.py files present.
  • ✅ FR-003 dual LLM — models.py:106-107 default_llm/reasoning_llm.
  • ✅ FR-004 CXP inject — harness/executor.py:36 CXPacketExecutor (real MessageDispatcher).
  • ✅ FR-005 direct call — executor.py:79 DirectCallExecutor (dynamic volk import + method call).
  • ✅ FR-006 3 trials — runner.py:94 loops range(1, trials+1).
  • ✅ FR-007 snapshots — harness/snapshots.py SnapshotManager; baseline auto-save (runner.py:307).
  • ⚠️ FR-008 deterministic evaluators — evaluators/ has syntax, lint, trace, file_exists. The type_check/mypy evaluator (FR-008, optional) is absent (grep type_check → none).
  • ✅ FR-009 LLM-judge — evaluators/llm_judge.py LLMJudgeEvaluator.
  • ✅ FR-012 command whitelist — executor.py:167 enforce_whitelist + SecurityError.
  • ✅ FR-013/014 history — harness/history.py JSONHistoryTracker + Neo4jHistoryTracker (Neo4j failures non-fatal, runner.py:62-69).
  • ✅ FR-015 GitLab CI — .gitlab-ci.yml present at repo root (not GitHub Actions).
  • ✅ FR-016 CLI — harness/cli.py subparsers: run, list, snapshot, report, trends, config (cli.py:332-364); __main__.py present.
  • ✅ FR-017 isolation — isolation.py:31 WorktreeIsolation + :130 IncusIsolation (real incus launch --ephemeral + disk device); Redis DB / Neo4j-prefix helpers present.
  • ✅ FR-018 timeouts — asyncio.wait_for on setup / execute / evaluators.
  • ✅ FR-019 config — harness/config.py load_config (YAML + env overlay); harness/config.yaml.
  • ❌ FR-011 self-healing loop is a non-operational stub. healing.py:279 _repaired_trial returns a new TrialResult(passed=True) carrying the original (failing) evaluator_results and snapshot_diff — it never re-executes the scenario or re-runs evaluators, so "healed" success is fabricated, not verified. Worse, healing.py:242 overwrites the volk's reply with the outgoing packet payload (response_dict["payload"] = packet.payload) before _extract_strategy reads it, so a volk's actual retry/patch strategy can never be seen — the CXP path therefore almost always exhausts to success=False, and the no-dispatcher fallback (_heal_fallback) unconditionally returns success=True after hop 2. Structurally complete, functionally a placeholder. (This is exactly the "false-confidence" pattern flagged in the project auto-memory.)
  • ❌ TST-002 --mock-llm mode is dead config. harness/config.yaml declares mock_llm: false, but HarnessConfig (models.py:102) has no mock_llm field, so _apply_yaml logs "Unknown config key: mock_llm" and drops it; there is no --mock-llm CLI flag and no tests/harness/fixtures/mock_llm_responses.yaml. Harness self-tests instead avoid LLM paths.
  • ⚠️ FR-020 parallel execution — run_volk (runner.py:417) runs scenarios sequentially; there is no multiprocessing pool. The spec proposes pytest -n auto, which the project bans (xdist deadlock) — so this FR is a divergence, not just missing.
  • ⚠️ Not registered in specs/SPEC_INVENTORY.md (103/104/105 are; 102 is not).
• Remaining work / plan:
  • Scope: (1) Make self-healing actually heal — after a repair hop, re-run the scenario stage + evaluators in the same worktree and only mark passed=True if the re-evaluation passes; fix the response_dict["payload"] overwrite bug so _extract_strategy reads the volk's reply; make the fallback stop fabricating success. (2) Implement --mock-llm (add mock_llm field to HarnessConfig, a CLI flag, and a mock response loader) OR delete the dead mock_llm config key and drop TST-002. (3) Optional: add the type_check (mypy) evaluator. (4) Register in SPEC_INVENTORY.
  • Target files: src/orchestrata_ultimate/harness/healing.py, harness/runner.py (re-verify hook), harness/models.py + harness/cli.py + harness/config.py (mock-llm), a new harness/evaluators/type_check.py, specs/SPEC_INVENTORY.md.
  • Approach: The healing re-verify is the only substantial change — thread the Scenario+Executor into SelfHealingLoop.heal (or return a "re-run requested" signal to the runner) so a hop that claims repair triggers a genuine re-execution + evaluator pass before success is declared. The mock-llm and type_check items are small and self-contained.
  • Dependencies: couples to the kernel MessageDispatcher/CXP (SPEC-004) for real repair dispatch, and to the volk implementations (agents/volks/) that must actually apply repairs. Decide (open spec question #4) whether a repair volk may edit source or only the worktree.
  • Effort: M (0.5–2d) — healing re-verify is the bulk; rest is S. Risk: med — real self-healing needs a live repair-capable volk + real CXP round-trip to test meaningfully; without it the loop stays a stub and should be honestly documented as such rather than reporting fake successes.
• Evidence log:
  • ls src/orchestrata_ultimate/harness/ → 15 modules + evaluators/; read runner.py, models.py, isolation.py, healing.py, executor.py, config.py in full.
  • evaluators/ dir → syntax.py, lint.py, trace.py, file_exists.py, llm_judge.py (no type_check).
  • harness/scenarios/ → qa/codedirector/architect each with scenario_001.yaml + setup.py.
  • .gitlab-ci.yml present; harness/config.yaml present (declares mock_llm: false).
  • grep 'def test_' → tests/harness = 218 test funcs, 20 modules (test_healing.py, test_cxp_live.py, etc).
  • grep mock_llm src/.../models.py → none; grep mock cli.py → none (TST-002 unwired).
  • grep 10[2345] specs/SPEC_INVENTORY.md → 103/104/105 listed, 102 not.

SPEC-103 Tool-Calling Code Refactoring — DONE

• Specifies: An LLM-driven (DeepSeek deepseek-v4-pro) tool-calling layer that refactors an existing codebase in place — OBSERVE (read/list/grep/ast_outline) → MUTATE (edit_file/write_file) → VERIFY (run_check) — via a sandboxed tool registry, a tool-use loop, a refactoring system prompt, and a refactor()/arefactor() entry point; plus provider-protocol tool fields, DeepSeek adapter tool support, and (FR-009) REFACTOR-intent pipeline routing.
• Verified: Docs capabilities-refactor-openspec.md §A match the code; spot-checked every layer:
  • ✅ FR-005/006/007 loop + prompt + entry point — refactor/agent.py:71 arefactor, :155 sync refactor (never raises → status="error"), REFACTOR_SYSTEM_PROMPT at agent.py:27, RefactorResult at agent.py:55. Loop echoes reasoning_content (thinking-model requirement).
  • ✅ FR-003/004 sandboxed registry + 7 tools — refactor/tools.py:38 RefactorTools, TOOL_SCHEMAS, dispatch (never raises); _resolve path-sandbox + _CHECK_WHITELIST. refs RefactorTools → used at agent.py:87. tests/refactor/test_tools.py = 21 tests (path-traversal, network-command block, schema shape, edit found/not-found/non-unique).
  • ✅ FR-001/002 protocol + adapter — call_with_tools DEF at llm/client.py:696, used by agent.py:104 and agents/volks/base.py:281; DeepSeek adapter serialises/parses tool_calls (per doc §A.5, providers/adapters/deepseek_adapter.py:33-86).
  • ✅ FR-009 pipeline wiring is DONE (spec framed it as "future", but it is implemented): orchestrator/intent_broker.py:75 enable_tool_refactor=True, Step 2b dispatch at :177-181 (impact_report.classification.type == IntentType.REFACTOR → _handle_refactor), _handle_refactor at :309 imports+awaits arefactor at :320/:336. Classifier REFACTOR tie-break precedence confirmed in doc §A.7. refs arefactor → 1 def, 2 imports, 2 uses (intent_broker + self).
  • ✅ FR-008 structured logging — agent.py:127 logs each tool call via core.logging.get_logger.
  • ✅ CLI — scripts/refactor.py present (exit 0 only on status=="completed").
  • Status is "Draft" and DoD boxes are unchecked, but per the stale-marker rule the code is complete and wired. The only non-implemented item is TST-103-004 (a real live-DeepSeek refactor), which the spec itself scopes as an on-demand integration test.
• Remaining work / plan: None required for the capability. Optional: (a) flip the SPEC-103 status to Implemented and check the DoD; (b) add the on-demand real-DeepSeek integration test (needs a live DEEPSEEK_API_KEY — S, low risk, not blocking).
• Evidence log: symbols arefactor, refs arefactor/RefactorTools/call_with_tools; read refactor/agent.py in full; grep enable_tool_refactor|_handle_refactor|IntentType.REFACTOR in intent_broker.py → wiring confirmed; pytest --co tests/refactor → collects (21 tool tests + tests/scripts/test_ast_refactor.py).

SPEC-104 OpenSpec Format Support — DONE

• Specifies: A self-contained orchestrata_ultimate.openspec module that detects a project's spec folder + classifies its format (openspec/orchestrata/unknown), parses/renders/validates OpenSpec markdown (### Requirement: + #### Scenario:), edits specs, and scaffolds change proposals — all writes sandboxed to the detected spec root.
• Verified: Docs capabilities-refactor-openspec.md §B match the code:
  • ✅ FR-001/002 detect + recognize — openspec/detect.py:42 detect_spec_folder (never raises), detect.py:25 is_openspec (case-sensitive markers; delta or Requirement+Scenario).
  • ✅ FR-003 parse — openspec/parse.py parse_spec / parse_deltas / validate (line-based state machine, never raises).
  • ✅ FR-004/005 render/create/edit — openspec/render.py (render_spec/render_delta/ render_proposal/render_tasks); openspec/repo.py:28 OpenSpecRepo with capabilities, read/write_capability, add_requirement, create_change (refuses overwrite → OpenSpecError).
  • ✅ FR-006/007 validate + sandbox — validate() reports violations without raising; repo._safe mirrors the SPEC-103 path-traversal sandbox.
  • Data models (openspec/models.py) and public re-exports (openspec/__init__.py) present. tests/openspec/test_openspec.py = 16 tests. Registered in SPEC_INVENTORY (line 29).
  • refs detect_spec_folder/OpenSpecRepo → 1 def + 1 import (only from __init__), 0 uses in src/ — this is expected and correct: the spec frames pipeline use as optional ("the refactor agent MAY invoke these as tools"). It is a library capability exercised by tests, not orphaned. The optional refactor-tool wiring (registering these fns in TOOL_SCHEMAS) is deliberately not done and is not a gap.
  • Status "Draft" + unchecked DoD, but code is complete per the stale-marker rule.
• Remaining work / plan: None required. Optional (both S, low risk): flip status to Implemented; if desired later, expose create_change/add_requirement as SPEC-103 refactor tools (couples to SPEC-103's RefactorTools/TOOL_SCHEMAS).
• Evidence log: refs detect_spec_folder, refs OpenSpecRepo (0 src uses — library, expected); ls openspec/ (7 modules); doc §B cross-check; pytest --co tests/openspec collects; SPEC_INVENTORY line 29.

SPEC-105 Ontology-Driven Domain Model Generation — DONE (with one documented DIVERGENCE)

• Specifies: A skill that reads an OWL/RDFS Turtle ontology → canonical DomainModel IR → deterministic Jinja2 codegen (screaming-architecture / modular monolith) + bidirectional IR↔ontology round-trip, generating a temporal, insert-only (bitemporal, RDF-star) Stardog repository and SHACL constraints. Target src/orchestrata_ultimate/domaingen/.
• Verified: Docs ontology-and-codegen.md match the code (line refs have drifted slightly — e.g. parse_ontology DEF is reader.py:159 not the doc's :77 — but every capability is present and @traces-decorated):
  • ✅ FR-001 ontology→IR — domaingen/reader.py:159 parse_ontology (rdflib; classes→Entity, datatype/object props→Field/Relationship, subClassOf→parent, functional→single, restriction minCardinality→required); raises OntologyParseError(BaseError) on bad Turtle.
  • ✅ FR-002/003 IR→ontology + round-trip — domaingen/ttl.py render_ontology (deterministic, sorted, no timestamps).
  • ✅ FR-004/006 IR→code deterministic — domaingen/generator.py:78 generate_code; templates templates/{entities,repository,init}.py.j2; _topo_sort for inheritance ordering; _safe sandbox; _FORBIDDEN_SPARQL no-destructive-SPARQL guard.
  • ✅ FR-005 centralized logging/exceptions — templates import core.logging.get_logger + define RepositoryError(BaseError); no print / bare except.
  • ✅ FR-006 temporal insert-only Stardog repo — repository.py.j2 build_insert/supersede (tombstone+insert)/build_as_of/build_as_recorded/build_as_valid; _iri() injection guard. domaingen/stardog.py:26 StardogConnection (SPARQL-over-HTTP).
  • ✅ FR-008 SHACL — domaingen/constraints.py:25 generate_shacl → constraints.ttl.
  • ✅ CLI — scripts/domaingen.py (parse → generate / --render-ttl).
  • tests/domaingen/test_domaingen.py = 18 tests. Registered in SPEC_INVENTORY (line 30). Status "In Review", DoD almost fully checked (only the live-Stardog integration is explicitly deferred — no Stardog server in this env; unit tests assert generated SPARQL shape).
  • ⚠️ DIVERGENCE (spec out of date, code intentional): spec.md:48-58 / REQ-105-005 describe a separate generated relationships.py; the implementation deliberately realizes relationships as attributes on the entity dataclasses + the repository PREDICATES map and emits no relationships.py (only entities.py, repository.py, __init__.py). This is documented as intentional in docs/ontology-and-codegen.md §5.1. Per _METHOD, action = update the spec text, not the code.
  • refs parse_ontology/generate_code → 1 def + 1 import (from __init__), 0 src uses — library capability driven by scripts/domaingen.py + tests, not pipeline-wired (spec says the refactor agent "may optionally" invoke generation). Not orphaned.
• Remaining work / plan: No code work required. (1) Update specs/105-ontology-domain-model/spec.md §"Architecture of the generated model" / REQ-105-005 to drop relationships.py and describe the attribute+PREDICATES realization (S, low risk — doc-only). (2) The live-Stardog round-trip stays an on-demand integration test until a Stardog server is available (already scoped that way in the DoD).
• Evidence log: refs parse_ontology, refs generate_code; ls domaingen/ + templates/ (confirmed no relationships.py); doc §5.1 divergence note; pytest --co tests/domaingen collects (18 tests); SPEC_INVENTORY line 30.

Cluster summary table

Only SPEC-102 has real remaining code work. 103/104/105 are complete; 105 needs a small spec-text update (relationships.py divergence); 103/104 only optionally need a status flip + on-demand integration tests.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks