Orchestrata Ultimate › Plans › spec-audit › F-volks-2

Cluster F — Volks (QA / Concierge / CheetahClaws) — spec audit

Source: plans/spec-audit/F-volks-2.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-04 · 273 lines.

Prose pending.

Plan text

Cluster F — Volks (QA / Concierge / CheetahClaws) — spec audit

Auditor: Cluster F. Read-only investigation of src/orchestrata_ultimate/ per plans/spec-audit/_METHOD.md. Markers in checklists/requirements.md ignored; judgement by code only. Prior work: docs/spec-gap-register.md (031 marked "clean"; 032 has 3 recorded entries). This pass CONFIRMS/UPDATES those.

SPEC-031 QA Volk (U:Qa) — PARTIAL

• Specifies: U:Qa receives generated code from U:Cd and runs auto-fix → 3 deterministic gates → import validation → LLM 31-item code review → test generation → test execution → escalating-context bounce-to-Cd / compliance handoff to U:Cp.
• Verified:
  • ✅ Class + routing: QaVolk.process routes review_and_test→_handle_review (qa/volk.py:226,294). Alias U:Qa, role "QA Engineer" (:188-189). ⚠️ extends BaseAgent, not BaseVolk — BaseVolk symbol does not exist (0 symbols); FR-001's "extends BaseVolk" is a naming divergence, not a hole.
  • ✅ 13 auto-fixers, all present and all called before gate 1: _run_autofix_pass (qa/volk.py:417-431) invokes aioredis, sync_redis, f821_datetime, xadd_dict, hset_dict, redis_decode, missing_stdlib_imports, missing_barename_imports, redis_host, relative_imports, line_length, import_order, unused_imports (qa/_autofix.py:75-863; a 14th _autofix_parse_args is a bonus). US-031-001 / FR-003 ✅.
  • ✅ 3-gate sequential pipeline: _run_review_gate loop over review_gate_syntax/review_gate_lint/review_gate_checklist from codedirector/review_checks.py, each bounces to U:Cd on failure with a gate label (qa/volk.py:352-388,438-471). US-031-002 / FR-004/005/006 ✅.
  • ✅ Import validation after gates (_install_deps + _validate_imports, qa/volk.py:391-396); FR-007 ✅.
  • ❌ LLM 31-item code review — SPECIFIED BUT NOT WIRED. CODE_REVIEW_PROMPT is defined (qa/volk.py:62) but has 0 uses (refs CODE_REVIEW_PROMPT → "0 def, 0 import, 1 use" = its own Store only). No _llm_code_review method exists (symbols _llm_code_review → 0). _parse_review_verdict (:995) and _parse_review_issues (:1011) are defined with 0 uses each — orphaned. _handle_review goes gates → import → test-gen with no LLM review stage. US-031-003 / FR-008/009/010 ❌.
  • ✅ Zero-mock test generation: QA_SYSTEM_PROMPT + _MOCK_PATTERNS + _extract_and_write_tests / retry-without-mocks in _testgen mixin; 6-phase prompt, @traces, real-Redis db=15 (qa/volk.py:652-793). US-031-004 ✅.
  • ✅ Test execution + fix delegation: _run_tests, _extract_fail_lines, _build_test_fix_packet with from_test_fix=True, _test_fix_count increment, no cycle cap (qa/volk.py:803-904). US-031-005 / FR-013/014 ✅.
  • ✅ Escalating context (G229): _build_escalating_context — cycles 1-2 basic, 3+ history (last 3), 5+ root-cause, 8+ full file content (qa/_context.py:144-320), matching the spec's level table exactly. US-031-006 ✅.
  • ✅ Compliance handoff: _build_compliance_packet → git_commit("QA complete: tests …") + compliance_check packet to U:Cp with test/review metadata (qa/volk.py:906-949). US-031-007 / FR-019/020 ✅.
• Remaining work / plan:
  • Scope: resolve the orphaned LLM-review surface. Two coherent options — this is a product/design call, not a blind fix: (a) Wire it — add _llm_code_review(source_files) that calls call_llm(CODE_REVIEW_PROMPT, …, temperature=0.0, volk="U:Qa"), parse with the existing _parse_review_verdict/_parse_review_issues, and insert it in _handle_review after the 3 gates + import validation, before test-gen; FAIL verdict → _bounce_for_fix. This is what US-031-003 literally requires. (b) Delete + re-spec — the spec's own Why argues gates "must be deterministic, not delegated to an LLM that may hallucinate a pass verdict"; if deterministic-gates-only is the intended design, delete the dead CODE_REVIEW_PROMPT + _parse_review_* and update the spec (drop US-031-003 / FR-008-010 as an LLM stage). Note verification/review/ already re-implements the 31 checklist items deterministically, which supports option (b).
  • Cross-cutting: SPEC-013 REQ-013-010 gap CONFIRMED — QA never validates the UI against UX wireframes. 0 hits for ux_design/wireframe/has_ui under qa/. has_ui is produced by Concierge and threaded through the pipeline but is never consumed by U:Qa; there is no wireframe/UX-conformance check.
  • Target files: src/orchestrata_ultimate/agents/volks/qa/volk.py (wire or delete); specs/031-qa-volk/spec.md (re-spec if option b). For the UX gap: thread ux_design/has_ui from _handle_review payload into a check.
  • Approach: option (a) is a ~1-screen insertion using symbols that already exist; option (b) is a deletion + spec edit. The UX-conformance check is a separate, larger design item shared with SPEC-013 / SPEC-035 (U:Ux).
  • Dependencies: SPEC-029 (CodeDirector — the bounce target + review_checks provider), SPEC-028 (LLM client call_llm), SPEC-013/035 (U:Ux wireframes for the UX-conformance gap), SPEC-005 (multi-agent CXP routing).
  • Effort: S for (a) or (b) in isolation; M including the SPEC-013 UX check. Risk: low-med — wiring an LLM review adds token cost + a new bounce path that the pipeline gate-tests must accommodate; the decision itself is the risk, not the code.
• Evidence log:
  • refs CODE_REVIEW_PROMPT → 0 def / 0 import / 1 use (its own definition).
  • grep CODE_REVIEW_PROMPT src/ → only the def + doc/comment refs in codedirector/review_checks.py, verification/review/* (comments).
  • symbols _llm_code_review → 0 symbols.
  • grep _parse_review_verdict|_parse_review_issues src/…/qa/ → def only, no callers.
  • grep _build_escalating_context → _context.py:144 def + callers at volk.py:882, _context.py:311.
  • grep def _autofix src/…/qa/_autofix.py → 13 spec'd fixers + _autofix_parse_args.
  • grep ux_design|wireframe|has_ui src/…/qa/ → 0 hits.
  • symbols BaseVolk → 0; QaVolk(... BaseAgent) at qa/volk.py:176-182.

SPEC-032 Concierge Volk (U:Cx) — DONE

• Specifies: U:Cx intake agent — elicits requirements over ≤4 Q&A rounds (D1-D7), forces finalization at MAX_QA_ROUNDS, parses structured fields, emits Prolog facts, detects UI need, hands off to U:BA.
• Verified:
  • ✅ Class + alias/role: ConciergeVolk(BaseAgent) alias U:Cx, role "Concierge (Intake)" (concierge/volk.py:115-117). MAX_QA_ROUNDS: ClassVar = 4 (:120). FR-001/002 ✅.
  • ✅ Subject routing: process → intent/response; unrecognised subject is logged and dropped (:187-202) — the register's REQ-032-014 logger fix landed (module now has logger, :16). ✅
  • ✅ First-round elicitation (G162-1): _handle_intent stores original_intent, always calls the LLM, and rewrites a premature REQUIREMENTS_COMPLETE into QUESTIONS so it never fast-paths (:216-254). FR-004 ✅.
  • ⚠️ round_count (spec) vs _qa_round (code): _handle_intent sets _qa_round = 0 and does not bump to 1 (:221). This is the register's surfaced decision (REQ-032-021: spec wants 1) — G213 forced finalization fires one clarification round later than the spec's number. UX call, not a bug.
  • ✅ MAX_QA_ROUNDS enforcement (G213): _handle_response increments _qa_round and force-finalizes at >= MAX_QA_ROUNDS without re-calling the LLM (:296-308). FR-006 ✅.
  • ✅ Structured parsing: _finalize parses PROLOG_FACTS / ESCALATE_TO / TASK_SUMMARY / PROJECT_NAME / WORK_TYPE tolerantly (missing fields don't raise) (:364-410). FR-007/008 ✅. ⚠️ minor divergence: _finalize is sync and does not re-call the LLM (spec REQ-032-033 says it calls the LLM with a finalization prompt) — intentional, avoids a QUESTIONS loop; uses the already-obtained llm_response.
  • ✅ UI detection (register FR-009 fix landed): detect_ui_requirement scans the whole conversation [llm_response, original_intent, *conversation_history] (:165-174,420-422) — a UI need stated only in a later answer is caught. ✅
  • ✅ Fixed U:BA handoff: always routes to U:BA, ignores LLM ESCALATE_TO, packs task_summary/project_name/prolog_facts/conversation_history/has_ui (:424-443). FR-010/012 ✅. ⚠️ subjects are clarification/ analyze_requirements in code vs spec's questions/requirements — functional divergence (spec text is stale).
• Remaining work / plan: None material. Two spec-text items to reconcile (not code fixes): (1) the round_count 0-vs-1 surfaced decision (register), (2) outbound subject names + "_finalize calls the LLM" wording. Recommend updating the spec, not the code. Effort S, risk low.
• Evidence log: read concierge/volk.py in full; register lines 35/36/75 confirm the three prior entries (UI-scan fix, logger fix, round_count decision) all match current code.

SPEC-061 CheetahClaws Volk Agent Integration — DIVERGENCE (superseded by SPEC-050)

• Specifies: Integrate CheetahClaws as U:Cd code director — a CheetahClawsTUI with create_checkpoint/rewind/query_memory/edit_file/search_files/ execute_command, checkpoint+rollback (US-003), CXP multi-agent coordination.
• Verified:
  • ❌ The specified class surface does not exist: symbols CheetahClawsTUI → 0, CheetahClawsService → 0, CheetahClawsConfig → 0, rewind → 0, query_memory → 0. (create_checkpoint exists only in session/recovery.py — session recovery, unrelated to CC file edits.)
  • ✅/⚠️ The concept is realized differently: the live "CheetahClaws integration" is integrations/cheetahclaws/__init__.py, which is a SPEC-050 module — MultiAgentCoordinator (:92), GovernanceBridge (:210), UltimateTUI (:264) exposing handle_edit/handle_intent/route_command. It is wired: gateway/handlers/ultimatetui.py:34,50 imports and drives UltimateTUI via tui.* JSON-RPC from the VizInt browser (refs UltimateTUI → 1 def, 1 import, 1 use). So CC-as-editor is live, but through SPEC-050's multi-agent edit-review flow, not SPEC-061's CheetahClawsTUI.
  • ❌ U:Cd is not CheetahClaws. The actual U:Cd is the deterministic CodeDirectorVolk (codedirector/volk.py:42, alias U:Cd, role "CodeDirector (Coder)"). SPEC-061's premise (import CC as U:Cd) is not how the port turned out — CC is a coordination front-end dispatching to U:Cd/U:Qa/U:Ar/U:Ux.
  • ❌ Checkpoint/rollback for edits (US-003, FR-003/004): absent from the CC edit path — MultiAgentCoordinator/UltimateTUI.handle_edit write the file directly (__init__.py:329-349); no checkpoint object or rollback. (Pipeline edits are git-committed by U:Qa instead.)
• Remaining work / plan:
  • Scope: this is a spec-alignment problem, not a build. SPEC-061 describes a CheetahClaws-native TUI that the port deliberately replaced with SPEC-050's UltimateTUI + deterministic CodeDirectorVolk. Recommended action = update SPEC-061 to declare it superseded by SPEC-050 (cross-reference), OR, if the rich CC TUI (checkpoint/rewind/query_memory) is genuinely wanted, scope it as new feature work. Do not hand-build a duplicate CheetahClawsTUI.
  • If checkpoint/rollback is wanted on the edit path: add a checkpoint snapshot before handle_edit writes and a rollback RPC — small, self-contained.
  • Target files: specs/061-cheetahclaws-volk/spec.md (re-spec / mark superseded); optionally integrations/cheetahclaws/__init__.py + gateway/handlers/ultimatetui.py for checkpoint/rollback.
  • Dependencies: SPEC-050 (UltimateTUI — the real integration), SPEC-029 (CodeDirectorVolk = actual U:Cd), SPEC-016 (gateway RPC), SPEC-023 (session/recovery checkpoint, if reused).
  • Effort: S (re-spec) / M (add edit checkpoint+rollback). Risk: low.
• Evidence log: symbols CheetahClawsTUI|CheetahClawsService|CheetahClawsConfig| rewind|query_memory → all 0; refs UltimateTUI → wired at gateway/handlers/ultimatetui.py; codedirector/volk.py:42 = U:Cd; read integrations/cheetahclaws/__init__.py (SPEC-050 header, lines 1-24).

SPEC-062 CheetahClaws Tools Gap Analysis — GAP (planning meta-spec, superseded by 063–072)

• Specifies: A gap-analysis roadmap claiming 27/41 Claude-Code tools done and enumerating 14 missing (REPL, Enter/ExitWorktree, LSP, PowerShell, RemoteTrigger, ScheduleCron, TaskStop, TaskOutput, TeamCreate, TeamDelete, ToolSearch, ListMcpResources, ReadMcpResource, McpAuth), each as a BaseTool subclass registered in a ToolRegistry.
• Verified:
  • ❌ All 15 named tool classes are absent — symbols for every one of REPLTool, EnterWorktreeTool, ExitWorktreeTool, LSPTool, PowerShellTool, RemoteTriggerTool, ScheduleCronTool, TaskStopTool, TaskOutputTool, TeamCreateTool, TeamDeleteTool, ToolSearchTool, ListMcpResourcesTool, ReadMcpResourceTool, McpAuthTool → 0 symbols each.
  • ❌ The assumed framework doesn't exist: symbols BaseTool → 0; there is no class-based ToolRegistry (only a IToolRegistry Protocol in plugin/interfaces.py:46 with a handler-based register_tool). SPEC-062's FR-021 ("follow the base tool interface in SPEC-001") is built on a BaseTool that was never ported.
  • ⚠️ The actual live tool surface is unrelated: agents/tool_registry.py exposes 8 volk-facing LLM function tools — read_file, write_file, list_dir, search_code, run_shell, ast_query, check_coverage, message_volk (:41-190, dispatched in ToolExecutor.execute :329-352). This is a different design (LLM function-calling for volks), not Claude-Code tool parity. So even the "27 completed" column is conceptual mapping, not orchestrata classes.
  • ⚠️ Capability keyword sweep found no matching tool impls: pty/REPL, LSPClient/pylsp/pyright, croniter/cron, paramiko/ssh, list_mcp/read_mcp → no tool classes (the worktree hits are the test harness in harness/, not a SPEC-064 tool; the cron hit is integrations/fireflies_sync.py).
  • ✅ Superseded by dedicated specs: specs/063-repl-tool, 064-git-worktree-tools, 065-lsp-tool, 066-powershell-tool, 067-remote-trigger-tool, 068-schedule-cron-tool, 069-task-management-tools, 070-team-tools, 071-mcp-resource-tools, 072-tool-search-tool all exist. SPEC-062's 14 tools map 1:1 onto these — 062 is the umbrella/roadmap, 063–072 are the work items.
• Remaining work / plan:
  • Scope: do not implement SPEC-062 as a standalone unit. It is a planning document whose "Status: Complete" refers to the analysis, not the tools. The genuine implementation gap lives in specs 063–072 (owned by other clusters). Action for 062 itself: reconcile it with reality — its BaseTool/ToolRegistry architecture and 27/41 inventory do not match the actual agents/tool_registry.py design; either update 062 to point at 063–072 + the real tool model, or retire it as an obsolete roadmap.
  • Target files: specs/062-cheetahclaws-tools-gap/spec.md (re-spec/retire). No src/ work under 062 — defer to 063–072.
  • Dependencies: heavy cross-cluster — 063–072 (the actual tools), SPEC-001 (the never-ported BaseTool base the whole spec assumes), SPEC-060 (PAM++ governance mapping the tool table references), SPEC-069↔SPEC-023 (task tools couple to the goal/task system), SPEC-070↔SPEC-005 (team tools couple to multi-agent).
  • Effort: S to re-spec 062; the 14 tools themselves are L in aggregate but belong to 063–072. Risk: low for the re-spec; med if anyone tries to build 062's tools on the phantom BaseTool framework.
• Evidence log: symbols <ToolName> for all 15 → 0 each; symbols BaseTool → 0; symbols ToolRegistry → only IToolRegistry Protocol; grep name= in agents/tool_registry.py → 8 tools; ls specs/ → 063–072 present; capability keyword grep (pty/lsp/cron/ssh/mcp) → no tool impls.

Cluster summary table

Cross-cluster dependencies noticed

• SPEC-062 ↔ tools-suite 063–072 — 062's 14 missing tools ARE specs 063–072 (1:1). 062 is the umbrella; the real implementation audit is those specs (other clusters). Whoever owns 063–072 should also know no BaseTool base class / class-based ToolRegistry exists — the live model is agents/tool_registry.py (8 function-tools) + plugin/interfaces.py IToolRegistry Protocol.
• SPEC-031 (QA) ↔ SPEC-013/035 (UX) — confirmed REQ-013-010 gap: QA never validates UI vs UX wireframes; has_ui is produced by Concierge but unconsumed by U:Qa. Any fix threads ux_design/has_ui from the pipeline into a QA check.
• SPEC-061 ↔ SPEC-050 — SPEC-061 is functionally realized by SPEC-050's UltimateTUI (wired at the gateway), not by a CheetahClawsTUI. Audit of SPEC-050 (other cluster) is where the CC integration actually lives.
• SPEC-062 ↔ SPEC-060 (PAM++) and SPEC-069↔SPEC-023 / SPEC-070↔SPEC-005 — the task/team tools couple to the goal-system and multi-agent subsystems.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks