Orchestrata Ultimate › Plans › spec-audit › reverify-2026-10-18 › F-volks-2

Cluster F (volks 2 + verification) — re-verification 2026-10-18

Source: plans/spec-audit/reverify-2026-10-18/F-volks-2.md at 8a2cb42ad · last changed 2026-10-18 · first committed 2026-10-18 · 319 lines.

Prose pending.

Plan text

Cluster F (volks 2 + verification) — re-verification 2026-10-18

Auditor: Cluster F. Read-only re-verification of specs 031/032/033/034/035 against current code (components branch, 337 commits ahead of the 3-month-old audits). Markers in checklists/requirements.md ignored; judgement by code only via the AST toolkit (symbols / refs / source / who-imports). Prior work: plans/spec-audit/F-volks-2.md (session 195), docs/spec-gap-register.md (2026-07-14), 00-MASTER-PLAN.md §2/§6. This pass CONFIRMS/UPDATES those against current code.

Two headline updates versus the 3-month-old record:

1. SPEC-031's LLM 31-item review is no longer orphaned — _run_llm_code_review is wired into _handle_review. The sole PARTIAL gap from the prior audit is closed.
2. SPEC-035 (verification) is no longer orphaned — the S10 Sentinel decision landed: VerificationCoordinator is wired into PipelineRunner via PipelineVerificationGate (audit-only by default). The three tiered agents remain orphaned.

SPEC-031 QA Volk (U:Qa) — DONE (was PARTIAL)

• Specifies: U:Qa receives code from U:Cd; runs 13 auto-fixers → 3 deterministic gates → import validation → LLM 31-item review → test gen → test exec → escalating-context bounce / compliance handoff.
• Verified:
  • ✅ Class + routing: QaVolk(AutofixMixin, DepInstallMixin, TestGenMixin, FailureContextMixin, BaseAgent) (qa/volk.py:179); process routes review_and_test → _handle_review (qa/volk.py:221,285). ⚠️ extends BaseAgent
    • mixins, not BaseVolk (BaseVolk → 0 symbols) — FR-001 naming divergence only.
  • ✅ 13 auto-fixers present and all invoked before gate 1: _run_autofix_pass (qa/volk.py:527) drives _autofix_aioredis…_autofix_unused_imports (qa/_autofix.py:87-893; 14th _autofix_parse_args is a bonus).
  • ✅ 3-gate sequential pipeline: _run_review_gate (qa/volk.py:553) over review_gate_syntax/lint/checklist from codedirector/review_checks.py (imports at qa/volk.py:18-22); each failure bounces to U:Cd with gate label.
  • ✅ Import validation after gates (_bounce_for_import_errors, qa/volk.py:589).
  • ✅ LLM 31-item review — NOW WIRED (the prior PARTIAL gap is closed). CODE_REVIEW_PROMPT = 31-item checklist (qa/volk.py:70); _run_llm_code_review (qa/volk.py:426) is called from _handle_review at qa/volk.py:396 after the 3 gates + import validation and before test-gen. It invokes compose_persona(CODE_REVIEW_PROMPT) via _call_llm_with_tools (temperature=0.0, qa/volk.py:487), parses _parse_review_verdict (:1102, used :496) and _parse_review_issues (:1123, used :501); FAIL → _bounce_for_fix with "LLM REVIEW FAIL: 31-item checklist". Runs on every cycle incl. test-fix returns (REQ-031-043). US-031-003 / FR-008/009/010 ✅.
  • ✅ Zero-mock test gen: TestGenMixin _extract_and_write_tests (qa/_testgen.py:33), _validate_no_mocks (:126), _retry_without_mocks (:141). US-031-004 ✅.
  • ✅ Test exec + fix delegation: _handle_test_rerun (qa/volk.py:631), _build_test_fix_packet (:973), _extract_fail_lines (:955). US-031-005 ✅.
  • ✅ Escalating context (G229): _build_escalating_context (qa/_context.py:148). US-031-006 ✅.
  • ✅ Compliance handoff: _build_compliance_packet (qa/volk.py:1014) → _git_commit (:1059) + compliance_check to U:Cp. US-031-007 ✅.
• Remaining work / plan:
  • Scope: none material. One spec-text item only — FR-001 "extends BaseVolk" vs the actual BaseAgent+mixins (BaseVolk does not exist as a symbol).
  • Cross-cluster: SPEC-013 REQ-013-010 gap persists — QA never consumes has_ui/ux_design; the Concierge produces has_ui but no QA wireframe/UI check exists. Owned by SPEC-013/035, not 031.
  • Target files: specs/031-qa-volk/spec.md (FR-001 wording). No src/ work.
  • Dependencies: SPEC-013/035 (U:Ux wireframe check, if pursued).
  • Effort: S (spec-text only). Risk: low.
• Evidence log:
  • symbols QaVolk → qa/volk.py:179 + 24 methods (incl. _run_llm_code_review:426).
  • refs CODE_REVIEW_PROMPT → 1 Store (qa/volk.py:70) + 1 Load (qa/volk.py:487).
  • refs _parse_review_verdict → 10 uses (USE qa/volk.py:496); _parse_review_issues → 8 uses (USE qa/volk.py:501). Both now have production callers (prior audit: 0).
  • refs _run_llm_code_review → DEF :426, USE :396.
  • symbols _autofix* → 13 spec'd fixers in qa/_autofix.py (+ _autofix_parse_args).
  • symbols BaseVolk → 0.

SPEC-032 Concierge Volk (U:Cx) — DONE (spec-text divergences remain)

• Specifies: U:Cx intake — elicits requirements over ≤4 Q&A rounds (D1–D7), forces finalization at MAX_QA_ROUNDS, parses structured fields, emits Prolog facts, detects UI need, hands off to U:BA.
• Verified (remaining items; FR-009 and REQ-032-014 already confirmed by main session):
  • ✅ Class/alias/role: ConciergeVolk(BaseAgent) alias "U:Cx", role "Concierge (Intake)", MAX_QA_ROUNDS: ClassVar[int] = 4 (concierge/volk.py:119-125).
  • ✅ Subject routing: process → intent/response; unrecognised subject logged via module logger (concierge/volk.py:195-217). REQ-032-014 ✅.
  • ✅ First-round elicitation (G162-1): _handle_intent stores original_intent, always calls LLM, rewrites premature REQUIREMENTS_COMPLETE: into QUESTIONS (concierge/volk.py:224-281). FR-004 ✅.
  • ⚠️ REQ-032-021 — round_count stays 0 (surfaced decision, UNCHANGED): _handle_intent sets self._qa_round = 0 and does not bump to 1 (concierge/volk.py:236). Spec's US-001 wants round_count = 1 after the first call. Reported factually, no change made. Attribute is _qa_round (spec names it round_count).
  • ✅ MAX_QA_ROUNDS enforcement (G213): _handle_response increments _qa_round and force-finalizes at >= MAX_QA_ROUNDS without re-calling the LLM (concierge/volk.py:284-322). FR-006 ✅.
  • ✅ Structured parsing: _finalize parses PROLOG_FACTS / ESCALATE_TO / TASK_SUMMARY / PROJECT_NAME / WORK_TYPE tolerantly (concierge/volk.py:376-463). FR-007/008 ✅. ⚠️ _finalize is sync and does not re-call the LLM (REQ-032-033 divergence — intentional, avoids a QUESTIONS loop).
  • ✅ UI detection: detect_ui_requirement(texts: list[str]) staticmethod scans the whole conversation [llm_response, original_intent, *conversation_history] (concierge/volk.py:174-190,447-449). FR-009 ✅.
  • ✅ Fixed U:BA handoff: always routes to U:BA, subject analyze_requirements, packs task_summary/project_name/prolog_facts/conversation_history/has_ui/work_type (concierge/volk.py:452-462). FR-010/012 ✅.
  • ⚠️ Outbound subjects are clarification/analyze_requirements (spec says questions/requirements) — functional divergence, spec text is stale.
• Remaining work / plan:
  • Scope: none material. Spec-text reconciliation only: (1) round_count 0-vs-1 (REQ-032-021 decision), (2) outbound subject names, (3) _finalize "calls the LLM" wording, (4) round_count vs _qa_round attribute name.
  • Target files: specs/032-concierge-volk/spec.md. No src/ work.
  • Dependencies: SPEC-005 (CXP), SPEC-012 (U:BA).
  • Effort: S (spec-text only). Risk: low.
• Evidence log: source agents.volks.concierge.volk (full module, 463 lines); _qa_round = 0 at concierge/volk.py:236; _handle_response increment + force-finalize at :305-314; detect_ui_requirement staticmethod at :174.

SPEC-033 Governed Autonomous Build Factory — PARTIAL

• Specifies: Prescriptive target — deterministic exoskeleton: assumption ledger + arbiter cascade (C1), interpretation mutation (C2), lexical/structural detectors (C3), logical detectors (C4), DoR construction procedure (C5), state-diff enforcement, in-toto/SLSA provenance, worktree DAG scheduler, amendment classifier, test discipline (T1–T5). §9 is a self-declared backlog (B1–B28).
• Verified (backlog items marked done in §9, checked against current code):
  • ✅ B13 assumption ledger + cascade: factory/assumptions/ledger.py::AssumptionLedger (:211), cascade.py, adjudicator.py, prefilter.py, sources.py.
  • ✅ B15a/b anchors: factory/anchors/ (evaluation/gate/intake/models/source/store); AnchorGateVerdict/fast_path_gate consumed by factory/fastpath/classifier.py:50.
  • ✅ B16 amendment classifier: factory/amendment/classify.py::classify_amendment (:115) + coupling.py + syntactic.py.
  • ✅ B2 conformance + B3 trace validation: factory/conformance/engine.py, factory/contracts/trace_validation.py.
  • ✅ B22 C3 detectors: factory/detectors/ (engine/clause/lexicons/models/rules/c1_bridge).
  • ✅ B23 DoR procedure: factory/dor_procedure/ (axes/artifact/validate/anchors_export/rulings).
  • ✅ B26 C4 formalization: factory/formalization/c4.py::run_c4 (imported by factory/decision/phase.py:60); verification/spec_completeness.py::evaluate_spec_completeness (:207).
  • ✅ B1a–c state-diff + B20 provenance: factory/enforcement/ (bootstrap/enforce/ledger/ attestation/attestation_store/signing); classify_delta (bootstrap.py:54), InTotoLink/SlsaProvenance (attestation.py).
  • ✅ B6 red-first + mutation: factory/evidence/red_first.py, mutation.py.
  • ✅ B12 fast-path classifier: factory/fastpath/classifier.py.
  • ✅ B5 projection: factory/projection/analyzer.py, completeness.py.
  • ✅ B4a promotion + B9 scheduler: factory/promotion/ (store/models), factory/scheduler/ (dag/queue/scheduler/worktrees); no_self_merge at factory/dispatch/gates.py (B21 no-self-merge partial).
  • ✅ B17 constitution meta-lint: scripts/prolog_meta_lint.py::check_stratification (:1010).
  • ✅ B18 admission contract: core/adapters/claude/gate_policy.py — GateOutcome.MODIFY (:51) + case GateOutcome.MODIFY: apply-mutation branch (:212, ADR-0032 #1); InterceptorDecision.MODIFY + InterceptorResult.modified_input (core/adapters/claude/tool_interceptor.py:32-53).
  • ✅ B19 proof-tree: prolog/explain.pl (module-level meta-interpreter).
  • ✅ B25 RTM integrity constraints: verification/rtm_constraints.py.
  • ❌ WIRING GAP — the whole factory/ package has 0 import sites inside src/orchestrata_ultimate (who-imports orchestrata_ultimate.factory → 0; every import of its submodules is intra-package). Reachable only via 4 CLI scripts (scripts/factory_amend.py, factory_promote.py, factory_dor_procedure.py, factory_c4.py); no console_scripts entry in pyproject.toml. The package docstring (factory/__init__.py) is stale — claims "Phase-0 scope only" while the tree holds all phases. 51 test files exist (tests/factory/).
  • ❌ Still absent (consistent with §9): B14 interpretation mutation (C2) — 0 symbols (InterpretationMutation/alternative_reading → 0); B1d OS trust boundary (TrustBoundary → 0); B1e red-team suite (0 src symbol); B10 escalation constants (blocked, human K/SLO needed); B27 defeasible layering (blocked on B17); B28 CLP(FD) scheduling; B7 infra-test parallelism; B4b incremental AST extraction.
• Remaining work / plan:
  • Scope: two distinct tracks. (a) Wire-or-declare the factory — it is built + tested (51 test files) but not reachable from any production path; decide the canonical entry surface (a console_scripts entry / a pipeline-adjacent hook) or formally record it as tooling-only. (b) Build the genuinely-absent backlog items B14/B1d/B1e/B10/B27/B28/B7/B4b.
  • Target files: src/orchestrata_ultimate/factory/** (wiring), pyproject.toml (entry point), specs/033-*/spec.md §9 (the done/to-build Status column is now stale — many "to-build" rows are built).
  • Approach: refresh §9 against the current tree first (it understates completion), then decide the wiring seam.
  • Dependencies: SPEC-001/007 (kernel/Prolog), SPEC-064 (worktrees), SPEC-106 (ontology).
  • Effort: M to re-baseline §9 + declare/wire the entry surface; L for the remaining backlog (B14/B1d/B1e are the heavy ones). Risk: med — the factory is a separate exoskeleton, not a drop-in for PipelineRunner; wiring it wrong couples two governance loops.
• Evidence log:
  • who-imports orchestrata_ultimate.factory → 0 import sites.
  • who-imports …factory.assumptions.ledger → 5 (all intra-package).
  • who-imports …factory.scheduler.scheduler → 0; …factory.amendment.classify → 0.
  • symbols AssumptionLedger → factory/assumptions/ledger.py:211.
  • symbols classify_delta → factory/enforcement/bootstrap.py:54; classify_amendment → factory/amendment/classify.py:115.
  • symbols check_stratification → scripts/prolog_meta_lint.py:1010.
  • source core.adapters.claude.gate_policy → GateOutcome.MODIFY:51, case MODIFY:212; SELF_EXCLUSION_SET:112.
  • symbols InterpretationMutation/TrustBoundary/red_team → 0 each.
  • find src/…/factory -name '*.py' → ~80 modules; find tests/factory -name '*.py' → 51.

SPEC-034 Governed Action Classification — GAP (design-only, not built)

• Specifies: Prescriptive design-for-review — property-derived classification (ADR-0034), two-source scoped authorization (ADR-0035), classification provenance under the trust boundary (ADR-0036), a single PDP + single reduce_to_action reduction function, and the §8 audit of the current Python bypass topology.
• Verified (status: "Draft (design for review — NOT approved to build)"; ADRs Proposed):
  • ❌ Nothing in the B34-1…8 backlog is built. reduce_to_action, ActionDescriptor, assert_action_facts, consent_grant, matching_consent_grant → 0 symbols each (B34-1/4/5 absent).
  • ❌ Name lists persist — constitutional.pl still carries the "BEFORE (rejected)" form: requires_human_consent(delete_data) etc. (constitutional.pl:125-131) and human_consent_given/2 (:50,122); security_threat/1 still a roster. B34-2 (property-derived rewrite) not done.
  • ✅ §8 topology is still accurate (all load-bearing bypass sites exist):
    • PolicyEngine (governance/policy_engine.py:47) — PDP choke point.
    • ToolInterceptor default-engineless: HookOrchestrator.__init__ does self.interceptor = interceptor or ToolInterceptor() (core/adapters/claude/hooks.py:90) → every tool call ALLOW when no engine. Top target B34-5 still open.
    • SELF_EXCLUSION_SET frozenset still present (core/adapters/claude/gate_policy.py:112).
    • ScopeGuard (gateway/handlers/_scope_guard.py:22), PostmanGovernanceManager (postman/governance.py:77), PrologGate (governance/prolog_gate.py:41), EthicalBoundary (governance/ethical.py:34), intent_security.py, orchestrator/engine.py, resource_access.py — all still present.
  • ✅ B34-0 (read-only enforcement-topology audit) is the only DONE item — its §8 findings remain valid against current code (only line numbers have drifted).
• Remaining work / plan:
  • Scope: ratification first — nothing should be built until ADRs 0034/0035/0036 are accepted (constitution-tier). On acceptance: B34-1 (descriptor ontology) → B34-2 (property rules) → B34-4 (scoped grants) → B34-5 (single reduction + PEP conversion) → B34-6 (audit→enforce) → B34-7 (amendment direction) → B34-8 (incompleteness routing); B34-3 gated on SPEC-033 B1d (identity boundary).
  • Target files: specs/034-*/spec.md (ratification), docs/adr/0034–0036, knowledge_graph/canonical_schema.py (B34-1), prolog/constitutional.pl (B34-2), the §8 PEP sites (B34-5).
  • Dependencies: SPEC-033 B1d (identity boundary) hard-gates B34-3/REQ-034-009; SPEC-007 (Prolog governance), SPEC-106 (canonical ontology), SPEC-060 (PAM).
  • Effort: S now (ratify the 3 ADRs); L to build B34-1…8 once ratified. Risk: med — touching the constitution is constitution-tier; a wrong property rule is a security regression.
• Evidence log:
  • symbols reduce_to_action|ActionDescriptor|assert_action_facts|consent_grant|matching_consent_grant → 0 each.
  • grep requires_human_consent|human_consent_given constitutional.pl → name-list form at :50,122,125-131.
  • source core.adapters.claude.hooks → self.interceptor = interceptor or ToolInterceptor() at :90.
  • symbols PolicyEngine|ScopeGuard|PostmanGovernanceManager|PrologGate|EthicalBoundary → all present (see file:line above).

SPEC-035 Verification Agent (U:Vfy) — PARTIAL (was ORPHANED)

• Specifies: Prolog-guided multi-tier verification — Execution/Supervisor/Verification agents, checkpoints, SATISFIED/NOT_MET via Prolog, max-5-attempt escalation.
• Verified:
  • ✅ S10 Sentinel decision landed — VerificationCoordinator is now wired. PipelineVerificationGate (orchestrator/verification_gate.py:435) composes VerificationCoordinator (verification/orchestration/coordinator.py:57), VerificationEngine, VerificationSessionManager, EscalationHandler, WorkTypeConstraintSource, ArtifactContextBuilder, VerificationRedisManager. PipelineRunner imports it (orchestrator/pipeline_runner.py:72), constructs it (:274, degraded-disable fallback :283), opens a session before stage execution (:382), verifies per-stage via after_stage (:649), and attaches the verdict to PipelineResult.verification via finalize (:721-726, field :123).
  • ✅ max_attempts reconciled 3→5: VerificationGateConfig.max_attempts = 5 and from_env VERIFY_MAX_ATTEMPTS default "5" (verification_gate.py:106-136); the gate passes max_attempts into VerificationCoordinator and EscalationHandler (:482-489). (Coordinator's own ctor default is still max_attempts=3 (coordinator.py:60), overridden by the gate.)
  • ✅ Prolog as source of truth: ConstraintEvaluator marshals ground terms and queries constraint_satisfied/3 in prolog/constraints.pl via PrologEngine — "no Python comparison ladder" (verification/constraints/evaluator.py:1-90).
  • ✅ Checkpoint verification: CheckpointVerifier (verification/checkpoints/verifier.py:32), get_checkpoint_predicates/get_next_checkpoint (checkpoints/definitions.py).
  • ✅ Session state + escalation: VerificationSessionManager, verification/session/verification_session.py, EscalationHandler.
  • ⚠️ AUDIT-ONLY by default: VerificationGateConfig.enforce = False, VERIFY_ENFORCE=1 opts into failing/escalating a run (verification_gate.py:106-136, 505-508). Enforcement is a staged rollout, not fail-closed today.
  • ❌ The three tiered agents are still ORPHANED (0 production import sites): ExecutionAgent (verification/agents/execution_agent.py:18), SupervisorAgent (verification/agents/supervisor_agent.py:28), VerificationAgent (verification/agents/verification_agent.py:48). The gate uses VerificationEngine + CheckpointVerifier + ConstraintEvaluator directly; the spec's US-002/003/004 agent roles are not wired.
• Remaining work / plan:
  • Scope: (a) wire-or-delete the three tiered agents (US-002/003/004) — the S10 decision named them but only the coordinator landed; (b) promote audit-only → enforcing via VERIFY_ENFORCE default flip, separately ratified per SPEC-034 REQ-034-021's shadow→enforce precedent.
  • Target files: verification/agents/{execution,supervisor,verification}_agent.py (wire or retire), orchestrator/verification_gate.py (enforce default), if enforced then orchestrator/pipeline_runner.py re-gate.
  • Approach: the agent tier is the multi-agent "Execution → Supervisor → U:Vfy" loop the spec's US-002/003/004 and architecture diagram require; either compose them into PipelineVerificationGate (replacing the direct engine path) or delete + re-spec those stories.
  • Dependencies: SPEC-004 (CXP), SPEC-005 (multi-agent), SPEC-007b/governance (Prolog), SPEC-003 (PipelineRunner).
  • Effort: M (wire/delete the agent tier + enforce flip). Risk: med — flipping enforcement is a fail-closed behavior change to the live pipeline.
• Evidence log:
  • refs VerificationCoordinator → IMP orchestrator/verification_gate.py:68, USE :451,465; refs PipelineVerificationGate → IMP pipeline_runner.py:72, USE :196,273,282.
  • who-imports orchestrata_ultimate.orchestrator.verification_gate → pipeline_runner.py:71 (1 site).
  • source orchestrator.pipeline_runner → open_session :382, after_stage :649, finalize :721, enforce gate :731-743.
  • refs ExecutionAgent|SupervisorAgent|VerificationAgent → DEF only, no src/ IMP/USE.
  • who-imports …verification.agents.{execution,supervisor,verification}_agent → 0 each.
  • source verification.constraints.evaluator → Prolog constraints.pl path (no Python fallback).
  • source verification.orchestration.coordinator → max_attempts: int = 3 ctor default :60.

Cluster summary table

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks