Orchestrata Ultimate › Plans › audit-full-2026-07-06

Whole-project spec audit — 2026-07-06 (all 57 specs, 6 auditor groups)

Source: plans/audit-full-2026-07-06.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-06 · 239 lines.

Prose pending.

Plan text

Whole-project spec audit — 2026-07-06 (all 57 specs, 6 auditor groups)

Chain scored per requirement family: SPECIFIED → TRACED → IMPLEMENTED → WIRED → TESTED-FOR-REAL. "Wired" = constructed in production (the RTM's historical blind spot: it verified decorator existence only).

Verdict legend

CONFORMANT | IMPLEMENTED-UNWIRED (→ wiring campaign) | PARTIAL | GAPPED (spec with no code — build decision needed) | SPEC-STALE (spec contradicts reality/rulings) | ENFORCED/UNENFORCED (standards specs)

Group 5 (062–071, tool suite) — LANDED

Headline: 9 of 10 specs are pure paper. Zero code, zero @traces, zero tests for SPEC-063..071; dependencies (paramiko/croniter/pyte) undeclared. Corroborated 3 ways (grep/AST, rtm.json zero-counts, per-spec compliance-audit.md "Implementation Source: Missing").

• 062 cheetahclaws-tools-gap: PARTIAL — enabling framework (tools/base.py, tools/registry.py, agents bridge) CONFORMANT; all 14 FR-mandated tools GAPPED.
• 063 repl-tool: GAPPED + SPEC-STALE — spec.md:6 claims "Status: Production Ready" with 0% implementation. Fix the status line (user-domain edit).
• 064 git-worktree-tools: GAPPED (caution: harness WorktreeIsolation is SPEC-102's, NOT a trace target here).
• 065 lsp-tool: GAPPED (pyright_ratchet.py is a CI gate, unrelated).
• 066 powershell-tool: GAPPED. 067 remote-trigger: GAPPED (no paramiko).
• 068 schedule-cron: GAPPED (no croniter). 069 task-mgmt tools: GAPPED (harness-side TaskStop is the SDK's, not ours).
• 070 team-tools: GAPPED — largest unimplemented FR surface (34 FRs), spec dated 2024-01-15; descope-vs-build is a USER decision.
• 071 mcp-resource-tools: GAPPED. Routing: builder (net-new) per prioritized spec; framework substrate ready.

Group 2 (009–018) — LANDED

RTM blind spot confirmed empirically: 100% of entries for all 10 specs read VERIFIED (decorator-existence only); 016/017 entries additionally misfiled across specs.

• 009 volk-context-injection: PARTIAL — D9 merge executed, but payload["task_context"] is produced (kernel/pipeline_kg.py:372) and NEVER consumed; 568-line test file proves the producer only. → wirer.
• 010 memory-engine: PARTIAL/GAPPED — 3 independent MemoryEngine sites; agent-runtime + kernel-CLI paths pass memory_engine=None (0% functional memory in those deployments); embeddings are hash bag-of-words not sentence-transformers (REQ-010-036). → wirer + builder.
• 011 provider-manager: PARTIAL — OpenAIAdapter unwired (Wave A2 in flight).
• 012 pm-volk: GAPPED — goal_registry/goal_report (1,190 LOC) unwired (Wave A5) AND a second goal system: PM's context.json["goals"] (raw file I/O) vs SPEC-023's ["goal"] via ContextFileManager — DESIGN RULING needed before wiring. spec.md never mentions goals; checklist devotes 25/50 reqs to them (SPEC-STALE).
• 013 ux-volk: CONFORMANT — JS-scaffold pair resolved DELETE (not mandated by 013/000/051; duplicates vizint-ui/lib; mis-traced to deprecated 000).
• 014 websocket-gateway: PARTIAL + CRITICAL auth bypass — FIXED f8bd14c8 (validate tokens both paths; anonymous = explicit opt-in; 6 enshrining tests rewritten). Remaining: no authz/scope checks in rpc.dispatch, no auth strategies (FR-005..008), no flow control (FR-011), hollow /metrics (FR-022), subscription filters stored but not applied (FR-009), gateway constructs Neo4j/Redis directly (D12).
• 015 plugin-sdk: IMPLEMENTED-UNWIRED — VolkRegistry unconstructed (Wave A3); alias-conflict guards NEVER invoked (a plugin could hijack U:Ox); name-collision twin: kernel/volk_registry.py vs plugin/registries.py.
• 016 vizint: PARTIAL/SPEC-STALE — spec claims 31 gadgets "verified", 21 don't exist; kg-explorer + notification-center fully built (1,710 LOC), checklist-marked 100%, absent from app.js registry. → wirer + spec fix.
• 017 calendar: PARTIAL — GoogleCalendarProvider defined, never imported (P0 story half-dead); US-003 event creation 0%; gadget in no layout.
• 018 fireflies: PARTIAL — gateway RPC conformant; fireflies_sync.py + fireflies_tools.py real-but-unwired (Waves H/I); cache attribute dead.

Group 3 (019–028) — LANDED

• 019: SPEC-STALE — spec.md describes a different feature than its own checklist (authoritative: Analyst Volk = CONFORMANT); false-trace collision: orchestrator/impact modules claim REQ-019-001/002 for unrelated behavior; REQ-019-041..047 live traces sit on dead legacy modules, the real prompt's decorator is commented out.
• 020 claude-code-harness: GAPPED — flagship P0 error_gate (1,073 LOC) built, NEVER registered in settings.json Stop hooks; session_auto_start registered in settings.json but SOURCE FILE MISSING from main; zero pytest coverage (claude/ outside testpaths).
• 021 opencode: PARTIAL — 3,234-line plugin real; ~50 reqs describe absent artifacts (KG-context agents, 5 KG query tools); scripts/neo4j_query.py is a name-colliding different tool.
• 022 tui: GAPPED — textbook false-confidence: console_cli.start() traces 4 REQs, body = "not yet implemented" + SystemExit(1), and its test asserts exit_code==1 (certifies the stub). No Textual dep. Build-vs- descope = user decision.
• 023 goal-system: PARTIAL — data model/CLI/persistence real; the BDI differentiators (P0 stop-hook goal guard, audio alarm, re-planning) 100% unimplemented; no requirements checklist (placeholder REQ ids).
• 024 DI: CONFORMANT (D4 executed; checker enforces roots).
• 025 drivers: CONFORMANT except ClaudeCodeDriver (Wave A1 in flight; NOTE: checklist never names its dispatch key — A1 must add the REQ).
• 026 compliance: CONFORMANT (D2 executed; checklist text stale, doc-only).
• 027 junior-volk: CONFORMANT — strongest spec in group.
• 028 llm-client: PARTIAL — D10 confirmed OPEN both halves ("default" sentinel absent — code raises; model-table spec/tests divergence).

Group 6 (072–106) — LANDED

• 072 tool-search: GAPPED (registry substrate exists; no search).
• 097 testing-standards: PARTIAL — xdist/architecture/pyright gates real; purity/isolation/parallel-safety audit tools UNREACHABLE from CI; flat 92% coverage gate ignores tiered thresholds; 92/103 RTM entries share one req id (blanket-tagging).
• 098 api-standards: UNENFORCED — FastAPI docs_url=None contradicts FR-021; no envelope/headers/CORS/rate-limit conventions anywhere; no gate.
• 099 security-auth: GAPPED (~38/40 FRs) — the CRITICAL bypass lived here (fixed f8bd14c8); jwt_secret defaults "dev-secret" with no prod guard (open item); no bcrypt/MFA/AES-256/refresh/lockout.
• 100 logging-exceptions: UNENFORCED — UltimateBaseError doesn't exist; no structlog/OTel; 714/987 RTM entries share one req id.
• 101 ultimate-architecture: UNENFORCED/SPEC-STALE-leaning — no @traces; checker enforces a different doc's rules; Todoist absent entirely.
• 102 volk-test-harness: CONFORMANT — CI matrix bug: parallel VOLK matrix declared but $VOLK unused (3 identical full runs); FR-015 aggregate pass-rate gate approximated per-trial only.
• 103 tool-calling-refactor: CONFORMANT + wired (intent_broker routes REFACTOR); trace tagging partial (several REQs untraced/tagged to 011).
• 104 openspec: IMPLEMENTED-UNWIRED — real+tested, zero production callers; REQ-104-001 traced to the wrong function.
• 105 ontology-domaingen: CONFORMANT (CLI-wired per its own DoD).
• 106 context-semantic-layer: PARTIAL — FR-001 landed (canonical schema); REQ-106-007 NOT done (context_injector still uses TRACES/TESTS non-canonical rels at :785/:970/:1108); kg_schema check not wired into CI; W8.4 selector/packer still worktree-only; other writers unmigrated.

Group 1 (000–008) — LANDED

• 000 orchestrata-cli: GAPPED — 367/367 RTM VERIFIED; ~8/10 user stories are literal "not yet implemented" stubs (init/stop/status/team/agents/ governance/verification/kg/session). Real intent/review lives only in the standalone pipeline_cli.py REPL, never under the orchestrata command tree. @traces on stub bodies; tests assert echoed strings.
• 001 kernel: PARTIAL — US-006 Workflow Orchestration is 85 RTM VERIFIED but has ZERO DAG/YAML code (REQ tags redistributed onto unrelated modules after workflow_engine.py's correct deletion) — worse than a gap, it misrepresents. D17 kernel-composition unexecuted; MessageDispatcher architecturally dead for live traffic (Postman.route_sync bypasses it).
• 002 postman: PARTIAL→GAPPED — delivery.py:357 _deliver_to_recipient is a fake-success stub (sleep→DELIVERED); admin.py:87 disconnect_agent twin (always True). 16-module engine layer unwired to itself (3 non-interop sqlite silos); governance.py never calls the real governance package. Wave C must fix the stub + pull governance forward BEFORE absorbing.
• 003 intent-broker: IMPLEMENTED-UNWIRED — intent_broker.py 0 prod imports; live path is the narrower pipeline_runner. US-004 routing drift (static map, not Prolog). Templates/workflows orphaned below the dead broker.
• 004 cxp: PARTIAL + CRITICAL(shared) — cxp.pl was never ported (FIXED 57b552c3, PrologBridge now loads it). CXPPacket unhashable (eq no hash); upgrade_packet() doesn't exist despite VERIFIED. review-*.md cites fabricated evidence. ws_client/ws_server dead (gRPC is real).
• 005 multi-agent: SPLIT — entire agents/*.py spec-literal runtime (15 modules) dead; live coordination is kernel/pipeline.py volk singletons. US-005 constitutional never gates real spawn. swarm/coordinator.py mis-traces US-005 while implementing US-009.
• 006 session: PARTIAL — local context-file/archive/integrity/CLI half CONFORMANT+wired (the daily session CLI); the namesake Redis+Neo4j+ Prolog dual-storage architecture 100% unwired (0 constructors).
• 007 governance (PAM++): GAPPED + CRITICAL — 3 of 4 .pl files never ported (FIXED 57b552c3). But enforcement STILL no-ops: production GovernancePrologClient(PrologBridge()) calls bridge.query/query_bool which don't exist on the cxp bridge, and that bridge never consults the governance rules — every constitutional/ethical check fails into its safe default (harmful actions pass). Builder dispatched 2026-07-06 to wire real enforcement + un-enshrine the fail-open tests (test_ethical.py:test_check_default_all_pass et al). test_prolog_gate asserts fail-OPEN, contradicting the closed-world mandate (separate, deliberately-staged; not bundled).
• 007b verification: IMPLEMENTED-UNWIRED (worse: wrong) — 63 modules/7,915 LOC, ZERO prod construction. 4 model wrappers hardcode status=verified; SupervisorAgent hardcodes approved=True; ConstraintEvaluator stores a prolog_bridge it never uses. ~1,900 LOC duplicates qa/codedirector. Wire order: consolidate into qa/codedirector first, fix model wrappers, THEN wire VerificationCoordinator post-QA. (Wave B / D16.)
• 008 knowledge-graph: PARTIAL — core sync/context/query wired; 12-module analysis layer + Neo4jSyncer (39KB) orphaned (gateway hand-rolls Cypher). D12 confirmed concrete: :Session keyed 4 incompatible ways across writers (session_id/connection_id/id/seq); :TestRun CREATE-not-MERGE duplicates. Canonical merge-key ruling needed before further KG wiring.

Group 4 (029–061) — LANDED

• 029 codedirector: PARTIAL — self-review loop never calls run_deterministic_review() (RTM REQ-029-062..066 false-VERIFIED); MAX_SELF_REVIEW_CYCLES spec=99 vs code=3.
• 030 architect: PARTIAL — functionality real+wired but 0/45 REQ-030 ids traced (all @traces point to spec=000/005); context_builder/project_init dead duplicates (DELETED in Wave A8).
• 031 qa: PARTIAL — _parse_review_verdict failed OPEN to PASS (FIXED f87472c1 + a8819e7d; contradicted TEST-031-030). REQ-031-054 120-line cap is prompt-only, unenforced.
• 032 concierge: PARTIAL + CRITICAL — gateway chat can bypass the real ConciergeVolk (3s gRPC readiness window → canned "Received:" fallback); forced-finalization emits zero Prolog facts; shared instance keeps session state on self with no thread_id keying (concurrency risk).
• 050 cheetahclaws: PARTIAL + CRITICAL — multi-agent edit review is a facade (no volk handles "review_edit" → every edit auto-approves); tui.read was unrestricted host file read (FIXED f87472c1, workspace containment); checkpoint/rollback 0% built; 6 of 40+ commands exist.
• 051 vizint-ui: SPEC-STALE (correctly deprecated, superseded by 016); e2e test.skip(!gatewayUp) can mask a dead gateway as green.
• 052 todoist: GAPPED — spec.md claims "Complete", ~0% of server sync engine exists; only a client-side browser widget; todoist-api-python declared-but-never-imported (dead dep).
• 053 openrouter: PARTIAL — routing/adapter/cost real+wired (OpenRouter IS registered, unlike OpenAI). FR-009 key mgmt 0%; FR-006 budget alert computed, never dispatched; no REST surface.
• 060 enhanced-pam++: GAPPED/SPEC-STALE — zero @traces, zero RTM; the overlapping capability is entirely SPEC-007's. Middleware/451/degradation modes/Prometheus absent. kernel/pam + kernel/audit are TRUE orphans (0% coverage real, Wave F1). Looks like a supersession candidate (user call).
• 061 cheetahclaws-volk: GAPPED — 0 traces; "import CheetahClaws" premise false (codedirector is independent). Rescope decision (user).

Cross-cutting (systemic, escalate)

1. RTM-VERIFIED ≠ correct anywhere — every spec near-100% VERIFIED while conformance ranges GAPPED..PARTIAL. Decorator existence only. Worst: 001 US-006 (redistributed tags), 000 (VERIFIED vs stubs).
2. Prolog files never fully ported (007: 3 files, 004: cxp.pl) — FIXED 57b552c3; but 007 needs the enforcement wiring too (builder in flight).
3. old-runtime-vs-volks twin pattern recurs (005 agents/*, 003 broker, 007b verification, 008 KG analysis) — consolidation, not wiring, is often the right first move.
4. compliance-audit.md uniformly unusable (generator boilerplate); several review-2026-03-25.md cite fabricated/stale evidence.
5. checklists/requirements.md self-contradict (all [x] but 0% summary) in 001, 002 — unreliable for gating independent of code inspection.

Fixes landed THIS audit (same-session remediation)

• CRITICAL gateway auth bypass (both JSON-RPC + handler paths) — f8bd14c8
• CRITICAL tui.read unrestricted file read — f87472c1
• HIGH QA review verdict fail-open → fail-closed — f87472c1, a8819e7d
• 5 Prolog rule files ported (007/004) — 57b552c3
• kernel session-tracking dead import fixed (SPEC-022) — via kernel merge
• Wave A wiring (ClaudeCodeDriver, OpenAIAdapter, VolkRegistry, PMVolk goals) + 6 dead-twin deletions — b44426a1
• IN FLIGHT: SPEC-007 governance enforcement wiring (builder)

Still needing USER decisions

• 9 paper tool specs 063–071 + 052 todoist + 022 tui: build vs descope
• SPEC-063 "Production Ready" status lie, SPEC-070 34-FR from 2024
• SPEC-060/061 supersession (retire into 007, or author real traces)
• Canonical KG merge keys (008 / D12)
• SPEC-012 vs 023 two-goal-system reconciliation (before A5 deepens)

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks