Orchestrata Ultimate › Plans › spec-implementation-gaps

Spec implementation gaps — evidence-graded audit (session 227)

Source: plans/spec-implementation-gaps.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-12 · 299 lines.

Prose pending.

Plan text

Spec implementation gaps — evidence-graded audit (session 227)

Question: which specs have genuinely unimplemented requirements? Method: check-traces + generate_rtm_evidence.py over specs/ vs src/, then code-level audits of the suspect clusters (4 parallel auditors + direct probes). UNLINKED (no @traces, no test citation) is the starting signal only — every verdict below is code-verified, not inferred from trace absence.

Headline numbers (RTM evidence, 2026-07-12)

• 4,882 checklist requirements across 42 REQ-namespaced specs.
• Grades: 104 CITED (test-evidenced), 2,549 DECLARED (@traces only), 2,229 UNLINKED, 6 GHOST (traces to nonexistent reqs).
• 19 further spec dirs have NO requirements checklist at all — invisible to check-traces and the RTM (see §4).
• SPEC-100 has a checklist in a nonconforming format (FR-NNN checkboxes, no REQ-100-* ids) — also invisible (see §5).

1. Per-spec UNLINKED distribution (worst first, code-verdicts in §2-§3)

GHOST rows (6): RESOLVED 2026-07-12 — all six verified as RTM-scanner false-positives on prose/fixtures, NOT real traces: three are intentional tooling-test fixtures (test_ast_trace_audit / test_ast_check_traces / test_generate_rtm_evidence), REQ-001 is docstring prose about merge keys (test_kg_sync_extended), REQ-024-044 is a comment documenting a descoped requirement (test_di), REQ-1 is a spec-writer generator fixture. No @traces decorator references a nonexistent req (check-traces: 0 dangling, 0 malformed after the session-227 purge). GHOST=6 in the RTM is stable, characterized noise.

2. Code-verified verdicts — suspected-missing cluster (021 / 020 / 000)

Framing discovery: SPEC-020 and SPEC-021 are as-built docs of harness code that lives OUTSIDE this repo (~/.opencode/plugins/orchestrata.ts 3,236-line TS plugin; ~/.claude/hooks/*), targeting the FROZEN old monorepo. Neither was ported into src/.

SPEC-021 opencode-integration — ~95-100% genuinely unimplemented in-repo

No plugin runtime, no lifecycle hooks, none of the 24 tools, no agents. Also missing everywhere (not just the plugin): Redis-queued KG sync + conversation-turn capture (Conversation/ConversationTurn node writers exist nowhere in this codebase), deception detection, memory search, and a project-scoped arbitrary-Cypher query CLI (scripts/neo4j_query.py is canned subcommands only). ~30-40 reqs have in-repo backends the plugin would call (session CLI, kg_context.py), but zero reqs are satisfied as specified. Recommendation: re-scope or retire the spec — porting it verbatim would target frozen paths.

SPEC-020 claude-code-harness — ~45% genuinely missing

• Implemented-untraced in-repo (~40%): session lifecycle mechanics (session/controller/_git.py:123 tagging, _changelog.py), context.json persistence (session/context.py, SESSION_LIMIT=3), the review-check LOGIC (verification/review/checks_structure.py stub/nesting/god-object detectors), RTM generation (scripts/generate_rtm_evidence.py), KG context injection (context_injector.py). Note: session_auto_start_hook.py (515 lines) exists in-package but is NOT wired in .claude/settings.json.
• Genuinely missing EVERYWHERE (~105-115 of 240): the entire deception- detection subsystem (stop_auditor/transcript/subagent — deleted even from ~/.claude), tool-call audit trail (JSONL + archive), read throttling, the Stop-event error GATE (checks exist, the enforcing hook doesn't), remember/search as repo-owned code.
• Silent hazard: the spec'd safety interceptors (intercept_bash/write/edit.py) exist user-globally but are UNWIRED in ~/.claude/settings.json — they enforce nothing today.

SPEC-000 orchestrata-cli — ~40-45% genuinely missing + FALSE-DECLARED stubs

• The orchestrata command tree is largely a stub shell: init, stop, status, config-team, governance check/log/policies, verify, knowledge, agents list/health all print "not yet implemented" or STATIC FAKE output (kernel/cli_main.py:457-620). orchestrata status always reports "stopped/none" even with a live runtime.
• Inverse defect — false traceability: those stubs carry @traces claiming REQ-000-021..024/042/044/062/064/086 → DECLARED grades on unimplemented commands. The exact opposite failure mode of untraced-but-built.
• No orchestrata intent/respond CLI ingress to the fully-working SPEC-003 intent engine (only session pipeline + gateway reach it).
• may_complete/2 (prolog/completion.pl:16-26) lacks the REQ-000-091 downstream_volk_not_blocked gate. No --json output (096). U:Vz reviewing volk (106) doesn't exist as a volk.
• Implemented-under-a-different-CLI (~35%): session commands + approve/reject live in the separate session entry point, working and partially cited.

3. Code-verified verdicts — built-but-untraced clusters

SPEC-007 (governance + verification) — mostly trace debt, five real gaps

The governance package is genuinely built (policy engine, ethical boundary, constitutional enforcer, escalation, audit, safe mode — all on real Prolog + Redis). Real gaps, code-verified:

• RBAC — no role model anywhere in governance/ (permission checks are volk-id/action based only).
• FR9 context validation — the spec'd pre-dispatch context-completeness gate has no implementation.
• Notifications — no governance notification channel (violations land in Redis audit only; nothing pushes to gateway/UI).
• Analytics — no aggregation/reporting layer over the audit stream.
• Kernel resource enforcement — resource_access.py decides, but the kernel never consults it when spawning/scheduling volks.
• U:Vfy verification subsystem (007b): entirely UNWIRED in production — verification/ (63 modules, tested) has zero production construction sites; nothing in kernel/gateway/pipeline instantiates a VerificationAgent. The biggest built-but-dead body of code in the repo.

SPEC-002 (postman) — checklist inflation confirmed, plus real gaps

postman/ (30 modules) implements the core routing/delivery/queue surface; most UNLINKED rows are trace debt. Real gaps:

• Fake governance check — postman/governance.py:356-382 _query_prolog is a placeholder that always returns False; postman-side Prolog governance never actually evaluates.
• No topic/subscription persistence — TopicRegistry/SubscriptionManager are in-memory; restart loses all subscriptions.
• Shallow observability — metrics collector exists but no export surface.
• No WAL — the spec'd write-ahead log for delivery guarantees is absent (guarantees rest on Redis persistence alone).

SPEC-014 websocket-gateway — ~78% of UNLINKED genuinely missing/deficient

97/142 MISSING, 14 PARTIAL, 13 implemented-untraced, 10 RETIRED (workflow 076-080 + config 081-085 — exactly 10, not ~15), 8 statically unverifiable. This is a real implementation gap, not tracing hygiene. Top items:

1. Subscription filters are dead code gateway-wide — Subscription.filters stored (models.py:135) but never read (pubsub.py:177-206, protocol.py:115-159); filters are accepted-and-ignored across CXP/volk/session streams.
2. Auth schemes absent — JWT only; no static-token or HMAC (13 reqs); scope hardcoded "read_write"; wrong error surface (close 4001, no -32001).
3. Zero Prometheus metrics — no orchestrata_gateway_* registrations; /metrics emits an empty default registry. /health always 200.
4. No heartbeat/presence/per-client buffering/rate limiting/batch RPC; config.py:162-168 ping fields are dead.
5. Publisher-less streams — session.events, session.facts, volk.lifecycle are subscribe-confirmation stubs; publish_lifecycle_event (handlers/volk.py:427) has zero call sites.
6. Ring buffer: Streams+MAXLEN (not spec'd sorted set), no 24h TTL, replay has no since/until/filter params, sequence counters reset on restart.
7. Suspect CITED: REQ-014-101/102 (Prolog gateway_access/2) — auth never calls Prolog.

SPEC-015 plugin-sdk — ~84% of UNLINKED genuinely missing

~94/112 MISSING. "Phase D done" holds but is narrow: Volk projection + git-install are real; everything downstream of context.register_X() is inert:

1. Never wired into kernel boot — no PLUGIN_MANAGER ComponentType, zero PluginManager hits in cli_main.py; plugins never run outside tests.
2. Tool/provider/event-bus/channel registries are Protocol-only stubs — IToolRegistry/IProviderRegistry have no concrete impl and are never injected.
3. Plugin Prolog rules never loaded — registration.py:254-274 appends filenames to a list, nothing consults them. False DECLARED trace (REQ-015-063 → orchestrator/engine.py:262) masks this.
4. No audit trail (:PluginAudit zero hits; 203/206 falsely trace to knowledge_graph/versioning.py), no CLI/ops surface, no config system, hot-reload reuses cached manifest.
5. Live defect: VolkRegistry has no unregister and context cleanup only clears local lists — a stopped plugin's volk alias stays dispatchable.

SPEC-018 fireflies — 100% of UNLINKED genuinely missing as features

All 58 are user-observable gaps: no fireflies gadget exists in vizint-ui (app.js registers 12 gadgets, none Fireflies), no SessionLinker, no settings UI, no test_connection, no fireflies.health/.sync RPCs. What exists (client + 4 gateway RPCs + MCP tools + FirefliesKnowledgeSync) is the backend US-006 slice only — and sync_recent() has zero production callers. Worse than the RTM shows: ~12 of the 22 DECLARED reqs are false traces (memory/ hot_store.py:37-188 claims REQ-018-015/026-029/041/044/046-049; ambiguity_resolver.py claims 065/078/079 — all unrelated code). Live bugs: fireflies_client.py:312/:172 report page size as total; :375 maps GraphQL questions → "decisions".

SPEC-016 vizint — 61% real JS (tracer-blind), 21% missing, 10% unwired

148/243 IMPLEMENTED in vizint-ui/ JS (matches the 017/022 ruling); 52 (21%) genuinely MISSING; 25 (10%) built-but-unreachable → 31.7% deliver zero value today. Top items:

1. NotificationCenter (769 LOC) + parts of KG Explorer fully built but unwired — absent from app.js gadget manifests and every layout profile; subscribes to governance.events/kernel.events topics the backend never publishes.
2. Personal-infra gadgets (Flashcards/Iqama/Time-to-POI/Chronus/Settings/ Embed) + Email gadget don't exist despite the checklist claiming 100%; PrayerTimes provider is a stub with no consuming gadget.
3. The human checklist is wrong in both directions — claims done things that don't exist AND marks the real CSV/JSON export undone. Don't trust it.
4. Mistraced DECLARED: 109-111 (thread dropdown), 164 (RTM viewer gadget), 182 (escalation chain) — no code.
5. Live bugs: lib/bus.js wildcard subscription registers the literal string 'vizint:' and can never fire; GadgetBridge is dead code (never instantiated); dev-auth demo JWT hardcoded valid to 2036.

4. Checklist-less specs (19 dirs, invisible to the tooling) — AUDITED

Genuinely missing code — the entire Claude-Code-parity tool wave, 9 specs (063-071): REPLTool, git-worktree tools, LSPTool, PowerShellTool, RemoteTriggerTool, ScheduleCronTool, TaskStop/TaskOutput tools, team-RBAC tools, MCP-resource tools. Zero code for any of them (grep-verified); the registry stopped at the 13 volk tools. These are exactly the "14 missing tools" SPEC-062's own gap analysis enumerated. Several read as imported boilerplate (066 targets Windows; 070 dated 2024, generic RBAC prose) — the real decision per spec is build vs formally archive.

Partially built (5):

• 052 Todoist — UI-only: vizint-ui/gadgets/todoist/gadget.js (543 lines, direct REST v2); NO Python backend sync engine, webhooks, HMAC, or conflict resolution anywhere in src/.
• 060 PAM++ — core real (kernel/pam/enforcer.py, governance/cache.py, dynamic_policy.py hot-reload, monitoring.py); missing the ops layer: HTTP-451 governance middleware (0 hits), Prometheus metrics, WS policy push.
• 061 CheetahClaws volk — CC-command→CXP bridge exists (integrations/cheetahclaws/) but @traces SPEC-050; CC TUI not vendored. Duplicate of 050 — fold, don't checklist.
• 099 Security & Auth — JWT session auth exists (gateway/auth.py TokenAuthenticator); NO API-key auth, NO MFA/TOTP (0 hits repo-wide), no encryption vault, no mass revocation. Plus the recorded dev-auth accepts-any-token caveat. Most consequential partial of the five.
• 101 Ultimate Architecture — umbrella; realized by the repo except where children gap (CC TUI, Todoist backend). No own checklist needed.

Effectively done, just untracked (2): 023 goal-system (models + GoalMixin

• CLI + stop-gate all live; only the audio-alarm sub-feature unverified) and 053 OpenRouter provider (adapter + router + cost tracking + per-volk routing, tested; spec's model list stale). Checklists here are pure RTM win.

No checklist warranted (3): 051 (deprecated → SPEC-016), 101 (umbrella), 061 (duplicate of 050).

098 API standards — decision needed: concrete MUSTs (versioned /api/v1, error-code taxonomy, Deprecation/Sunset headers, gateway rate limiting), none applied anywhere; adopt (all-red checklist) or archive as boilerplate.

Note: SPEC-099 requirements.md generation is already a recorded backlog item. The auto-generated compliance-audit.md files in these dirs are class-name-diff noise, not usable verdicts.

5. SPEC-100 logging-exception-handling — double defect (direct probe)

1. Format outlier: its checklist uses FR-NNN section checkboxes with no REQ-100-* ids — invisible to check-traces AND the RTM generator.
2. Headline requirement unimplemented: the spec mandates structlog with JSON output; structlog appears nowhere in pyproject.toml or src/orchestrata_ultimate/core/logging.py (stdlib logging only).

6. Known structural blind spots of this audit

• Python-only tracer: JS deliverables (016/017/022/051 in vizint-ui/) never show as traced; verdicts there come from code inspection, not the RTM.
• DECLARED (2,549) means an @traces stamp exists — it does NOT prove behavior; the false-confidence-test audit (recorded memory) showed stamps can decorate unwired code. CITED (104) is the only test-evidenced grade without running --verify.
• Systemic FALSE-DECLARED confirmed this pass (auditor-4): SPEC-018 (~12 of 22 DECLARED trace to unrelated memory/hot_store.py + ambiguity_resolver code), SPEC-015 (063/203/206 → unrelated modules), SPEC-016 (109-111, 164, 182 → nonexistent JS behavior), SPEC-014 (CITED 101/102 suspect — auth never calls Prolog). Pattern matches add_traces_from_evidence.py matching mechanically, not semantically. The RTM is optimistic in the DECLARED direction; a false-trace purge pass is warranted.
• Trace-debt tier (§1 "others") was NOT deep-audited this pass; the trace-remediation program (plans/trace-remediation-plan.md waves) is the existing vehicle for it.

7. Synthesis — where the real work is

1. Tool wave 063-071 (§4): zero code, highest functional leverage — ACTIVE, see plans/tool-wave-plan.md (user ruling 2026-07-12).
2. Gateway hardening (014): dead subscription filters, absent auth schemes, zero metrics, publisher-less streams — largest single real-gap cluster in built code.
3. Plugin SDK activation (015): wire PluginManager into kernel boot + concrete tool/provider registries, or the whole package stays test-only.
4. U:Vfy wiring (007b): 63 tested modules with no production caller.
5. False-trace purge: fix the mechanical @traces mis-stamps (018/015/016 clusters + GHOST rows) so the RTM stops overstating DECLARED.
6. Live defects found in passing (need owners): VolkRegistry no-unregister (stale plugin volk stays dispatchable); fireflies_client total=page-size (:312/:172) and questions→"decisions" (:375); vizint lib/bus.js wildcard never fires; postman _query_prolog always-False placeholder; plugin/governance.py docstring falsely claims Prolog backing.
7. Retire/rescope decisions for the user: SPEC-021 (retire — targets frozen paths), SPEC-098 (adopt or archive), 066/070 (archive per tool-wave plan), SPEC-014 workflow/config rows (formally retire the 10 reqs).

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks