Orchestrata Ultimate › Plans › analysis-2026-07-06-findings

Deep-Analysis Findings — 2026-07-06 (Session 199)

Source: plans/analysis-2026-07-06-findings.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-05 · 198 lines.

Prose pending.

Plan text

Deep-Analysis Findings — 2026-07-06 (Session 199)

Evidence-based snapshot of code, architecture, structure, features, process, and hygiene, produced by an Opus orchestrator + 5 read-only agents (hygiene, AST metrics, architecture, feature inventory, spec-gap triage) + direct measurement. Companion execution plan: plans/swarm-remediation-plan-2026-07-06.md. Nothing in this doc is a vibe; every claim was measured or cited this session.

1. Headline verdict

The system is substantially real, not vaporware: intent→volk pipeline proven by 344 trace files (114 with live llm_call events, e.g. 34s U:Cp stage latency in pipeline_output/traces/pipeline_3408344e.jsonl); 7 working provider adapters behind configs/spike-profiles.yaml routing; vizint-ui with executed Playwright artifacts; docker(4 services)+incus composition. Architecture is a governed layered monolith with a CXP/Postman message spine — the 6-layer import model in scripts/architecture_checker.py holds at 622 files / 1 live violation (from 91 historical). The debt is duplicated organs, not bad bones (port residue: twin runners/routers/policy stacks), plus broken truth instruments (RTM validation, pyright baseline, projectstatus.md) that misreport project state.

2. Scale & quality metrics (measured 2026-07-05/06)

• 671 src files / 163,477 LOC; 1,197 test files / 349,577 LOC; 21,933 test functions (19,196 handwritten + 2,737 generated). Test:src = 2.14:1.
• Coverage 98% overall (0 modules <90%), verified contamination-guarded (S198).
• Typing split verdict: untyped ratchet fully paid off (186 → 0 functions missing hints). But full-tree pyright (pinned 1.1.410) = 631 errors, 0 warnings — not the "36" in circulation (that was a scoped/changed-files number). Breakdown: 286 reportAttributeAccessIssue, 107 reportOptionalMemberAccess, 96 reportArgumentType, 83 reportGeneralTypeIssues. By package: session 165, kernel 135, agents 120. Top files: qa/_testgen.py 27, kernel/queue_manager.py 25, session/kg_sync/_stage_code.py 25, qa/_dep_install.py 24. No pyrightconfig.json exists; naive pyright src mis-binds the env (gives 650 w/ import noise).
• Complexity: 194 functions CC≥10 (worst core/threading/test_executor.py:: _attribute_from_junit = 35); 21 modules >800 LOC (worst agents/volks/codedirector/review_checks.py = 1,664).
• Dead code: 153 dead-param functions — incl. all 4 verification model adapters ignoring context in verify AND generate_proof (8 sites = unimplemented interface promise); duplicate generate_rtm_markdown in BOTH core/traceability/__init__.py:392 and core/tracing/traceability.py:212 (copy-paste twins, both CC 17). 2 import cycles: knowledge_graph.anomaly_detector↔synchronizer; session.cli.achievements→main→archive→achievements.
• Test-light large packages: tools 0.69 (the AST toolkit itself!), governance 0.74, cxp 0.98 (generated suites excluded from ratios for agents/core/orchestrator/ gateway, which are effectively higher).
• Fan-in kings: core.base.traces imported by 444/671 modules (66%); cxp.packet 63; core.traceability 25; verification.types 22; core.interfaces = core.protocols = 20 each (the tie IS the split-brain).

3. Broken truth instruments (highest leverage)

1. RTM validation is false-confidence garbage: docs/rtm.json (regenerated 2026-07-06T01:40) reports 1 verified / 35 missing / 4,585 orphan / 2.8% coverage. Cause: spec-000's checklist uses non-canonical format (~112 REQ-000-* invisible) and 5 specs (009, 050, 097, 103, 104) lack checklists/requirements.md, so generate_rtm.py's parser can't see the requirements that 4,600+ @traces decorators point at → nearly everything counts as "orphan". The value-based check-traces gate gives the real picture: 241 genuinely untraced REQs + 265 dangling + 0 malformed (506 findings). Untraced by spec: 102: 96, 105: 40, 007: 23, 003: 22, 005: 18, 022: 13, 028: 13, 017: 10, 009: 5, 004: 1. Dangling clusters exactly in the 5 checklist-less specs (103: 9, 104: 8, 050: 2, 097: 1, 100: 1 by unique REQ) → parser artifacts, not code gaps.
2. Pyright baseline myth: gates check changed-files only ("no NEW errors"), so whole-tree debt grew unmeasured to 631. Needs a committed pyrightconfig + full-tree ratchet (the untyped --max ratchet pattern demonstrably worked).
3. projectstatus.md is 2 months stale (2026-04-27: "77% coverage, 16,000 tests, v1.0.0 pending") and contradicts context.json. Two authorities, one wrong.
4. ast_query_cli snapshot staleness is silent: snapshot said 618 modules / 155,085 LOC vs live 671 / 163,477 — 105 src files newer than the snapshot, no freshness warning.
5. unused query is structurally blind to the worst orphans: it skips @traces-decorated defs and string-literal-mentioned names — which hid BOTH confirmed orphan classes (below). Nearly every public symbol is decorated, so it can never find a decorated orphan.

4. Architecture findings (ranked)

1. Two competing PipelineRunners + a third engine — kernel/pipeline.py:83 ("…Sole orchestrator for the pipeline") vs orchestrator/pipeline_runner.py:109 ("executes dynamic agent stages through Postman routing") vs kernel/workflow_engine.py (992-LOC generic DAG engine). Both kernel files open with # mypy: ignore-errors. The kernel-vs-orchestrator ownership question is THE root decision; routers/policy duplication flows from it.
2. Five traceability implementations under the 444-fan-in symbol: core/base/traces.py, core/traceability/__init__.py, core/traceability_utils.py, core/tracing/traceability.py, verification/traceability.py. Drift silently corrupts the RTM.
3. Port-residue duplicate pairs: cxp/router.py CXPRouter vs kernel/router.py also named CXPRouter vs postman/router.py RoutingEngine; 3 policy stacks (governance/policy_engine.py, kernel/policy_evaluator.py, orchestrator/engine.py); postman/types.py vs postman/postman_types.py; kernel/error_handler.py vs core/patterns/error_handling.py; LRUCache ×2 (core/lru_cache.py, kernel/performance/resources.py); architect mixin near-twins (architect_context.py≈context_mixin.py, architect_project.py≈ project_mixin.py).
4. DI container (951 LOC) bypassed at real composition roots — instantiated once outside its package (gateway/server.py:252), resolved via plugin/context.py:136; gateway actually wires by setters (gateway/rpc.py set_processor:124 / set_neo4j_driver:136 / set_memory_engine:150 / set_kernel_client:172 / set_policy_engine:197); kernel/cli_main.py _boot_monolithic:250 hand-constructs a parallel world.
5. ~60 function-scoped lazy imports in agents/ satisfy the layer checker while preserving runtime L3-peer coupling (e.g. agents/volks/analyst/agent.py:246 lazily imports knowledge_graph.kg_design). Invisible to the gate.
6. Entry-point sprawl: 12 src entrypoints vs 4 docker services + incus path; kernel/cli.py vs kernel/cli_main.py vs root cli.py overlap; no shared composition root.
7. agents/volks layout non-uniform: 10 package-volks; senior_dev.py flat; analyst exists TWICE (package analyst/agent.py+parsers.py AND legacy flat mixins analyst_parsing.py/analyst_storage.py at volks/ root).
8. architecture-standards.md is a 13-line stub; the real layer model lives only in scripts/architecture_checker.py:20-68 (+ KNOWN_VIOLATIONS at :84-89, 1 codified exception: cxp.prolog_bridge→orchestrator.prolog_worker). The checker's header references architecture/layer-dependency-rules.md which does not exist. Per-package interfaces.py files still define ~43 protocols (postman/interfaces.py alone: 19) with no written placement policy.

5. Feature reality

Known orphans (tested-but-never-wired false confidence): AnalystStorageMixin (agents/volks/analyst_storage.py:15) and VolkContextInjector (knowledge_graph/volk_context_injector.py:160) — each 1 def / 0 src imports / 0 uses, each with dedicated passing test files.

6. Process & operational findings

• Context death is the recurring failure mode: S198 compacted 6+ times; during this analysis 2 auditor agents died of autocompact thrash and 1 reviewer agent misfired (0 tool calls). The new CLAUDE.md → @docs/CODEBASE_MAP.md import (1,177,408 bytes ≈ 250–300K tokens, larger than the documented ~218K) injects into EVERY agent that touches the repo — a feature-inventory agent burned 554K tokens on 12 tool calls. Mitigation shipped this session: context-safety section in all 7 roster agents. Structural fix still open: two-tier map (full for main session, header+scripts ≈25K for subagents).
• Launch-from-home hazard: claude launched from /home/mohamed means the repo's .claude/agents needed symlinking (done, documented in plans/agent-roster.md) and the project SessionStart hook (map refresh) does NOT fire. Launching from the repo root removes both hazards.
• Test-runner ceiling is shared infra, not CPU (now documented in scripts/run_tests.py --workers help): 4 workers = ~13m all-green; 10 workers = ~9.5m + 111 timeout failures against ONE Redis/Neo4j/Gateway. Real speedup requires per-worker service isolation, not more workers.
• In-flight uncommitted work on master (S198, verified complete + functional): SessionStart map wiring (.claude/settings.json + hook + gen_codebase_map.py + docs/CODEBASE_MAP.md + CLAUDE.md @import) and run_tests worker-doc changes,
  • .zed settings, rtm regen. Plus S200's .claude/agents/*.md hardening.

7. Hygiene inventory

• Ignored artifacts, plain-delete, ~130 MB: htmlcov/ 42M + coverage.html/ 41M + coverage_html/ 40M (=123M), ci_.txt 5.6M, coverage_.txt 2.1M, pipeline_output/ 1.4M (runtime output — keep traces? see plan), test_value/, .coverage.*, context.json.bak, non_empty, pycache/, node_modules/, logs/.
• Tracked junk needing git rm + commit, ~15 MB: json (root, 12M — mislabeled PostScript blob), 50× instructions-.md, 4× kimi-export-.md (1.8M), hunt_{report.md,results.json,stubs.py}, 4× test__.txt, empty spec.md, stub class_diagram.mmd/module_diagram.mmd, session-transcript dump 2026-04-19-.txt, redundancy_report.md, original_test_file.txt, tmp_scanner.py, features/'.py (misnamed conftest — pytest never loads it), plus committed src/orchestrata_ultimate/session/context.py,cover coverage artifact.
• DECIDE before removing: port_all.sh (one-off migration tool? move to scripts/ or delete); Read ux/ (space-named package holding 18K volk.py — diff vs canonical volk first); reviews/ — 359 files tracked in defiance of .gitignore intent (rules exist; files predate them) + 435 more ignored on disk.
• .gitignore gaps: queue_data/, coverage.html/, coverage_html/, 2026--.txt, kimi-export-.md, hunt_.
• 3,363 tracked files total; only 4 untracked non-ignored (all legit S198 work).

8. Conversation-history observations (S195–198 + 341-session archive)

• The Opus-orchestrates / Sonnet-builds / review-every-diff wave model is proven: coverage campaign closed 103 sub-90% files across ~60 commits and surfaced 20+ real production bugs (Prolog injection, Neo4j driver leak, route_command 100% broken, dropped Anthropic tool_use, RPCError arg-swap, stages_completed always-0…). The "fix the code, never fake the test" doctrine works in practice.
• Standing user directives currently in force: work autonomously ("keep going until all tasks done"), no AI attribution in commits, specs are authoritative, no mocks / no skips / no xdist.
• Runaway-fork incident (S198) root-caused: background slash-fork with --permission-mode auto committing in parallel; killed + daemon roster cleared; global memory saved.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks