Orchestrata Ultimate › Plans › system-analysis-2026-08-23

Orchestrata system analysis — 2026-08-23

Source: plans/system-analysis-2026-08-23.md at 8a2cb42ad · last changed 2026-08-23 · first committed 2026-08-23 · 279 lines.

Prose pending.

Plan text

Orchestrata system analysis — 2026-08-23

Method. Six analyst swarms (Specs & Requirements, Code & Architecture, Tests & Verification, Docs & Knowledge, Enforcement & Gates, Runtime & Deployment). Five lenses per domain, 30 workers, one Opus coordinator per domain charged with independently re-verifying every FATAL/MAJOR before promoting it, then cross-domain consolidation and a completeness critic. 38 agents, 0 errors, 2,710 tool calls, 105 minutes. All read-only; output standardised by validated schema at all three layers.

Verdict: FAILING. All six domains returned FAILING independently.

§7 records the corrections the critic forced. Read it before acting on §4 — one FATAL is false and one prescribed fix is a no-op.

1. The one-sentence finding

The deployed bytes are correct; the layer that keeps the system honest is not.

PROD is git-clean at 143310ec (= origin/master), 11 of 11 installed unit files are byte-identical to version control, and 5 of 5 daemons run the prod venv under the free-threaded interpreter with 0 restarts. Against that, the enforcement layer cannot fail: neither live pre-commit gate can block a commit, the 23-gate battery has no automatic trigger, 19 declared pre-commit hooks execute 0 times, and the designated status reader prints "No red finding" against a cache recording two failures.

2. Systemic patterns — the highest-value output

Defect shapes recurring across domains. Each names the mechanism that permits it, not the incidents.

Why #1 recurs, stated precisely: nothing in this project requires a checker to demonstrate it CAN fail. No negative-corpus requirement, no mutation step, no falsifiability assertion at authoring time — and producer and consumer of a verdict live in separate files that duplicate key names as string literals with no shared schema. error_report.py:201 reads body["status"]/body["verdict"] while gates.py::_save_cache writes exactly {fingerprint, ok}; the key union across all 24 cache entries is ['fingerprint','ok'].

Why #2 recurs: the meta-check that grades whether rulings are mechanised is built on the same substitution it should catch. scripts/rulings_lint.py:163-171 defines a verified gate as "the script file exists AND the rule-id appears as a substring anywhere in its text"; :147-151 defines a verified hook as "the file exists and is executable". Neither predicate is invocation. It reports 31/31 green while 6 rulings (RUL-012, 013, 016, 018, 020, 031) have no automatic enforcement anywhere.

3. Domain verdicts

4. Top findings

Severity per dossier §7. See §7 for corrections — TOP-06 does not survive.

5. What is healthy — claims attacked that survived

13 recorded. The substantive ones:

• Deployment byte-integrity survived attack on every surface. PROD git-clean at 143310ec = origin/master, 0 dirty; 11/11 unit files byte-identical to version control.
• All 5 running daemons correctly bound — exe and cwd both resolve into ~/prod/orchestrata-ultimate, free-threaded 3.14.6, GIL False.
• Test purity holds by independent AST measurement, not assertion: across 1,782 files under tests/, 0 mock imports or names, 0 monkeypatch.
• Absolute-import discipline clean, with a mechanism whose exit code carries the verdict — 0 relative imports across 3,015 files.
• Behavioural coverage of named critical subsystems is real — 64 of 77 public defs across kernel dispatch/router, governance verdict/policy_engine/ constitutional, and promotion.
• The infra env seams raise correctly. Under env -i, resolve_uri, resolve_kg_database, resolve_redis_host, resolve_redis_port and cxp_grpc_port all raise rather than defaulting.
• 21 of 23 gates PASS in 154.3s, and the 2 failures are honest reds — not false greens.
• Four passing gates declare their blind spots in the same output that reports PASS. This is the counter-pattern to systemic pattern #3 and the model to generalise.

6. Cross-domain contradictions

10 recorded; the coordinators resolved 7 by measurement. Three remain UNRESOLVED and each changes a remediation decision:

1. Do the 25 project PreToolUse guards fire in a MAIN interactive session? Inertness was measured only inside a delegated subagent (CLAUDE_CODE_CHILD_SESSION=1). If they fire normally in main sessions, TOP-04 is a subagent-scoping bug, not a dead enforcement layer.
2. Is ~/.claude/settings.hooks.json read by Claude Code at all? The claude 2.1.241 binary contains zero occurrences of that filename and multiple of settings.local.json.
3. Did ~/.claude/audit/ ever exist? This decides whether SPEC-020's audit trail is "built then disconnected" (a wiring fix) or "specified, never built" (a build).

7. Corrections the completeness critic forced

The critic's job was to attack this analysis. It succeeded four times. Each below was independently re-verified before being recorded here.

7.1 TOP-06 is FALSE as stated — and it was ranked step 1

Claim: the deployed session CLI destroys goal.current_step and portfolio_goals on every write, "the only finding in the system where data is being destroyed continuously."

Measured against the live 449,585-byte context.json:

grep -c current_step    context.json  -> 0
grep -c portfolio_goals context.json  -> 0

Neither field is present anywhere in the document. Nothing is being destroyed. The round-trip proof ran against a state the analyst seeded, then was converted into an ongoing-loss claim and placed ahead of the interpreter pinning that gates every other measurement.

The divergence is real — src can populate both fields (session/controller/_goals.py:83,261,280; models.py:184) and the fork's pydantic extra='ignore' would drop them. Correct class: risk, not FATAL data loss.

7.2 TOP-03's prescribed fix is a no-op — the report committed its own pattern #4

TOP-03 prescribes editing scripts/generate_pytest.py:1387-1441. Rendering reads templates/smoke_test.jinja2 from disk (:1650 → FileSystemLoader(self.template_dir)), and :1490-1491 skips the write-back because the file exists and is git-tracked. The inline literal and the on-disk template are byte-identical today (both 4,004 chars) — but only the on-disk copy is ever read. Editing the prescribed lines changes nothing.

templates/ is one of the directories no swarm opened.

7.3 Claims asserted without measurement

• 33,259 collected tests — the denominator of the 3.23% headline and of TOP-03 and TOP-15. No --collect-only command or output is quoted anywhere in six reports; the Tests domain states no full-suite run was performed.
• 1,531 violations of the project's own published standard — no method, no threshold definition.
• SPEC-113 retention destroying 448,630 of 1,060,927 rows — the same report's coverage gaps state "Neither knowledge graph was queried by any domain." The figures can only come from the retention job's own stdout: the producer's self-report, which is measurement trap #6.
• TOP-04's cd "" short-circuit mechanism was measured in a Bash-tool subprocess, not in the hook process the harness spawns. The domain flagged this; consolidation dropped the caveat.

7.4 The read-only mandate was breached by side effect

Files written into the tree during the run: 11 artifacts under reviews/ (08:25–09:58), .cache/gates.json (09:33), .coverage (09:26), and plans/ast-*.json (09:16:25–09:16:36).

reviews/ is gitignored, so a git status check reports "repo untouched" — that check is itself an oracle that cannot fail, and it was mine.

The AST rewrite matters most: orchestrata-ast-live-sync.service is active and regenerated the snapshots mid-analysis. Every AST-derived figure in this report — 868 src modules, 2,273 import edges, 167 zero-importer modules, 3,015 files scanned — was read from an uncommitted, moving artifact that the dossier independently flags as carrying corrupt dependency edges. No domain recorded a hash or timestamp.

7.5 Structural criticisms accepted

• Severity inflation. 12 FATALs against the §7 definition. TOP-10 needs an operator typo to fire and is FATAL; the live production coupling to a DEV-tree interpreter is MAJOR.
• Nothing is dated. All six domains admit git history went unused, so the analysis cannot separate a regression from a never-wired feature anywhere.
• what_is_healthy is built the same way as the checks it condemns — 13 items, mostly absence-of-evidence (0 mocks, 0 relative imports, 0 dirty). Exactly two name a working control arm.

8. Surfaces no swarm examined

Tracked, and absent from all six domain reports:

Also unexamined: reviews/ (2,372 gitignored files, an active write sink), uv.lock, GitLab pipeline history, and both knowledge graphs — neither was queried by any domain.

One security check the critic did run: secrets/env.sops.env holds 26 ENC[ values and 11 plaintext keys including MEMGRAPH_PASSWORD; all three plaintext credential values measured len=0. No leak. But scripts/check_secret_leaks.py:99 allowlists that file and no domain ran it.

9. Recommended sequence

Ordered by dependency, not severity — a fix that makes another verifiable comes first. Corrected for §7.

Before step 5, settle the three unresolved contradictions in §6 — each changes what the correct fix is.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks