Orchestrata Ultimate › Plans › suite-runtime-and-graph-defects-plan

Suite runtime + graph-defect remediation plan

Source: plans/suite-runtime-and-graph-defects-plan.md at 8a2cb42ad · last changed 2026-08-12 · first committed 2026-08-12 · 140 lines.

Prose pending.

Plan text

Suite runtime + graph-defect remediation plan

Date: 2026-08-12 Status: CLOSED — production-side DELIVERED; suite-side objective PROVEN FAILURE, rolled back Measure-first state: checklist completed 2026-08-12 and disarmed (.kimi/measure-first.json removed)

Final record (2026-08-12, user-directed): the suite-side program (§3, work items WI-4..WI-8) is a proven failure — the authoritative full run on the merged tree measured 455s wall vs the 445.7s baseline (+2%) and 2,250s call CPU vs 2,312s (−2.7%), both inside the ~24% run-to-run noise band. No suite-level win exists. Per user directive the five suite-side merge commits were reverted on master (c7043047, 3097834c, f44c64a0, b55077e3, 8bafe038). Kept: the production-side defect fixes (D1, D2, D2b, D7 — merge 429fd08c, merge 5d3ed30c, purity fix db0609f9) and this documentation. The module-level "savings" measured standalone during the swarm did not transfer to the saturated 8-lane suite; treat standalone module benchmarks as non-evidence for suite wall claims from now on.

This plan contains only adversarially-reviewed numbers. Two claims were corrected and one projection was withdrawn during the first adversarial review; a second independent review (2026-08-12, §6b) corrected four more and left the measurement core intact. Anything not marked WEAK is reproduced-evidence-backed.

1. Baseline (measured)

Artifact: reports/run-tests-latest.json @ cf59d5c8, scope=full.

• 445.7s batch-phase wall · 32,490 passed / 0 failed / 0 errors / 0 skipped · 1,611/1,611 modules
• Test-call CPU 2,312s (65% of 3,566 lane-seconds) → packing floor 289s for this run at 8 lanes (-j 8 -b 6); see the range in the arithmetic bullet below
• Non-test lane time 1,254s. Its split is WEAK (adversarial finding): junit-cache-minus-call CPU gives a ~793s bucket that conflates fixture setup/teardown, collection, imports, and scheduling idle; the ~397s "session overhead" is a residual of two soft numbers. Both figures are directional, not exact.
• Spawn tax 51 batches × ~1.3s ≈ 66s CPU (~8s wall) — small; resident-worker elimination of it was measured and regressed: 587.4s vs 445.7s (+31.8%), artifact wt-RESIDENT-WORKERS/reports/run-tests-latest.json on branch feat/resident-workers (1,610 modules / 32,463 tests vs the baseline's 1,611 / 32,490 — a 0.08% scope difference). That artifact records rerun.count = 0, solo_retry = null, failed = 0; the earlier "4 solo-retry recoveries" claim and the "cross-batch state accumulation" cause are not evidenced by it and are withdrawn (§6b-4). The lever stays closed on wall time alone. Never on master; do not retry.
• Arithmetic: halving to ~223s is impossible by overhead work alone. The floor is 289–358s, not a single number: call CPU is a noisy quantity, and the three consecutive full runs of 2026-08-11 measured 2,312 / 2,417 / 2,862s (floors 289 / 302 / 358s at 8 lanes). 289s is the most favourable sample, not a constant. The conclusion holds at every sample — any 240s-class goal requires call-CPU redesign on top of zero overhead.

2. Defect register — production-side (adversarially verified)

D1 — MemgraphSemanticGraph.retrieve N+1 (semantic_graph/adapters/memgraph.py:512)

• Measured (scratch db on the test cluster, real 13,747 entities + real-code chunks, adversarial rerun): retrieve(top_k=5, depth=1) = 16.19s with 10,964 individual get_neighbors Bolt round-trips; top_k=10, depth=2 = 37.57s / 22,894 round-trips. _all_entities() = 0.90s; Python substring scan = 0.18s; >93% of cost is per-match round-trips (1.39ms each on this cluster).
• Root cause: full-graph MATCH (e:Entity) RETURN e per call + Python-side chunks × entities substring matching + one Bolt round-trip per matched entity.
• Also a correctness defect, undisposed: the substring heuristic matched ~2,079/13,747 entities per chunk in seeding tests — retrieve returns ~15% of the graph as "relevant context". Mechanism: _any_string_property_in_text (memgraph.py:1027) accepts any string property value appearing anywhere in the chunk, so an entity carrying a short property value matches nearly every chunk. Decide the semantics before optimizing them (§6b-3): either the replacement query reproduces this matching exactly, or entity linking is fixed first — in which case the match count, and with it most of the round-trip cost, collapses on its own and part of this item is wasted work.
• Fix: neighbor expansion as ONE batched query (MATCH (e) WHERE e.id IN $ids …-[*1..depth]-(n)) — that is where the win is, since >93% of the cost is the 10,964 per-match round-trips. Candidate selection also moves into Cypher, but not as an indexed lookup: the predicate is WHERE $chunk_text CONTAINS e.id — the entity property is the needle and the chunk text is the haystack, so no property index can serve it under any engine and the scan stays a full label scan, server-side. Expected win is round-trip elimination plus one scan instead of chunks × entities Python comparisons; do not target "tens of ms" on the strength of an index that cannot exist.
• Verify: same scratch-db method, same instrumentation; round-trips ≤ 3 per call, and wall measured and reported at identical scale against the 16.19s baseline (no wall target is asserted in advance — the server-side scan cost is unmeasured). Then scripts/run_tests.py tests/semantic_graph/ --force.
• Regression guard (required, §8-2): tests/semantic_graph/ holds 26.8s of suite call CPU and test_memgraph_adapter.py 6.6s — nothing at this scale. Land a scale fixture that seeds ≥10k entities in a scratch database and asserts the round-trip count, or the fix has no guard.

D2 — CodeIntelligenceGraph.impact_score recompute-per-query (refactor/codemod/_cig.py:401)

• Measured: real repo CIG (38,369 nodes / 58,838 edges, load 2.7s): 0.66–0.72s per single-node query (adversarial rerun agrees). Full 20-iteration PageRank computed inside the per-node method.
• Fix: lazy memoization — compute the rank vector on first query, serve every later query from it. No invalidation machinery (§8-3, corrected 2026-08-12): CodeIntelligenceGraph exposes 22 symbols in _cig.py and none mutates _nodes/_csr — the graph is constructed from a prebuilt CSR index and a workspace reload builds a new instance, so a cache keyed to the instance cannot go stale. Lazy also beats "compute once per graph build", which would pay 0.66–0.72s on every workspace load even when nobody queries impact_score.
• Also in this item (§8-4): rename one of the two colliding impact_score metrics. BlastRadius.impact_score (= |reached| / |nodes|, computed inside blast_radius, _cig.py:95) and CodeIntelligenceGraph.impact_score (PageRank, _cig.py:401) are unrelated numbers under one name, both served from the same JSON-RPC surface (server.py:277 vs server.py:343). Caching one and not the other under a shared name is how the next reader gets it wrong. BlastRadius.impact_score → reach_fraction is the smaller blast radius of the two renames (one dataclass field, _types.py:57, plus its emitters/readers — confirm with refs before applying).
• Verify: first query ≤ 0.72s, K subsequent queries ~0ms each, and a second CIG instance recomputes (proves the cache is instance-scoped, not global); run_tests.py tests/refactor/codemod/ --force.

D2b — CodeIntelligenceGraph.centrality exact Brandes (refactor/codemod/_cig.py:357)

• Measured: 340.37s per call on the real CIG (adversarial 200-source extrapolation: 322s, 6% off). Served via CodemodServer (server.py:338) and scripts/codemod.py:461 — a production query_centrality on the real repo is a 5.6-minute hang.
• Fix: pivot-sampled approximate betweenness + lazy per-instance cache (same discipline as D2, same absence of mutators). Exact mode stays opt-in for tests. Cost model: sampling k of V sources is linear in k, so k=256 of 38,369 predicts 340.37 × 256/38,369 = 2.27s — arithmetic, and it is the only part of the k≈256 choice that is. Top-20 fidelity at k=256 is an assumption, not a measurement: pick k by measuring overlap against exact on the real CIG (the 340.37s exact run already exists as the reference), and raise k until overlap holds. Do not ship k=256 on the strength of the latency number alone.
• Verify: approximate call < 5s on the real CIG; top-20 overlap vs exact on the real CIG ≥ 90% (a mid-size fixture graph cannot answer this — fidelity loss from pivot sampling scales with graph size and structure, which is precisely what a fixture discards); codemod module tests green.
• Regression guard (required, §8-2): test_analysis_passes.py + test_code_intelligence_graph.py hold 0.1s of suite call CPU between them — the suite exercises this function only on toy graphs and can catch neither the 340.37s defect nor a regression of the fix. Land a real-CIG benchmark with the fix (build once, assert the approximate call's wall band), outside the default gate if it is too slow for it, and wired into a runner that actually runs.

D3 — FactExtractor live pipeline repeated per test (gates/fact_extractor.py)

• Measured: extract+render 2.07s, engine consult 4.04s (342,018 facts, 45.8MB) per live-pipeline test; adversarial attribution: 5 live-snapshot tests hold 32.7s (96%) of the module's 34.1s call CPU.
• Fix: session-scoped shared extraction + consult via the repo's T4 immutable-module-scope pattern (frozen JSON/result snapshot consumed per test); per-test tmp snapshot tests unchanged.
• Verify: run_tests.py tests/gates/test_fact_extractor.py tests/gates/test_freshness_inference.py --force; module wall from 34.1s CPU class to single-digit.

D7 — permanent errors enter the memgraph retry storm (semantic_graph/adapters/memgraph.py:899)

• Measured (adversarial, worse than first stated): the adapter's own _execute_with_reconnect retries on _is_storage_contention and never consults core/graph/retry.py's _PERMANENT_MARKERS; a 50N42 "unknown database" error burns 75.75s of pure backoff per operation (12 attempts of min(0.25·2^(n−1), 10.0) = 0.25+0.5+1+2+4+8+6×10, arithmetic reproduced), and connect() to a nonexistent database did not fail within a 250s probe (each setup op burns its own 12-retry loop).
• Classifier, corrected (§6b-2): _is_storage_contention (memgraph.py:891) is a disjunction — type(exc).__name__ == "TransientError" OR any _STORAGE_CONTENTION_MARKERS substring hit — not the type-name test alone. The defect surface is therefore wider than first written. Corroboration the first pass missed: retry.py's own _PERMANENT_MARKERS docstring already records that naming a nonexistent database answers Memgraph.TransientError.MemgraphError, which is exactly why that marker list exists on the other seam.
• Fix: permanent-marker classification inside _execute_with_reconnect, evaluated first and short-circuiting ahead of both disjuncts — adding the marker list without reordering leaves a permanent error whose message happens to contain a contention marker still burning the full 75.75s. Share retry.py's list rather than duplicating it.
• Verify: unit-level test pointing at a nonexistent database asserts failure < 2s; a second case asserts a permanent error carrying a contention-marker substring also fails < 2s (the short-circuit, not just the marker list); memgraph adapter module tests green.

3. Suite-side program (levers 1+3, from the decomposition)

All classes adversarially confirmed in size; per-lever savings are NOT projected (the "~340-360s" figure was withdrawn — see §6). Each fix is verified against its own module baseline.

Target definition (corrected 2026-08-12, §6b-1): work the ≥1.0s cut — 581 tests holding 1,678.1s / 72.6% of call CPU — not the "2-60s band", which is 264 tests / 1,261.4s / 54.6%. The two were fused in the research record; the ≥1s cut is both the larger mass and the one the 581 figure actually described.

1. Fixture sharing where semantics allow (lever 1): probe measured fixture shares 11% / 52% / 2% across three modules (test_trace_intel / test_search / test_cli). Start with the high-share modules; use session-scoped clients/fixtures. The 793s bucket is WEAK — module-by-module measurement decides, not the suite estimate.
2. Pipeline-once fixtures (lever 3): test_agent_check_e2e (3 tests, 39.6s) + test_agent_check (24.8s) run real ruff/mypy/pytest per test + sleep(1.5) handshakes; run once module-scope (T4 pattern, e.g. _async_multi_results_json precedent) and replace sleeps with events.
3. Retry-budget DI in tests (lever 3): core/graph/retry.py MAX_ATTEMPTS=8 exponential ceilings to 3.2s; tests exercising retry paths inject a shortened budget (precedent: test_session_graph_store.py 42.7→4.24s on 2026-08-11).
4. AST/index sharing (lever 3): extend the AST-daemon session-fixture pattern (shipped 2026-08-11, 4–6× on CLI tests) to test_gen_brief_pack (41.2s), test_orchestrata_tools matcher (18.5s), test_evaluate_semantic_graph_qa (24.8s).
5. Nested-runner modules (132s CPU total): smaller nested targets + shared fixtures (test_run_tests.py 56.1s, test_test_executor.py 32.7s, test_run_tests_main_infra.py 29.6s).

4. Execution order (implementation phase, post-signoff)

Reversed 2026-08-12 (§8-1). The original order ran the four production defects first. Measured against this plan's own objective that is backwards: tests/semantic_graph/ holds 26.8s (1.16%) of baseline call CPU and tests/refactor/codemod/ 77.4s (3.35%), so D1/D2/D2b/D7 move the 446s by approximately nothing, while §3 holds the mass. Suite-side first; production defects follow on their own merits (per-call latency), not on suite-wall grounds.

1. D3 (test-infra pattern, contained, 32.7s of one module's 34.1s) — it is a suite-side fix that happens to live in the production-defect register.
2. Suite-side §3, order 2→3→1→4→5 by module, each with its own targeted run_tests.py --force verification against that module's baseline CPU.
3. D7 (smallest production item, unblocks sane failure modes).
4. D1 (biggest production latency) — gated on the §2 D1 semantics decision (§6b-3) being settled first.
5. D2 + D2b (same file, same lazy-cache discipline, one rename).
6. Final gate: full scripts/run_tests.py --force; report PASS/FAIL and total wall vs the 445.7s baseline (same measurement method, same -j 8 -b 6). No intermediate 8-minute runs — targeted modules only until the final gate. Report call CPU alongside wall: at 24% run-to-run CPU variance (§6b-5), a wall comparison alone cannot separate a real win from a quiet run.

Work happens in a wt-* worktree per item or small item group (scripts/worktrees.py create), merge --no-ff into master, push from a worktree (git push origin master:master). No AI attribution in commits. Never mutate git state while a suite runs (git_sentinel).

RUL-008 (§8/R4). --no-verify on the in-worktree commits is a hook-time convenience, not a review waiver: scripts/gates.py plus the code/test/architecture reviewer pass run per item, before the --no-ff merge, and every defect they surface gets fixed regardless of whose scope it was (RUL-006/RUL-024). An item whose gates were not run does not merge.

5. Ruled out (read and rejected, not skipped)

• code_parser.resolve_calls / resolve_inheritance — index-based, linear. Not defects.
• ToolExecutor.execute (nesting depth 19) — branch nesting, zero loops. Maintainability issue, not runtime.
• 40 constant sleeps in src/ — production poll/grace intervals, not suite-hot; class closed.
• transitive_coupling O(n³) (Warshall, dict matrix) — latent; callers are amendment-scoped only. Add a size guard + bitset closure when a full-repo caller ever appears.
• Resident-worker executor — measured regression (590s vs 446s), reverted; per-batch subprocess isolation buys correctness and speed.

6. Withdrawn / corrected claims (adversarial review record)

• Withdrawn: "~340-360s realistic suite outcome" — no per-lever savings model exists; endpoints (446s, 289s) are arithmetic, the middle is not.
• Corrected: D1 retrieve cost ~16s/call on the shared cluster (not ~8s); round-trip 1.39ms.
• Corrected: D7 is ~3× worse than first stated (75.75s per operation, multiple operations per connect).
• Marked WEAK: the 793s fixture / 397s session-overhead split (cache-minus-call conflates collection/imports/teardown/idle; sample-of-3 probe cannot generalize).

6b. Second adversarial review — corrections (2026-08-12)

Independent re-derivation from the artifacts (fresh segmentation of test-durations.ndjson, fresh _prepare_batches replay, AST-toolkit reads of every cited site). The measurement core reproduced exactly — 2,312.0s call CPU, 289.0s floor, 51 batches, 3,104.6s cache mass / 792.6s bucket, the 8.4/35.9/53.3/85.3% distribution, the 3/24/48/189-test histogram, the module CPU table, the 32.7-of-34.1s D3 attribution, and the 75.75s backoff sum. Four inferences did not survive.

• 6b-1 — WRONG, corrected in the research record §3. "581 tests in the 2-60s band ≈ 52% of CPU" fuses two different cuts. The 2-60s band is 264 tests / 1,261.4s / 54.6% (and 3+24+48+189 = 264, from that section's own histogram). 581 is the count of tests ≥1.0s (1,678.1s, 72.6%). Mis-sized the addressable mass by 2.2× in the count.
• 6b-2 — WRONG, corrected in D7. The classifier is a disjunction, not the type-name test alone; the fix needs short-circuit ordering, not just a shared marker list.
• 6b-3 — WRONG, corrected in D1. The proposed candidate selection inverts needle and haystack ($chunk_text CONTAINS e.id); no property index can serve it. The batched-neighbor query carries the win; the "tens of ms" target is withdrawn pending a measured server-side scan cost. The heuristic's correctness defect is now an explicit precondition rather than a noted smell.
• 6b-4 — UNSUPPORTED, corrected in §1. The resident-worker regression is real at 587.4s (artifact located on feat/resident-workers), but that artifact records zero reruns and zero solo retries; the "4 solo-retry recoveries" and the "cross-batch state accumulation" cause are withdrawn.
• 6b-5 — WEAK, corrected in §1. The 289s floor is one sample of a 24%-noisy quantity; quoted as a 289–358s range from the three consecutive 2026-08-11 runs. The plan's conclusion is unaffected.
• 6b-6 — WEAK, recorded in the research record §4. The falsify probe names "test_cli" where at least five test_cli.py modules exist and none matches the quoted 21.8s at baseline; the experiment is not reproducible as written.
• 6b-7 — minor. test_agent_check_e2e is 4 records at 39.6s in the baseline window, not 3.

7. Open decision

Scope of the sign-off: production-side only (D1, D2, D2b, D7), suite-side only (§3), or both. The measured floor/ceiling for the suite wall is 289–358s (floor, sample-dependent) to 446s regardless; the production defects carry their own per-call measured costs independent of the suite.

Reviewer recommendation: both. The order reversal is already applied in §4 (§8-1) and each production defect now carries the scale guard it lacked (§8-2), so the two scopes no longer compete: the suite-side program holds 96%+ of the leverage on the objective that triggered this plan and runs first; the production defects follow on their own per-call merits. One item still needs a decision, not an implementation — D1's matching semantics (§6b-3) — and it gates §4 step 4 rather than the sign-off itself.

8. Second-review findings — all folded into the plan body (2026-08-12)

Every item below is now implemented in the sections named at the end of each entry; this section is kept as the evidence record, not as an open list.

• 8-1 — The four production defects have near-zero leverage on the stated objective, yet §4 ran them first. Measured share of baseline call CPU: tests/semantic_graph/ = 26.8s (1.16%), tests/refactor/codemod/ = 77.4s (3.35%), and the two modules that exercise D2/D2b — test_analysis_passes + test_code_intelligence_graph — total 0.1s. The plan stated the defects are independent of the suite goal, then ordered them ahead of every suite lever. → folded into §4 (order reversed, rationale inline).
• 8-2 — No regression guard at real scale for any of the four. centrality costs 340.37s on the real 38,369-node CIG and 0.1s in the tests: the suite is structurally incapable of catching the defect or a regression of the fix. Same shape for D1 (6.6s of test_memgraph_adapter). D1's and D2b's verify criteria were one-off manual measurements. → folded into §2 D1 (seeded scratch-db fixture, round-trip assertion) and §2 D2b (real-CIG benchmark). D7's sub-2s assertion already covered it; D2's instance-scoped cache check added.
• 8-3 — D2's "invalidate on graph mutation" built machinery for a path that does not exist. CodeIntelligenceGraph exposes 22 symbols (_cig.py), none of which mutate _nodes/_csr — it is constructed from a prebuilt CSR index and a reload builds a new instance. Lazy memoization suffices and beats "compute once per graph build", which would pay 0.66–0.72s on every workspace load even when nobody queries impact_score. → folded into §2 D2.
• 8-4 — Name collision the register should own. BlastRadius.impact_score (= |reached| / |nodes|, _cig.py:95) and CodeIntelligenceGraph.impact_score (PageRank, _cig.py:401) are unrelated metrics sharing a name, both served from the same JSON-RPC surface (server.py:277 vs server.py:343). → folded into §2 D2 as a rename shipped with the cache.
• 8-5 — The suite-side program was aimed at the wrong cut. Consequence of §6b-1: the target is the ≥1.0s cut (581 tests, 1,678.1s, 72.6%), not the 2-60s band (264 tests, 54.6%). → folded into §3 as an explicit target definition.
• 8-6 — A wall-only final comparison cannot prove a win. Call CPU varies 24% run-to-run (§6b-5), which is larger than several of the levers. → folded into §4 step 6: report call CPU alongside wall.
• R1–R4 (all applied): R1 order reversal → §4. R2 scale artifacts → §2 D1/D2/D2b. R3 settle D1 semantics before writing the query → §2 D1 precondition + §4 step 4 gating. R4 RUL-008 reconciliation → §4 closing paragraph.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks