Orchestrata Ultimate › Plans › suite-runtime-failure-postmortem-2026-08-12

Post-mortem: suite runtime optimization — PROVEN FAILURE (2026-08-12)

Source: plans/suite-runtime-failure-postmortem-2026-08-12.md at 8a2cb42ad · last changed 2026-08-12 · first committed 2026-08-12 · 151 lines.

Prose pending.

Plan text

Post-mortem: suite runtime optimization — PROVEN FAILURE (2026-08-12)

Verdict: the objective "make the suite meaningfully faster" (originally: halve it, ~240s) failed. Final authoritative measurement on the merged tree: 455s wall vs 445.7s baseline (+2%), 2,250s call CPU vs 2,312s (−2.7%) — both inside the ~24% run-to-run noise band. The suite-side changes were rolled back on master by user directive (revert commits c7043047, 3097834c, f44c64a0, b55077e3, 8bafe038; record commit ae8a491e). Kept: production defect fixes (429fd08c, 5d3ed30c, db0609f9) and all documentation.

What this document is: the honest account of WHY it failed, so the next attempt — if there is one — starts from evidence instead of repeating the sequence.

1. The objective was unreachable as first stated — and the work started anyway

"Halve the suite" was agreed before any decomposition existed. The decomposition (done later) showed test-call CPU = 2,312s → a 289s packing floor at 8 lanes. Halving (~223s) was therefore arithmetically impossible by overhead work from the very first minute. Every hour spent on overhead levers before that arithmetic existed was spent blind. Two estimates were asserted before measurement and both were wrong:

• "Resident workers save 100–140s" — the 30-second calculation that kills it (1.3s/batch × 51 batches ÷ 8 lanes ≈ 8s) was done only after the claim. Built, measured 590s vs 446s, reverted.
• "~340–360s realistic outcome" — never had a per-lever savings model; withdrawn under adversarial review as interpolation between correct endpoints.

2. The levers chosen were too small for the structure of the problem

Measured structure: 65% of wall is test-call CPU; 72.6% of that CPU lives in the 581 tests ≥1.0s (1,678s). The suite-side program optimized fixture setup/teardown (a bucket whose split was never even closed — WEAK throughout), pipeline repetition in two modules, AST-query sharing in three, and fixture sharing in one. Even the agents' own (standalone, optimistic) numbers summed to ~100s of CPU ≈ 12s of wall — inside noise by an order of magnitude. The mass that could move the wall — the ≥1s test cut — was identified in the research and then not worked on.

3. Standalone module benchmarks do not transfer to a saturated suite

The swarm's module-level savings were measured on an otherwise idle machine, and the user correctly called the suite-level result "longer, not faster." Mechanisms:

• Contention laundering: a module that takes 34s standalone → 11s standalone does not yield 23s of lane time inside an 8-lane suite where every lane is busy; part of the standalone cost was never suite cost to begin with, and part of the "saving" is absorbed by neighbors.
• Concurrency is not elimination: one item's win was running two pipelines in parallel threads — free when lanes idle, pure contention at saturation.
• Work relocation is not work reduction: shared-module fixtures moved cost from per-test calls into one-time fixture setup. Real where it eliminated repetition (fact extraction 16.95s→0.78s call CPU), cosmetic where it only moved the timing boundary.

4. The measurement plan could not have proven a win of the available size

Run-to-run call-CPU variance is ~24% (±550s against a 2,312s mean; floor across three consecutive runs ranged 289–358s). A ~50–70s real improvement is 3% — undetectable in a single full run. The plan never included a measurement design sized to the noise (repeated runs, paired comparison, or per-module in-suite tracking); "wall vs baseline, one run each side" was the implicit plan, and it cannot answer the question.

5. Process failures that came first

1. Estimate before measurement — twice (§1). The hook-enforced measure-first checklist + adversarial-reviewer agent now exist because of this.
2. Stale approval reused — the user approved a 100–140s lever; measurement shrank it to ~29s; work continued on the old approval instead of re-confirming. Checklist rule now: premise shrink ⇒ new sign-off.
3. Framing as hiding — module "wins" led every summary while the suite-level result was a wash. User ruling: results are summarized at the end, verdict up front; numbers buried mid-paragraph count as hidden. Recorded here as a standing rule.

6. What was not the failure

The production defect work (D1 retrieve batching, D2 pagerank cache, D2b centrality, D7 permanent-error guard) was measured at its own scale and shipped green: retrieve 16.19s→~2s (10,964→≤3 round-trips), pagerank ~0ms cached, permanent errors fail <1s instead of 75.75s. That work had per-call acceptance criteria from the start — which is why it fared better than the suite program.

Correction, 2026-08-12 (post-rollback review). The line originally read "centrality 340.37s→0.66s". That number was real but it was not the same function answering: 0.66s was a pivot-sampled estimate that was shipped as the default and never checked against the exact result, because D2b's stated acceptance criterion (≥90% top-20 agreement vs exact on the real CIG) was never run. It was run now, on the live 38,380-node repo CIG:

The shipped default disagreed with the exact top-20 on 17 of 20 nodes and reported a top score 7.2× high. Sampling never reached 90% agreement at any tested budget, and random pivots do not rescue it past k=1024 — the estimator needs a source count where it is no longer meaningfully cheaper than being right. On a 600-node chain, whose highest-betweenness node is the midpoint by construction, the sampled default names node 293 instead of 299/300: wrong on a case with a closed-form answer.

Why no test caught it: all four centrality tests ran on graphs smaller than the 256-pivot sample count, and the implementation's own if exact or n <= samples guard sent every one of them down the exact branch. The sampled path shipped with zero coverage.

What the rewrite did deliver, and it is real: replacing the per-source dict-of-dicts accumulator with array-based _brandes took the EXACT computation from 340.37s to 88.29s (3.9×). That is the whole of the honest win, and it is enough — the original defect was a 5.6-minute hang per call.

Fixed (this review): exact now defaults to True and the exact vector is cached per instance (first call 88.3s, ~0ms after — same discipline as the D2 pagerank cache, and sound for the same reason: the graph has no mutators). Sampling stays available behind exact=False with the measured error table in its docstring. Three guards added in tests/refactor/codemod/test_analysis_passes.py, all on a graph LARGER than the sample count so the branch is actually reachable: the default is exact, sampled diverges from exact, and the exact vector is cached per instance.

The lesson this adds to §7: an acceptance criterion that is written but not run is not a criterion. D2b had the right one — "≥90% top-20 overlap vs exact on the real CIG" — sitting in the plan while a 15%-accurate default shipped past it.

7. Rules going forward (binding)

1. Suite wall claims require suite-scale evidence. Standalone module benchmarks are non-evidence for suite wall time.
2. Acceptance criterion + measurement plan before work starts: the wall-time target AND the design that can prove it at the measured noise level (n repeated runs or paired sampling), or the work does not start.
3. Only levers with floor mass. Any suite-time proposal must state its share of the 2,312s call CPU it can eliminate; anything below the noise floor (~5%) needs the measurement design of rule 2 written down, explicitly.
4. No estimates before decomposition; new sign-off when the premise shrinks (measure-first checklist, hook-enforced).
5. Verdict up front, results table at the end. Numbers mid-paragraph = hidden.

8. Rollback record

Reverted (suite-side, no proven gain): WI-4 fact-share, WI-5 agentcheck-once, WI-6 retry-budget DI, WI-7 ast-share, WI-8 search-fixtures — merges 157690ff, 166326c3, 39b55789, 51722f2f, 5e948f92; reverts c7043047…8bafe038. Kept (verified at scale): merges 429fd08c (D7+D1), 5d3ed30c (D2+D2b), purity fix db0609f9, housekeeping 543217ee, this record ae8a491e. Post-rollback affected gate: 898/898 green, 47/47 modules (non-authoritative subset; the pre-rollback full authoritative run was 32,498/0).

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks