Orchestrata Ultimate › Plans › suite-runtime-2-plan

Suite runtime, attempt 2 — plan

Source: plans/suite-runtime-2-plan.md at 8a2cb42ad · last changed 2026-08-12 · first committed 2026-08-12 · 182 lines.

Prose pending.

Plan text

Suite runtime, attempt 2 — plan

Date: 2026-08-12 Baseline: 6ceef2ef + uncommitted D2b fix — 426.6s wall, 32,499 passed / 0 failed / 0 errors / 0 skipped, 1611/1611 modules, -j 8 -b 6. Call CPU 2,202.4s, 8-lane floor 275.3s. Success: lower total suite wall, proven at suite scale, with test count non-decreasing and 0 failures. Nothing else counts.

This plan is bound by plans/suite-runtime-failure-postmortem-2026-08-12.md §7. Attempt 1 failed because it optimized module-local costs that did not transfer to a saturated 8-lane run, and measured the result with a design that could not detect a win of the available size. Every lever below carries its share of the lane budget, and the measurement design comes first.

1. Measured structure of the 426.6s (this tree, not last week's)

• Mean lane occupancy 5.18 / 8; idle lane-seconds 1,198s (35.4%).
• Tail: last 60s runs at 3.53 / 8 — roughly 268 idle lane-s, ~33s of wall.
• Call-CPU concentration: tests ≥1.0s = 532 tests holding 1,579.5s (71.7%). Top 10 = 8.8%, top 100 = 37.0%.
• Machine: 12 logical CPUs (6 cores × HT), 31 GB RAM, Python 3.14.6 free-threaded, GIL off.
• Infra is NOT the lane cap: worker slot index is prefix-agnostic and unbounded, orchkg_w0..orchkg_w24 are provisioned, Redis uses slot+1 against a 16-DB default.

The headline structural fact: the suite runs 8 lanes on a 12-core machine and leaves 35% of even that reduced budget idle. Lane count is the only lever that touches 100% of the wall.

1b. Lane-budget decomposition — MEASURED, not inferred (2026-08-12)

Each component measured directly rather than left as a residual. This replaces attempt 1's WEAK 793s/397s split, which was cache-minus-call arithmetic.

Correction, 2026-08-12 (second review). This table previously assigned the whole ~837s residual to "fixture setup/teardown", which contradicted §3c's independently-measured 516.7 lane-s for the same bucket, in the same document. The residual is not a measurement: it is everything the four measured terms do not explain, and only 516.7s of it is measured fixture cost. ~321 lane-s remain unattributed and no number in this plan may be derived from them. Two further defects in the same arithmetic, both open:

• The 516.7 lane-s → ~279 core-s conversion in §3c is underived. Lane-seconds of executing work are core-seconds; the factor is unexplained and is not to be reused until it is.
• The AST-daemon claim (336 core-s for ONE fixture) exceeds the 279 core-s figure for the ENTIRE fixture bucket. At least one of the two is wrong. Nothing may be spent against the 336 until a paired n=3 core-second measurement settles it — which is exactly the "estimate before measurement" failure the post-mortem §1 records.

What settles all three: test_executor._process_batch_result already parses every batch's JUnit XML — the only source that separates setup/call/teardown — and then deletes it. Retaining those rows, keyed by run/batch/lane, converts the weakest number in this decomposition into a measured one at zero extra parsing cost. See plans/prolog-learning-subsystem-research-2026-08-12.md.

Two hypotheses died here. Per-batch fixed cost is 1.22s, not the ~23s the two-configuration arithmetic suggested — so batch-count reduction is not a lever, and neither is import cost: collecting the entire tree costs 11.74s. The non-call time is fixture setup/teardown, nothing else.

1c. Is the slow work CPU or waiting? — MEASURED

The 40 slowest tests (638.5s of call time) run in one process: WALL 162.75s, USER 127.26s, SYS 10.09s → 84% CPU. They are CPU-bound. Consequences:

• Removing sleep() calls is not the lever. (A census found 309 constant-sleep sites summing 395s per single execution, but the run reports zero tests over 60s, so the large ones — 100s, 60s, 30s — are loop bounds and timeout ceilings, not executed sleeps. The census is non-evidence.)
• On a CPU-saturated box, wall ≈ total-CPU ÷ cores. Reducing wall means removing CPU work; moving it between fixtures and calls changes nothing. This is the mechanism behind attempt 1's "contention laundering" post-mortem finding, now with a number attached.

1d. Lane count — MEASURED AND REJECTED

17s of wall is inside the noise band (identical trees have measured 426.6s and 445.7s), while the CPU inflation is unambiguous: load reached 16.05 on 12 cores and every test ran slower. L1 is rejected — not on one run's wall, but on the mechanism the CPU number exposes.

1e. The suite is already at its physical-core floor — MEASURED

Full run under /usr/bin/time (includes every waited-for child process, which matters because many tests shell out):

wall 489.15s | user 2368.77s | sys 198.93s | cpu 524% | maxrss 3.4GB
32,499 passed / 0 failed / 0 errors / 0 skipped · 1611/1611 modules · batch phase 455s

• Total CPU = 2,567.7 core-seconds.
• The box is an i7-8850H: 6 physical cores, 12 threads. Hyperthreads add throughput but are not cores.
• CPU floor = 2,567.7 ÷ 6 = 428s. Measured walls: 426.6s, 455s, and 410s at 12 lanes.

The suite is sitting ON its floor. Average utilisation is 524% ≈ 5.2 of 6 physical cores, which is also why -j 12 inflated per-test wall by 35%: at 8 lanes the physical cores are already busy, and the extra lanes contend for hyperthread siblings.

Consequence, and it governs everything below: the ONLY change that lowers suite wall on this machine is one that lowers total core-seconds. Scheduling, batching, lane count, fixture scope, and work relocation cannot — they redistribute the same 2,568 core-seconds. This is the mechanism behind attempt 1's "contention laundering", stated as arithmetic instead of a metaphor.

3b. Primary metric upgrade — core-seconds, not wall

Wall is the goal but a poor instrument: identical trees measured 426.6s and 455s (6.6% apart). Total core-seconds (user + sys from /usr/bin/time around the whole run) is the primary acceptance metric — it is near-deterministic, it is what actually causes wall on a core-bound suite, and a change that does not reduce it cannot reduce wall except by luck.

• Report BOTH per run: core-seconds and wall.
• A lever ships only if it lowers core-seconds by a margin larger than the core-second spread measured across the baseline's own n=3 runs.
• Wall remains the headline number in the final report, since it is what the user asked for.

2. Lever inventory — each with its floor mass

Any lever below the ~5% noise floor needs the measurement design in §3 written down before work starts (post-mortem rule 3).

Ruled out before starting, with reasons: resident workers (measured +32%, reverted, see the post-mortem); pytest-xdist (banned, deadlocks); reducing test count, scope, or assertions in any form.

3. Measurement design — fixed before any change

Noise is the reason attempt 1 could not tell a win from a quiet machine: identical trees measured 445.7s and 426.6s (4.5% apart) and call CPU has ranged 24% across days.

• n = 3 runs per configuration, same machine, no other load, scripts/run_tests.py --force.
• Compare medians, and report every individual run — no cherry-picking, no single-run claims.
• A configuration wins only if its median wall is lower than the baseline median by more than the observed spread of the baseline's own three runs.
• Every run reports: wall, call CPU re-derived from the durations sink, passed/failed/errors/skipped, module count, and solo-retry count.
• Quality gates on every run, non-negotiable: test count non-decreasing vs 32,499, 0 failed, 0 errors, 0 skipped, and solo-retries not above baseline. A configuration that trades failures or flakes for wall time is a FAILURE, not a win.

3c. Measured ceiling — stated before the work, not after

Ceiling ≈ 10–15% — 427s to roughly 370–385s. Not a halving. The suite's CPU is spread across 32,499 tests doing real work against real infrastructure; there is no single big win, and any claim of one should be treated as an error. This number is stated before the work starts, per the post-mortem's rule 4 (no estimates before decomposition; premise shrink is disclosed).

4. Execution

L1 is a configuration probe and needs no swarm; it is measured first because it is the largest lever and it can invalidate the rest. L2/L3/L5 are code work and are swarmed only with a measured floor mass in hand. L4 is entered only if L1-L3 fall short, and only for work that is genuinely eliminated rather than moved.

Definition of finished (user, 2026-08-12): merged, promoted, deployed, worktree deleted. A lever that does not survive §3 does not merge.

5. Results

Filled in as runs complete. Verdict first, table second.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks