Orchestrata Ultimate / Specs / 113-measurement-retention

113-measurement-retention: Measurement Retention

The authoritative definition of orchestrata's self-learning subsystem for test/runtime performance: retain every measurement with joinable identity, and answer questions about it with deterministic readers — never with statistical predictors, and never by asserting statistical facts into the Prolog gate rulebase. This spec turns the research record's five instrument recommendations into numbered requirements and pins the parts already built.

At a glance

Item	Value
Spec directory	`specs/113-measurement-retention/`
Status line in the spec	Draft
Requirement IDs defined	13
With a `@traces` citation in `src/`	0 (0.0%)
With a `@traces` citation in `tests/`	0 (0.0%)
Tracked files	2
RTM page	113-measurement-retention

Reading this spec

This spec is the authoritative definition of orchestrata's self-learning subsystem for test and runtime performance. Its position is to retain every measurement with a joinable identity and to answer questions about the data with deterministic readers, never with statistical predictors, and never by asserting statistical facts into the Prolog gate rulebase. It turns the five instrument recommendations of an earlier research record into numbered, testable requirements.

The reasoning is a measured refutation. The appealing idea of learning from history to predict and schedule better was replayed against this repository's own packer at production settings on 10 held-out runs. The estimator that ships today (last value) averaged a makespan of 291.0 s, mean-of-history 290.1 s, an exponentially weighted average 290.5 s and median-of-history 299.0 s, which is 8.0 s worse. A perfect oracle reached 274.4 s, worth 3.4 s (1.0%) once the CPU floor binds. The noise is common to all runs (machine load, thermal state, scheduler), so averaging history chases a mean that does not exist. What pays is the profile, not the model, and the repository had been discarding the identity that made its profile joinable.

Six functional requirements follow. FR-001 puts run identity (run_id and batch_id) on executor-produced duration rows and appends each header and row in one write so concurrent appends stay intact. FR-002 retains per-phase setup, call and teardown durations without extra parsing passes. FR-003 adds a per-fixture setup cost instrument, gated by an environment variable and off by default. FR-004 wires and tests a freshness guard that makes the test runner abort on a non-zero exit. FR-005 makes external core-seconds reportable next to the suite's own in floor claims. FR-006 makes the Prolog-boundary prohibition a standing rule: the governance learning lifecycle of spec 114 is the only path from observed defects to the rulebase. As of the spec's dated status note, FR-001, FR-002, FR-004 and FR-005 are built and this spec pins their contracts, while FR-003 is split, with the instrument built and the reporting side open.

The spec's status line reads Draft. The requirements traceability matrix lists every ID. It tracks 2 files. No src/ symbol cites any of the 13 requirements, so measured trace coverage is 0.0%, and in tests/, 0 requirements are cited by a test-side @traces. All 13 requirements have no citing symbol, the largest groups being REQ-113-FR1 (4), REQ-113-FR2 (2) and REQ-113-FR3 (2); the behaviour may still exist in code that was never annotated.

Requirements

Checkbox is the state recorded in the requirements checklist; Traced by is the count of src/ symbols whose @traces names the ID.

ID	Done	Requirement	Traced by
`REQ-113-FR1-001`	yes	Executor-produced duration rows carry `run_id` + `batch_id`; bare-pytest rows omit them (omission ≠ null)	none
`REQ-113-FR1-002`	yes	One `{"kind": "run", …}` header line per batch flush; readers join rows by `(run_id, batch_id)`, never position; per-run constants live on the header only	none
`REQ-113-FR1-003`	yes	Header + rows appended as ONE `os.write` on an `O_APPEND` fd; concurrent-append integrity pinned by `tests/core/threading/test_durations_identity.py`	none
`REQ-113-FR1-004`	yes	Identity producers stay wired (`run_tests.py` sets run/commit env; `_batch_env` derives batch id from the junit path; lane from worker slot)	none
`REQ-113-FR2-001`	no	Per-test `setup`/`teardown` durations retained in the sink keyed by `(run_id, batch_id)`, with zero additional parsing passes	none
`REQ-113-FR2-002`	no	One batched run suffices to answer "total setup seconds per module for run R" from retained rows alone	none
`REQ-113-FR3-001`	no	`pytest_fixture_setup` hookwrapper in `pytest_live_observer.py`, env-gated and OFF by default (the `ORCH_RCG_DIR` pattern), recording per-fixture per-batch setup cost keyed by run identity	none
`REQ-113-FR3-002`	no	ONE gated run produces a top-N per-fixture setup-cost report; no multi-run history required	none
`REQ-113-FR4-001`	yes	`run_tests.py` invokes `freshness_guard.py` and aborts on non-zero exit; both the wiring and the guard keep their test modules	none
`REQ-113-FR5-001`	no	Two cgroup CPU readings around each batched run; delta recorded keyed by `run_id`	none
`REQ-113-FR5-002`	no	External core-seconds reportable next to the suite's own core-seconds in floor claims	none
`REQ-113-FR6-001`	no	No statistical/aggregated measurement asserted as a fact or loaded as a rule into any rulebase consulted by `GateEngine`; retained measurements are consumed by deterministic Python readers	none
`REQ-113-FR6-002`	no	The SPEC-114 lifecycle is the only path from observed defects to the rulebase (human-promoted deterministic rules, never statistical facts); changes to the five risk mechanisms require re-audit of this FR	none

Code that traces this spec

No src/ symbol cites a requirement of this spec.

Checklists

`checklists/requirements.md`

The requirement list above comes from this file; 5 of 13 checkbox items are ticked.

Full spec text

Verbatim from specs/113-measurement-retention/spec.md; headings are pushed down two levels.

SPEC-113: Measurement Retention

Metadata

Spec ID: 113-measurement-retention
Title: Measurement Retention — "self-learning" as retained measurement + deterministic inference, never statistical prediction
Author: Mohamed
Status: Draft
Created: 2026-08-19
Status re-verified: 2026-08-20 — FR-002/FR-003/FR-005 had landed between drafting and this check; their UNBUILT lines were corrected in place, with the superseded justifications retained beneath each. Only REQ-113-FR3-002 (the per-fixture report) remains open across FR-001..FR-005.
Priority: P1
Version: 0.2.0
Related: SPEC-114 (governance rule learning — the rule-lifecycle half), SPEC-097 (testing standards), SPEC-023 (goal system / GIL-off doctrine); ADR-0025, ADR-0026, ADR-0027, ADR-0031
Supersedes (jointly with SPEC-114): docs/architecture/learning.md (unbuilt vision doc — see its status header)
Evidence base: plans/prolog-learning-subsystem-research-2026-08-12.md (14 agents / 528 tool calls / 39 min; every number measured on this box). All file:line citations below re-verified 2026-08-19.

1. Overview

What

Why

The appealing half of "a subsystem that learns from test runs" — learn from history, predict, schedule better — was measured dead on this codebase, not argued dead (research record, Verdict, line 13). The refutation replayed this repo's actual LPT packer at production settings against 10 held-out runs (measured 2026-08-12; the per-agent artifacts were NOT retained — the record's evidence pointer resolves to 0 files today, so these numbers are the record's, not re-runnable):

estimator	mean makespan (10 held-out runs)
last-value (what ships today, `_estimate_durations`)	291.0s
mean-of-history	290.1s
EWMA(α=0.4)	290.5s
median-of-history	299.0s (8.0s WORSE)
perfect ORACLE	274.4s (346.9s vs 343.5s = 3.4s once the 6-core CPU floor binds, which it did in 9 of 10 runs)

Learning beats the trivial estimator by 0.5–0.9s against a 28.2s noise band; the "robust" choice loses by 8.0s; a perfect oracle is worth 3.4s (1.0%). Per-module cross-run coefficient of variation is p50 0.329 and the noise is common-mode (machine load, thermal, scheduler), so averaging history chases a mean that does not exist (research record lines 25–49). The minimum detectable effect at n=3 is 57.6s; proving an 8.4s win would take n=141 runs per arm, 19.1h per arm (line 195–199).

What did pay, in the entire ML4Code field and here: the profile, not the model (research record line 98). This repo already collects the profile and, until 2026-08-14, threw away the identity that made it joinable. That identity — and the four other instrument gaps — is this spec.

How

Five functional requirements lift the research record's recommendations 1–5 (line 115–134) into testable contracts; a sixth makes the Prolog-boundary prohibition a standing rule. Status as of 2026-08-20 (re-verified against the live tree; the 2026-08-19 text below predated the builders' work landing): FR-001, FR-002, FR-004 and FR-005 are BUILT and this spec pins them. FR-003 is SPLIT — its instrument (REQ-113-FR3-001) is BUILT; its report (REQ-113-FR3-002) is the one remaining open item in FR-001..FR-005.

2. Definition: what "self-learning" means here

Self-learning = (a) retained ground-truth measurement, keyed by run identity, plus (b) deterministic inference over those facts by ordinary code, plus (c) the SPEC-114 rule lifecycle for governance rules. It explicitly does not mean statistical estimation, prediction, or model fitting over the retained measurements.

Burden-of-proof clause: any future proposal to reintroduce a learned/statistical estimator into scheduling, selection, or gating MUST refute the recorded measurements with new measurements of at least equal power (n=141 per arm for an 8s effect, per the research record's MDE analysis) — argument does not reopen this question, and neither does a new library.

3. Functional Requirements

FR-001: Run identity on duration records (research rec 1) — BUILT; this FR pins the contract

Built 2026-08-14 by the durations-identity work; verified live 2026-08-19 against reports/test-durations.ndjson (410,217,804 bytes / 2,085,361 lines): rows in the current sink carry run_id/batch_id (e.g. run_id "1787155204-912845", batch_id "batch_0"), and 386 of 445 header lines in the tail 20MB carry full identity.

REQ-113-FR1-001: Every per-test duration row produced under the batched executor SHALL carry run_id and batch_id. A row from a bare pytest invocation legitimately OMITS them — omission means "not run under the executor"; writing null would claim the field was measured and found empty (pytest_live_observer.py::run_identity, lines 220–245).
REQ-113-FR1-002: Each batch flush SHALL be preceded by one header line {"kind": "run", "ts", "run_id", "head_commit", "batch_id", "lane"} (absent keys omitted), and readers SHALL join rows to headers by (run_id, batch_id), never by file position (_append_test_durations, lines 248–294; reader scripts/test_durations.py::_read_all, lines 110–157). Per-run constants (head_commit, lane) live on the header only: denormalising them onto every row was measured at 2.7× record growth (121→323 bytes) and rejected (pytest_runtest_logreport, lines 692–745).
REQ-113-FR1-003: The block (header + rows) SHALL be serialized in full and appended with ONE os.write on an O_APPEND fd, so line integrity under 8-way concurrent appenders is kernel semantics, not io-buffering luck. Pinned by tests/core/threading/test_durations_identity.py — every line parses, every row joins to its batch.
REQ-113-FR1-004: The identity producers SHALL remain wired: scripts/run_tests.py sets ORCH_RUN_ID (line 713) and ORCH_HEAD_COMMIT (line 719); test_executor._batch_env (lines 1289–1299) derives ORCH_BATCH_ID from the batch's unique junit path; lane comes from ORCHESTRATA_WORKER_SLOT.

FR-002: Retain per-phase (setup/call/teardown) durations (research rec 2) — BUILT 2026-08-20; this FR now pins it

Status corrected 2026-08-20. The paragraph below describes the pre-build state and is retained for the rationale. Wiring verified against the live tree: phase_durations_ndjson_path() at pytest_live_observer.py:255 (used at :385), _append_test_phase_durations() at :368, called in production at :803, inside pytest_sessionfinish (:790) — i.e. the phase rows are flushed at session finish, not left in the transient live log. Covered by tests/core/threading/test_durations_identity.py and tests/core/threading/test_live_observer.py (10 references between them). Line numbers are as-of 2026-08-20; re-verify, do not trust them.

The observer already measures all three phases per test — pytest_runtest_logreport (lines 692–745) builds {"event": "report", "when": setup|call|teardown, "duration": …} for every phase — but only the terminal-outcome duration reaches the retained sink; setup/teardown go to the transient live log and are lost. On the executor side, _process_batch_result (test_executor.py lines 1229–1286) parses each batch's JUnit XML for counts and module totals and then discards it. That discard is why the fixture bucket was a residual marked WEAK throughout the 446s decomposition (research record line 111–114).

REQ-113-FR2-001: Per-test setup and teardown durations SHALL be retained in the durations sink, keyed by the same (run_id, batch_id) identity as FR-001 rows, introducing ZERO additional parsing passes (the measurements are already in hand at the observer; retention is a write, not a new parse).
REQ-113-FR2-002: Acceptance: after one batched suite run, a scripts/test_durations.py query SHALL answer "total setup seconds per module for run R" from retained rows alone — converting the weakest number in the suite-runtime decomposition into a measured one.

FR-003: Per-fixture setup cost instrument (research rec 3) — SPLIT: instrument BUILT 2026-08-20, report STILL OPEN

Status corrected 2026-08-20. The "zero hits" claim below is no longer true — the instrument landed after this spec was written.

REQ-113-FR3-001 — BUILT. pytest_fixture_setup() at pytest_live_observer.py:633–688, declared @pytest.hookimpl(wrapper=True), gated on ORCH_FIXTURE_COST_DIR (_FIXTURE_COST_DIR_ENV, :630) and a bare pass-through when unset, exactly as this FR requires. Each record carries fixture argname, scope, nodeid, duration and excepted, keyed by run_identity() filtered to run_id/batch_id per FR-001, appended to <dir>/fixture-durations-<pid>.ndjson. Covered by tests/core/threading/test_live_observer.py:811–1029, with the gate-unset case asserted to produce zero bytes.
REQ-113-FR3-002 — NOT BUILT. This is the only remaining open item across FR-001..FR-005. No reader consumes fixture-durations-*.ndjson: the sole glob of that pattern in the tree is a test helper (test_live_observer.py:835). The instrument writes; nothing reports. Until a top-N-by-total-setup-seconds reader exists, this FR's acceptance criterion — "ONE gated run SHALL suffice to produce a per-fixture cost report" — is unmet.

The original (now stale) justification follows.

pytest_fixture_setup has zero hits repo-wide (verified 2026-08-19: grep -rl pytest_fixture_setup src/ scripts/ tests/ → 0 files). This is the one genuinely new signal with demonstrated floor mass, and it needs n=1, not history — it is the instrument that would have found the AST-daemon defect automatically instead of by hand (research record lines 124–128).

REQ-113-FR3-001: A pytest_fixture_setup hookwrapper SHALL be added to pytest_live_observer.py, gated behind an env var and OFF by default, the same pattern as runtime_callgraph.py gating on ORCH_RCG_DIR. When enabled it records per-fixture, per-batch setup cost keyed by FR-001 identity.
REQ-113-FR3-002: Acceptance: ONE gated run SHALL suffice to produce a per-fixture cost report naming the top-N fixtures by total setup seconds. No multi-run history may be required for the report to be meaningful.

FR-004: Freshness guard wired + tested (research rec 4) — SATISFIED; this FR pins it

The research record's own retraction (lines 167–191) established the guard was already wired and the follow-up work gave it tests. Verified 2026-08-19: scripts/run_tests.py invokes scripts/freshness_guard.py as a subprocess (line 1089) and tests/scripts/test_freshness_guard.py exists.

REQ-113-FR4-001: scripts/run_tests.py SHALL invoke the freshness guard before the suite and abort the run on non-zero exit; the invocation and the guard's own behavior SHALL each keep a test module (tests/scripts/test_run_tests_main_infra.py, tests/scripts/test_freshness_guard.py). Removing either wiring or tests is a regression against this requirement.

Status corrected 2026-08-20. The "no reader exists" claim below is no longer true. Verified against the live tree: _snapshot_container_cpu_before() at scripts/run_tests.py:952 and _measure_container_cpu() at :979, both called in production (:1314 and :1325 respectively) so the two readings bracket the suite; standalone CLI at scripts/container_cpu_accounting.py; sink reports/container-cpu.ndjson (git-ignored per-file, matching the durations-sink convention); covered by tests/scripts/test_run_tests_container_cpu.py. Line numbers as-of 2026-08-20 — re-verify, do not trust them.

The original (now stale) justification follows.

Memgraph and Redis run on the same 6 physical cores and are EXCLUDED from the measured 2,567.7 core-s, so the true CPU floor is above 428.0s by an unmeasured amount; every "the suite is at its floor" statement is conditioned on that omission (research record lines 130–134). Verified 2026-08-19: no cpuacct/cpu.stat reader exists in scripts/ or src/ (grep → 0 files).

REQ-113-FR5-001: The batched runner SHALL take two cgroup CPU readings (e.g. cpu.stat usage_usec) for the test-fleet containers, one before and one after the suite, and record the delta keyed by run_id alongside the run's own totals.
REQ-113-FR5-002: Acceptance: after one run, the recorded external core-seconds SHALL be reportable next to the suite's own core-seconds, so any floor claim states both terms.

FR-006: Statistical facts SHALL NOT enter the Prolog gate rulebase — standing prohibition

The gate layer earns its place on proof trees (explain.pl, ADR-0031), the three-valued PASS/FAIL/INVALID verdict (freshness.pl), and rules-as-reviewable-data in a .pl diff — none of which is a speed argument. Adding statistical facts to that rulebase is a specific, not stylistic, risk. All five mechanisms re-verified 2026-08-19:

Flat namespace. The gate's consult set (gates/engine.py::GateEngine._loaded, lines 352–388: code_facts.pl + the facts snapshot + freshness.pl + caller rulebases gate_rules.pl/gate_severity.pl + explain.pl on demand) loads into ONE process-global SWI database, and only explain.pl declares :- module (explain.pl line 53). Repo-wide, 12 of the 26 rulebases under src/orchestrata_ultimate/prolog/ declare no :- module (measured; the research's "6 of 8" counts have drifted, the mechanism has not). A statistical predicate would share namespace with enforcing rules.
Non-monotonic negation-as-failure. 142 non-comment \+ sites across the 26 rulebases in src/orchestrata_ultimate/prolog/ (156 including comment lines; measured 2026-08-19; ADR-0027 recorded 121 on 2026-07-24 — the count GREW), including 9 in gate_rules.pl. A fact base that grows as statistics accumulate silently changes the meaning of every \+ that ranges over it.
Statistical facts would be INVISIBLE to the freshness discipline — outside its safety net, not failed by it. staleness/2 ranges over content_hash(snapshot|disk, File, H) FILE hashes (freshness.pl lines 48–51), which the extractor emits for .py files only, and gate_precondition/1 (lines 114–117) requires callers to return invalid on anything but fresh. A statistical fact derives from thousands of runs, not from a hashable file: freshness could neither verify it nor flag it INVALID — it would sit permanently outside the one mechanism that keeps the gate honest about stale evidence. (Corrected 2026-08-19: the research record phrased this as "a learned fact can never satisfy hash-equality"; the measured mechanism is invisibility, which is the worse exposure. See SPEC-114 REQ-114-FR3-005 for the rule-file half of the same gap.)
Confidence thresholds reopen ADR-0026. "No governance predicate may return a truncated answer set" (docs/adr/0026-no-truncated-answers-in-governance-predicates.md) — filtering derived answers by a confidence cutoff is truncation-by-threshold, the exact shape ADR-0026 retired.
explain.pl renders every unit clause as the same leaf. The meta-interpreter emits _{kind: fact, head: …} for any unit clause (explain.pl line 113) — a hash-verified content_hash/3 fact and a statistical p50_duration/2 fact would be indistinguishable in a proof tree, corrupting ADR-0031's guarantee that a decision's derivation is inspectable evidence.

REQ-113-FR6-001: No statistical or aggregated measurement (durations, percentiles, EWMA values, flake rates, co-occurrence counts, confidence scores) SHALL be asserted as a fact or loaded as a rule into any rulebase consulted by GateEngine. Retained measurements are consumed by deterministic Python readers (scripts/test_durations.py and successors) — and every proposed query over them is a GROUP BY, not a recursion (research record line 60). The 370× Prolog-vs-dicts figure from reports/prolog-codeintel-benchmark.md is HISTORICAL: it was measured at 696 modules / 7,869 symbols, and at today's 2,858 modules / 53,784 symbols the benchmark no longer completes inside the project's own 5-minute cap (two runs, exit 124, 2026-08-19) — cite it as "measured 2026-08-12-era scale", never as a live number.
REQ-113-FR6-002: SPEC-114's learned-rule lifecycle is the ONLY sanctioned path by which observed defects influence the rulebase, and what crosses that boundary is a human-promoted deterministic rule, never a statistical fact. Any change to the five mechanisms above (module-scoping the rulebases, replacing NAF, relaxing freshness) requires re-auditing this FR, not silently voiding it.

4. Non-Goals (explicitly dropped, with the numbers attached so they are not re-proposed)

From the research record (lines 136–142), binding on this spec and its successors:

Learned/EWMA duration prediction — 0.5–0.9s better than last-value; median-of-history 8.0s worse; oracle ceiling 3.4s (measured 2026-08-12; artifacts not retained — see §1's provenance caveat).
Wiring the CLP(FD) scheduler for speed — same 3.4s ceiling; its own header records ~24s solve at 20 tasks; 1,615 modules would not terminate usefully.
A Prolog fact base for the observation stream — historically 370× slower than dicts on this repo's own benchmark; the benchmark itself no longer completes at today's scale (see REQ-113-FR6-001), and the GROUP-BY-not-recursion shape of every proposed query is unchanged.
Risk-ordered test ordering — 0s of wall on a green no-skip suite, against 59% of Python flaky tests being order-dependent (Gruber et al.).
Conflict-pair mining until the sample is adequate. Measured 2026-08-12: 793 candidate pairs, 785 seen once, 0 at n≥3. Recounted 2026-08-19: 2,422 distinct pairs, 13 at n≥3 — the sample is growing but remains far below a powered analysis; the non-goal stands and the numbers must be re-measured, not quoted, before any revisit.
Replacing test_executor._estimate_durations (lines 908–922: last-value with median default) — every measured "improvement" lost or tied within noise.

5. Traceability

Implementation code for FR-001..FR-005 SHALL carry @traces to REQ-113-*. The evidence base is plans/prolog-learning-subsystem-research-2026-08-12.md; the built-state citations in this spec were re-verified 2026-08-19 and any future audit should re-verify against the live tree, not trust this document's line numbers.