Orchestrata Ultimate › Plans › prolog-learning-subsystem-research-2026-08-12

A Prolog subsystem that learns from test runs — research record

Source: plans/prolog-learning-subsystem-research-2026-08-12.md at 8a2cb42ad · last changed 2026-08-12 · first committed 2026-08-12 · 202 lines.

Prose pending.

Plan text

A Prolog subsystem that learns from test runs — research record

Date: 2026-08-12 Question (user): build a Prolog subsystem that learns from test runs, stores the facts it learns, updates them as code changes, and produces optimized ways to run tests and code and to maximize code performance — does that make sense? Also: the idea of ML4Code. Method: 14 agents / 528 tool calls / 39 min — 4 ground-truth inventories of this repo, 4 prior-art surveys, 3 independent designs, 3 independent refutations. Every number below is either measured on this box or carries its source.

Verdict

Yes, and the repo is already two-thirds of the way there — but the valuable two-thirds is MEASUREMENT RETENTION, not learning. The idea is sound and the substrate is unusually ready. The specific half that gives the idea its appeal — learn from history, predict, schedule better — is dead on this codebase, and it is dead for reasons that were measured, not argued.

Three independent agents reached that split without coordination. The refutation that mattered did not argue it; it ran the experiment.

The measurement that settles the learning half

An adversarial agent recovered 12 clean single-run segments from the 334 MB durations sink (628,604 records, 50.9h window), replayed this repo's actual LPT packer at production settings (48 bins, 8 lanes, 1.22s spawn), and scored five estimators against each held-out next run's true durations:

Learning beats the trivial estimator by 0.5–0.9s. The most obvious "robust" choice loses by 8.0s. And a PERFECT oracle is worth 3.4s (1.0%) once the CPU floor binds — which it does in 9 of 10 simulated runs — against a 28.2s noise band.

The root cause is measurable: per-module coefficient of variation across runs is p50 0.329, and single-run call-CPU ranged 2,202.4s → 3,440.5s (56%) across 15 segments. That noise is common-mode — machine load, thermal, scheduler — so averaging over history does not reduce error on the next run. It chases a mean that does not exist.

test_executor._estimate_durations is literally history.get(m, median(history)) — an EWMA with α=1.0. The "learning" being proposed is a last-value dictionary lookup that already ships, and every real learner was measured worse than it.

Why the rest of the learning half is dead here specifically

And the invalidation half — the part of the idea that is hardest and most interesting — is defeated by churn arithmetic: tests/conftest.py, in the static import closure of all 1,611 test modules, has 28 commits in 30 days against a gate that runs about daily. Under the only sound invalidation tier, each of those edits voids every observation in the store. 350 of 917 src modules changed in 7 days (38.2%), which is exactly freshness_guard's retention window. The store would be invalidated at roughly the rate it is populated, so an observation-count-gated stable_cost/2 is structurally unreachable and the steady state is the fallback — today's _estimate_durations.

What ML4Code actually is, and the one lesson that transfers

ML4Code spans the naturalness-of-software line (Hindle et al.), learned code representations (code2vec, GraphCodeBERT), and learned performance systems. Its honest production record:

What shipped and paid: AutoFDO — 5–15% in Google datacenters, 85% of instrumented-FDO gains — is the largest "learn from observed execution" system in existence and it contains no model. Just sampled counts keyed to source locations. MLGO replaces one heuristic inside one LLVM pass; five years on, both learned policies are opt-in cmake flags and the hand-written heuristic is still the default. Bao emits hints to PostgreSQL's optimizer and keeps the deterministic engine. AlphaDev's sorting networks were merged into libc++ as ordinary assembly — the model does not ship, so invalidation does not exist.

What failed: Neo (end-to-end learned optimizer) — 24h to reach parity, catastrophic plans, no fallback, superseded by hint-steering. Learned indexes — any insert invalidates the model. code2vec as a general representation — did not transfer, programme died. Deep vulnerability detection — >50% loss benchmark→real, models keying on variable names. ML-based RTS as a safety mechanism — inherently unsafe, and the deterministic method reduces suites more (Ekstazi 84% vs ML 68%).

The four production answers to "facts get updated as code changes": freeze the output as deterministic code (AlphaDev); tolerate drift via stable coordinates (AutoFDO keys to function-relative line numbers; BOLT's stale-profile matching recovers 80% of max benefit from profiles several revisions old); continuously retrain behind a veto (Bao); or measure the aging instead of assuming it (MLGO re-tested at 3 months, found slight degradation, and set periodic rather than per-commit retraining on that basis).

The transferable lesson: the highest-yield learned artifact in the entire field is a profile, not a model — and this repo already collects one (2.23M duration records, 7,113 runtime-call facts from sys.monitoring) and then throws away the identity that would make it joinable.

What is genuinely missing (and it is small)

reports/test-durations.ndjson records are exactly {duration, test, ts} — verified live. No run_id, no batch_id, no lane, no commit, no outcome. That absence is why the 446s decomposition had to reconstruct run boundaries by segmenting on timestamp gaps in a throwaway script — and why the refuting agent had to write that script again today. The research keeps being done and deleted.

Second gap: test_executor._process_batch_result already parses every batch's JUnit XML — the only source that separates setup/call/teardown — and then deletes it. That is precisely why the fixture bucket is a residual and marked WEAK throughout the record.

Recommendation

Build the instrument, not the subsystem. In priority order:

1. Stamp run_id / batch_id / lane / head_commit / outcome onto the durations record (~10 LOC at the existing writer). The precondition for every future performance question.
2. Retain the JUnit setup/call/teardown rows the executor already parses and discards, keyed by the same identity. Zero extra parsing; converts the weakest number in the decomposition into a measured one.
3. Per-fixture, per-batch setup cost — a pytest_fixture_setup hookwrapper in pytest_live_observer.py, gated behind an env var the way runtime_callgraph.py gates on ORCH_RCG_DIR. This is the one genuinely new signal with demonstrated floor mass, and it is the instrument that would have found the AST-daemon defect automatically instead of by hand. It needs n=1, not history.
4. Wire scripts/freshness_guard.py and give it a test — see defects below.
5. Measure the Incus container CPU (two cgroup cpuacct reads around one run). Memgraph and Redis run on these same 6 physical cores and are excluded from the 2,567.7 core-s, so the true floor is above 428s by an unmeasured amount. Every "the suite is at its floor" statement, including this document's, is conditioned on that omission. Cheapest item here and it changes what every other conclusion means.

Explicitly dropped, with numbers attached so they are not re-proposed: learned/EWMA duration prediction (0.5–0.9s better than last-value, median-of-history 8.0s worse, oracle ceiling 3.4s); wiring the CLP(FD) scheduler for speed (same 3.4s ceiling; its own header records ~24s solve at 20 tasks with no symmetry-breaking; 1,615 modules would not terminate usefully); a Prolog fact base for the observation stream (370× slower than dicts on this repo's own benchmark); risk-ordered test ordering (0s of wall, against 59% order-dependent flakes); conflict-pair mining until the sample is far past n=24.

Keep Prolog exactly where it is. The gate layer earns its place on proof trees (explain.pl), the three-valued PASS/FAIL/INVALID verdict, and rules-as-reviewable-data in a .pl diff — none of which is a speed argument. Adding statistical facts to that same rule base is a specific, not stylistic, risk: six of the eight rulebases declare no :- module and share a process-global flat namespace; negation-as-failure is non-monotonic and gate rules use it (121 \+ repo-wide per ADR-0027); freshness.pl demands hash-equality or INVALID, which a learned fact can never satisfy; a confidence threshold reopens ADR-0026; and explain.pl renders a hash-verified fact and a statistical inference as the same leaf.

Live defects this research exposed

Found while verifying claims; all confirmed by direct measurement, all being fixed:

Retraction — the "unwired guard / leaking sink" finding was wrong (2026-08-12, same day)

This document originally claimed freshness_guard.py had zero callers and that reports/test-durations.ndjson had therefore grown unchecked to 334 MB. Both are false, and an adversarial verifier caught them. Corrected by direct measurement:

• The guard IS wired. scripts/run_tests.py::_main invokes it as a SUBPROCESS (subprocess.run([sys.executable, guard_script], timeout=900)) and aborts the run on non-zero exit. Call-site tests already existed in tests/scripts/test_run_tests_main_infra.py.
• The instrument was wrong. who-imports measures import sites. A subprocess invocation is not an import, so "0 import sites" was true and "0 callers" was a non-sequitur drawn from it. Using a structural query to answer a behavioural question is exactly the error class this record criticises elsewhere.
• The sink is the correct size. Measured over all 2,135,718 records: span 7.01 days, oldest 7.02d ago, only 1,001 records past the 7.0d retention window, 0 malformed and 0 shape-bad. 320 MB is simply what 7 days × ~10 suite runs × 32,499 tests weighs. It had also already shrunk 334 → 320 MB between the two measurements, which is compaction working.

What survives: the guard genuinely had no test module (now 33 tests), and check_durations_sink genuinely crashed with AttributeError on a valid-JSON non-object line — reproduced against the pre-change module. That bug was real. The claim that it caused the sink size was fabricated: the live file contains zero non-object lines, so the crash path was never reachable on it.

The lesson, and it is the same one as D2b's: a defect that is real does not license a causal story about it that was never measured.

Standing caution

The subsystem-attributable ceiling is ~8s (1.9%) against a 28.2s noise band and a 57.6s minimum detectable effect at n=3. Proving an 8.4s win would take n=141 runs per arm — 19.1 hours per arm. Building a learning subsystem to chase 1–2% inside a 6.6% noise band is the exact sequence plans/suite-runtime-failure-postmortem-2026-08-12.md was written to prevent. The daemon fix alone is 13.1%, requires no facts, and is a fixture-scope edit.

Per-agent detail: /tmp/claude-1000/…/scratchpad/research/*.json (14 files), journal at subagents/workflows/wf_e7f26870-208/journal.jsonl.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks