Knowledge Wiki Orchestrata Ultimate

RUL-029 — Scope verification by query, not by blanket

Ruling

Before running the whole suite, ask what the change can reach.

scripts/affected_tests.py resolves changed files to the test modules that could cover them, over the serialized import closure plus a manifest of non-import edges. It returns one of two things:

A whole-tree run is therefore never "just to be safe": it is either what the oracle asked for, or a stated exception. Exceptions are legitimate and common — certifying a release, hunting cross-module interference, re-baselining timings — and none of them is derivable from a diff, which is exactly why they have to be said out loud rather than assumed.

This does not weaken any existing rule. The kernel is still required ([[RUL-010]]), runs are still observed ([[RUL-009]]), and the gate battery still precedes every commit ([[RUL-008]]). It governs one thing: how much of the tree a given diff has earned.

Why

The full suite is 1608 modules and ~400s on a 6-core box. Reaching for it reflexively is not caution. It costs the machine, it delays the answer, and when the radius was three modules it tells you nothing the targeted run would not have told you in twenty seconds.

Measured, session 260 (2026-08-10). A two-function retry in scripts/_irc_conversation.py got a full 400s suite thrown at it. The after-the-fact query took two seconds:

who-imports _irc_conversation  ->  scripts/_irc_listen.py, scripts/_irc_send.py
who-imports _irc_send/_irc_listen (tests)  ->  0

Two importers, one covering module — already green at 23/23.

And yet the run was correct. affected_tests.py returns escalate: true for that file: it lives outside src/ and tests/, where the import-closure map cannot see subprocess or hook edges. The oracle's conservatism was right and the reflex was wrong, and they happened to agree. That coincidence is the whole argument for mechanizing this — a habit that lands on the right answer without consulting anything will land on the wrong one as soon as the case changes, and a bounded src/ diff is the far more common case.

The oracle had a hole, and it was found by using it

Asked about scripts/_irc_conversation.py, affected_tests.py named tests/scripts/test__irc_conversation.py — a file that has never existed. The manifest rule scripts/*.py -> tests/scripts/test_{stem}.py is a naming CONVENTION expanded by string substitution, and nothing checked whether the convention had been honoured. The real covering module is test_irc_conversation_capture.py, and no rule reached it.

A path that is not on disk is worse than no path: pytest errors on it, or the caller reads a non-empty tests list and believes the change is covered. Two fixes landed before the guard was built, because a guard is only as good as the oracle under it:

  1. compute_affected now drops test paths that are not on disk, and escalates when it does — a rule that promised coverage and could not deliver it has not bounded anything.
  2. A manifest rule maps the whole IRC producer trio (scripts/_irc_*.py) to test_irc_conversation_capture.py. They are one SPEC-021 producer — the listener holds the session, the sender joins it, the shared helper owns the Redis keys — so the behaviour only means anything end to end, in one module.

Enforcement

Notes on the mechanism

A PreToolUse(Bash) deny hook, wired in .claude/settings.json, tested in tests/scripts/test_proportionate_test_scope.py against a real git worktree of this repo with a real file modified in it, letting the real affected_tests.py answer. git worktree add --detach costs under a second, which is what makes a per-test tree state affordable — and a guard whose oracle is faked proves only that the fake agrees with itself.

It fires only on a whole-tree invocation: a runner with no path argument, or one aimed at tests/ itself. Anything narrower is already scoped. Flag values are not mistaken for targets (-j 14 is not a path) by requiring a path shape — a separator, a .py suffix, or the literal tests directory — rather than tracking each flag's arity.

Three properties carried over from force-trace-intel.sh, all of which cost something to learn:

  1. The deny payload is serialized, never printf'd. The reason embeds a shell command; commands carry quotes; a harness that cannot parse the decision reads it as allow.
  2. The python3 -c payload contains no apostrophe. It is single-quoted for the shell, so one apostrophe ends the program mid-string. The first draft of this guard broke exactly there, writing v['changed'] inside it — which is why the reason is now built in the same block that computes it, leaving nothing to re-quote.
  3. The escape hatch is loud, not silent. FULL_SUITE_REASON="..." prefixed to the command is always allowed, and leaves the reason in the transcript. A guard with no sanctioned way past it gets routed around (force-test-observation.sh once made another repo's suite unrunnable).