Knowledge Wiki Orchestrata Ultimate

A checker must prove it looked, not just that it found nothing

The class

A checker reports zero violations. There are two possible reasons: the tree is actually clean, or the checker structurally cannot see the violation shape that's present. From the checker's own output — a green PASS line, an empty list — the two are indistinguishable. Nothing about "0 violations" says which one happened.

This is a sibling defect to a gate must separate the artifact from a description of it: that page is about false positives — a scanner firing on prose about the thing it governs. This one is about false negatives that look like clean passes — a scanner silent about the thing itself, because it never modeled that shape as something to look for. Both defeat the same purpose (a checker that can't be trusted), by opposite mechanisms.

The tell, in every measured instance below, is the same: the checker's design had a scope narrower than its claim, and nothing forced the two to be checked against each other. A generator claims "I regenerated the map." A purity audit claims "no forbidden imports." A parallel-safety gate claims "no unscoped Redis keys." Each one was true of what the tool actually inspected, and false of what its name promised.

Three measured instances

# Checker What "clean" / "success" actually meant Fix
1 inject-codebase-map.sh (session-start codebase-map hook) Concurrent daemon regeneration made every symbols call return empty; the hook wrote the empty result over a working 1.5 MB map and reported success. 2>/dev/null and || true hid the failure from both the hook's own exit code and the log. Build into a temp file; promote it over the live map only if it carries ≥1000 signatures — a stale map is recoverable, a confidently-empty one reads as "that symbol does not exist."
2 audit_test_purity.py (RUL-002 no-mocks/no-fakes gate) EXCLUDED_PATTERNS = ["fakeredis", "fake_redis"] suppressed any violation on a line containing the substring "fakeredis" — which is also the literal text of the actual banned import (import fakeredis.aioredis, tests/kernel/test_event_bus_extended.py). The whitelist and the violation matched the same string; the checker excused the exact thing it was built to catch. Delete the substring carve-out; read the shared .claude/banned-libraries.txt registry directly via core.banned_libraries.is_banned, matched against parsed ast.Import/ast.ImportFrom nodes — an import check, not a text-exclusion list.
3 gate_parallel_safety.py Gate 0.4 (Redis key namespacing) The AST taint model trusted any function call that merely received a test_id-derived argument as itself deriving its result from it (_is_testid_derived returned True for any(...) over a call's arguments, with no check on what the call returns) — a helper that receives test_id and returns a hardcoded key passed clean. A companion line-based fallback additionally exempted any Redis write inside a function whose def test_... line merely contained the token test_id (a fixture parameter), regardless of whether that write's actual key used it. Receiver shapes (aclient, _conn, redis_conn, …) and write verbs (psetex, setrange, lset, zincrby, flushdb, …) were also incomplete — flushdb had a measured 0% detection rate before this fix, 5 live call sites. Narrowed taint model: only str.format(...) is trusted to embed a tainted argument in its result: if isinstance(func, ast.Attribute) and func.attr == "format" — everything else returns False. Added a _Scope class with alias tracking. Expanded receiver and verb lists from measured false negatives, not guesses.

Instance 2 is the sharpest statement of the class: the whitelist's condition and the violation's signature were the same string. A checker cannot be excused from catching X by a rule that matches on the text "X".

A fourth instance was reported alongside these three — a before/after measurement tool that scanned zero files because it resolved its own root from __file__ while running against a temporary copy of the tree, rather than the real checkout. It is recorded here only as reported: a search of this session's history, the repo's plans//context.history.md, and the scripts most likely to carry it (verify_package.py, freshness_guard.py, check_native_build_freshness.py, the AST daemon) found no corroborating citation. Per this page's own thesis, an unverified claim does not become a fourth confirmed instance by being asserted — if you find the actual incident, replace this paragraph with it.

Gate 0.4, measured in full

This is the best-instrumented case, because the rebuild left an artifact trail that supports direct re-measurement rather than a recollection.

Current state. python scripts/gates.py --only parallel-safety --all → rc=0, ALL GATES PASS, Gate 0.4 (Redis key namespacing): PASS, 25.6s wall (session measurement; re-verifiable any time from the repo root).

The baseline the rebuild found. A snapshot taken with the rebuilt detector against the pre-fix tree recorded 130 violation sites across 51 files (/tmp/baseline_g04.json, 2026-08-12). Every one of those 130 sites was disposed of one of two ways: the key was rewritten to derive from test_id (verifiably, by the same detector), or the site earned a reasoned # gate-ok: <reason> waiver — the ONE waiver convention six checkers share (core/gate_waivers.py: WAIVER_RE = re.compile(r"#\s*gate-ok:\s*\S"), WAIVER_REACH = 1 — a marker waives its own line and the line directly below, never more, and a bare # gate-ok: with no reason waives nothing).

The old-vs-new gap, reproduced on one frozen tree. To separate "the mechanism got stricter" from "the tests got worse," both gate versions were run against the same pre-fix tests/ tree (a git show HEAD: checkout at the moment before the shard fixes landed, --tests-dir override):

sites found files with findings
old gate (pre-rebuild) 101 32
new gate (rebuilt) 138 52

Per-file delta over that same frozen tree: 0 files where the new gate found fewer than the old one; 23 files where it found more, for a net +37 sites the old gate structurally could not see on an unchanged tree. The mechanism, not the test suite, is what moved.

What to do when writing a checker

  1. State the checker's actual scope before writing its detection logic — which call shapes, which receivers, which verbs. A checker's docstring should let a reader answer "what would this miss?" without reading the regex.
  2. A whitelist and a detection pattern must never share their matching text. If the string that exempts a line can also be the string that would have violated it, the whitelist is unsound by construction — test this directly, not by inspection.
  3. Trust nothing "arbitrary" without proof it forwards the property you care about. A call that merely receives a tainted value does not return one — Gate 0.4's fix narrowed trust to the one call shape (str.format) actually proven to embed its arguments.
  4. A generator that can produce an empty/degenerate artifact under contention must refuse to promote it over a working one. Write to a temp location; promote only past a sanity floor (a signature count, a byte-size floor, a non-zero site count) — never trust "it ran without raising" as proof the output is real.
  5. Re-measure old vs. new on the identical input when rebuilding a checker. A bigger finding count on a changed tree proves nothing about the mechanism; the same tree, both versions, is the only controlled comparison — see the Gate 0.4 table above for the shape this takes.