Knowledge Wiki Orchestrata Ultimate

Orchestrata Ultimate › Plans › briefs › 2026-09-22-rul-032-run-once-ledger

Brief: enforce RUL-032 — a test module runs once per code state

Source: plans/briefs/2026-09-22-rul-032-run-once-ledger.md at 8a2cb42ad · last changed 2026-09-22 · first committed 2026-09-22 · 138 lines.

Prose pending.

Plan text

Brief: enforce RUL-032 — a test module runs once per code state

Ruling (Mohamed, 2026-09-22, verbatim): "with the rule enforced that test only run once unless the code in the test or code is changed, and watch for escalation to full suite when the tests have already been run, this should never happen." Recorded as knowledge-wiki rulings/RUL-032-tests-run-once-per-code-state.md.

GOAL

scripts/run_tests.py never schedules a test module whose content key matches its last green run, and scripts/affected_tests.py never escalates to the full suite because an env/*.env file changed.

FACTS (measured 2026-09-22)

FILES

DESIGN (settled)

  1. scripts/test_green_ledger.py
    • module_key(test_path, *, repo_root, closure) -> str: sha256 over, in this order, the bytes of: the test module; each src/ file in its transitive import closure (sorted by path); each conftest.py from repo root to the module's directory; pyproject.toml. Env files are NOT inputs (ruling). Missing closure files are skipped, never fatal.
    • closure_for(test_path) built from the selector's maps (forward closure of the test's orchestrata_ultimate.* imports through the src dependency graph, bounded by the selector's MAX_DEPTH).
    • Ledger at reports/test-green-ledger.json: {module: {"key": str, "passed": int, "run_id": str, "head_commit": str, "ts": float}}, written atomically (tmp + os.replace).
    • load_ledger(), record_green(modules_with_counts, run_id, head), partition(modules) -> (to_run, reused) where reused maps module → recorded entry.
    • CLI: --status <paths…> prints per-module run/reuse; --seed-from reports/run-tests-latest.json records the report's modules as green ONLY when the report's head_commit is an ancestor of HEAD and git diff <head_commit>..HEAD --name-only touches none of that module's key inputs; modules that fail that check are listed and skipped.
    • The module docstring names RUL-032 (the rulings lint checks the rule-id appears in each declared gate script).
  2. scripts/run_tests.py
    • In _main, after affected resolution and the report-reuse check and before _run_suite: unless --force, partition(modules); print [run_tests] ♻ RUL-032: N module(s) unchanged since their last green run — reused, not re-run; M to run. If M == 0, do not start the executor: build the result from the reused entries alone (passed = sum of recorded counts, failed 0) and write the canonical report as usual.
    • The report gains "reused": {"modules": N, "tests": n}; test_totals.passed includes the reused tests so ship_already_verified and _try_reuse_report keep working unchanged.
    • After a green batch/solo round, record_green for every module that passed, with its per-module passed count and this run's ORCH_RUN_ID / HEAD.
    • Escalation path: when --affected escalates, the full module list is still passed through partition, so an escalation over proven modules runs only the changed ones. Print the reused count on that path too.
    • The docstring names RUL-032.
  3. scripts/affected_tests.py
    • _is_env_endpoint_config files no longer set escalate. Instead: diff the env file against base_rev (_diff_changed_lines exists), extract the changed KEY= names, and resolve each key to every test module and src/ module whose text contains the key name (walk tests/**/*.py and src/**/*.py once; src hits resolve to their dependent tests via the existing reverse-dep + test-import maps). Reason text: env/test.env: keys A, B changed — resolved to <n> tests that name them (RUL-032: no escalation). A key nothing names resolves to the smoke floor only and is reported.
  4. Tests (real files, tmp git repos where the existing tests already do that; no mocks):
    • ledger: key changes when the test file, a closure src file, a conftest or pyproject changes; key is unchanged when an env file changes; partition reuses only on exact key match; seed refuses a module whose inputs changed after the recorded head.
    • run_tests: a second invocation on the same tree schedules 0 modules and reports reused; --force schedules all.
    • affected_tests: an env file edit that changes TEST_MEMGRAPH_HA_MAIN_URI yields escalate=False and selects the modules naming that key.

CONSTRAINTS

DONE WHEN

VERIFY (report verbatim)