Knowledge Wiki Orchestrata Ultimate

Home / Projects / Orchestrata Ultimate / Decisions / ADR index / ADR-0088

ADR-0088: One shared green ledger per repo; a test reruns only when its own code or the code it tests changes

field value
Status Accepted — Accepted (ruling, Mohamed, 2026-10-11) · R1 ruled (a) 2026-10-11
Date 2026-10-11
Register factory (docs/adr, 4-digit ids)
Source docs/adr/0088-one-shared-green-ledger-per-repo-a-test-reruns-only-when-its-code-or-the-code-it-tests-changes.md — recorded on branch feat/shared-ledger at 099eb1773, not yet on master
Wiki register entry ADR-0088 (wiki status: Accepted)

← ADR-0087 · ADR-0089 →

The record

Status: Accepted (ruling, Mohamed, 2026-10-11) · R1 ruled (a) 2026-10-11 · Date: 2026-10-11 · Amends: RUL-032 mechanism 1

Context

RUL-032 (2026-09-22) says a test that passed is not run again until its code or the code it tests changes. Its mechanism, scripts/test_green_ledger.py, falls short of that in four measured ways (2026-10-11, worktree wt-GREENLET-FT at f0d9f23d8):

Gap Measurement
G1 one ledger per working tree reports/test-green-ledger.json lives in each checkout. A fresh worktree reuses nothing: test_green_ledger.py --status over the four greenlet-ship modules printed 0 reused, 4 to run, and the file did not exist. Every ship runs from a fresh worktree (platform convention 3).
G2 only run_tests.py writes it Greens from ad-hoc pytest, run_tests_streaming.py and the per-change red/green runs are never recorded, although the live observer plugin (-p orchestrata_pytest_observer, loaded for every pytest on this box via PYTEST_ADDOPTS) sees each one.
G3 the key misses non-import subjects The key is the test module + its Python import closure + conftest chain + pyproject.toml. tests/deploy/test_ship_merge_master.py (6 key inputs) does not include deploy/ship-merge-master.sh, the script it runs; tests/core/test_uv_lock_contract.py (5 key inputs, empty closure) does not include uv.lock, the file it reads. Today's two changes were exactly those two files: under the current key both tests' greens would have been reused after their subject changed.
G4 module granularity One key and one entry per module: a module with one failing test re-runs its passing siblings.

Decision (Mohamed, 2026-10-11, verbatim)

"yes. one shared per repo with the test file and hash and test id and only rerun when the code of the test or the code the code tests changes" — in answer to: record greens from every observed run (the live observer), and keep one shared ledger per repo instead of one per worktree.

  1. One ledger per repository. Stored under the git common directory ($(git rev-parse --git-common-dir)/orchestrata/test-green-ledger.sqlite3), so the primary checkout and every worktree read and write the same store; never committed; it outlives any worktree. SQLite (stdlib sqlite3, WAL, busy timeout) because writers are concurrent: eight run_tests.py workers and several worktrees at once. The per-tree reports/test-green-ledger.json is retired; its entries are imported once, and only where the recorded key still equals the key recomputed now.
  2. One row per test id and key. Row: test id (pytest node id), test file, content key (sha256), outcome, run id, commit, worktree path, timestamp, duration. A test is reused when a passing row exists for (its id, its current key); otherwise it runs. Rows for several keys of one test coexist, so two worktrees at different code states each keep their own greens.
  3. The key is the code of the test and the code it tests. sha256, in a fixed order, over the bytes of:
    • the test module and its conftest.py chain (the code of the test);
    • every first-party module in its transitive import closure (unchanged from RUL-032);
    • every non-Python file the test reaches by reference — the paths the reference-edge extractor already resolves (.sh .pl .feature .json) — and every file plans/affected-tests-manifest.json maps to the test (closes G3: deploy/ship-merge-master.sh, uv.lock);
    • the inputs every test's code runs on: pyproject.toml, uv.lock, and the interpreter build (sys.version, GIL state). env/*.env stays out of the key, as RUL-032 ruled. The key is computed at collection time, from the bytes the run imported.
  4. Every observed run records. The live observer plugin writes a row when a test's setup, call and teardown all pass; run_tests.py keeps recording. A ledger write failure is logged and never fails a test.
  5. Who reuses. run_tests.py — and through it ship's verify step — partitions per test id before batching, prints the reused count, and folds reused tests into the report with their source run ids; --force runs everything. A plain pytest invocation runs exactly what it was asked to.

Open ruling R1 — does the prod gate reuse?

ADR-0087 (Accepted 2026-10-11) requires "one full correctness run of the promoted commit". Under this ADR, a test whose recorded key equals its key at the promoted commit has been proven on byte-identical code.

Until R1 is ruled, the prod gate keeps today's behaviour (no reuse).

RULED 2026-10-11 — option (a) (Mohamed: "yes R1 (a), go ahead, that is LAW"): the prod gate composes. promote_env.py test binds the promoted commit's sha and covers every test of that commit, executed in this run or reused from a passing row whose key equals the key computed from the promoted tree; the record lists each reused test id with its source run. This satisfies ADR-0087 item 1's "one correctness run of the promoted commit": a reused row proves byte-identical code, and a different tree cannot be admitted because its keys differ. --force stays available to execute everything, and is never required by the gate.

Alternatives rejected

Consequences

Provenance

Addendum 1, 2026-10-14 (append-only; ruling, Mohamed, verbatim: "code state only")

Context: the independent review of the P1 key-soundness change (reviewer report, plans/briefs/qwen/B2b-review-2026-10-14/, 2026-10-14) measured every one of the 1,962 test modules keying 4,572–4,573 of the repository's 5,504 files (HEAD median: 55), because the harness closure (D5/D6) walks scripts/run_tests.py and the conftest chain whose read and path edges name the harness's OWN outputs, directory-prefix expansion (D3/D4) admits tests/, plans/ and reports/ wholesale, and _is_file admits gitignored files. The inputs that change on every run: reports/test-durations.ndjson, reports/run-tests-latest.json, .cache/test_durations.json, .flake_recoveries.ndjson, .cache/gates.json, docs/CODEBASE_MAP.md, plans/ast-ultimate.json, context.json, .env, five .venv/bin/* paths. One unchanged rerun of a module changed its key; six identical runs recorded six keys; --status over 38 modules printed 0 reused, 38 to run. Reuse under item 3 never occurred. The P1 brief forbade removing any key input, so the exclusion needed a ruling.

Ruling: a key input is code state only. Item 3 is read as follows:

  1. A file is a key input only if it is tracked by git at the repository under test (git ls-files) or is one of the declared lock/interpreter inputs (pyproject.toml, uv.lock, sys.version, GIL state). A gitignored or untracked file is never a key input, whatever edge names it.
  2. A file the harness WRITES is an output, not an input: the targets of write-mode open, redirections and Path.write_* edges in scripts/run_tests.py, the observer plugin, the ledger CLI and the conftest chain are excluded from every key, and so is any path under reports/, .cache/ and plans/ast-*.json/docs/CODEBASE_MAP*.md (generated artifacts, RUL-030).
  3. A directory-prefix edge (D3/D4) expands only over tracked files, and never over reports/, .cache/, .venv/ or any gitignored subtree.
  4. Environment files (env/*.env, .env) stay out of the key, as item 3 already ruled. Consequence: the per-test row and the content key again describe the code the test runs on, nothing the run itself produces; a key changes only when a tracked input changes. Acceptance for the next build: the reviewer's measurement reversed — an unchanged rerun keeps every module's key; the median input count returns to the order of HEAD's 55; --status over the 38 V3 modules reports reuse after one green run. The brief's principle "a missed edge means a stale green reused" is unchanged for tracked inputs.

Addendum 2, 2026-10-14 (append-only; ruling, Mohamed, verbatim: "reach by caller")

Context: the P1 reach guard (D7) marks a module repo-state when any git invocation happens inside the module's run window, and the observer then withholds the module's greens. The harness plugin tests/plugins/git_sentinel.py runs git ls-files --others at every module's last teardown, inside that window, to catch tests that leave untracked files. Measured by the review: every real module is marked (424 repo-state rows across 42 modules since 2026-10-12, 8 guard-unavailable), so item 4 ("every observed run records") never held with the guard on; the D7 tests passed only because their fixture repositories carry no sentinel.

Ruling: reach is attributed by caller. A file access or process spawn observed by the guard belongs to the test only when the calling frame is in the test module, its fixtures or conftest chain, or the first-party code the test exercises. A call whose nearest first-party frame is harness code (the plugins under tests/plugins/, the live observer, scripts/run_tests.py, the ledger itself) is the harness's reach: it marks nothing and withholds nothing. The harness is already a key input through its own section (D6), so its reads are accounted for there. The sentinel stays where it is and keeps its per-module attribution. Rejected: moving the sentinel outside the window (loses the attribution it exists for); exempting read-only git commands (a test that reads repository state is exactly what the guard must catch). Acceptance: with the guard on, a run of the V3 set records greens for every passing test and writes no repo-state gap row caused by the sentinel; a test that itself calls git inside its body is still marked.

Addendum 3, 2026-10-14 (append-only; ruling, Mohamed, 22:54, verbatim: "approved addendum 3")

Context: the P2a executor measured (plans/briefs/qwen/B2b-P2a-report.md §1.6–1.8) that addendum 1 applied to the letter still leaves 4,486 key inputs for tests/core/test_reference_edges_anchors.py, 4,353 of them in the D6 harness section, because the harness _walk follows the reference edges of the harness roots (scripts/run_tests.py, the live observer, the ledger CLI and the conftest chain) into 3,388 Python files: the roots carry directory-prefix edges over tests/, scripts/, plans/, src/orchestrata_ultimate/, docs/adr/, specs/ and claude/, all tracked, so addendum 1's "tracked only" does not shrink them, and only its generated-artifact rule excludes the four tracked volatile files (reports/run-tests-latest.json, docs/CODEBASE_MAP.md, plans/ast-ultimate.json, context.json). The pure import closure of the same roots is 17 files; with the conftest closure (about 129) and the module's own inputs the key is of the order of 150 inputs.

Ruling: harness roots contribute their code bytes only; their path and prefix edges are not key inputs.

  1. The D6 harness section is keyed by import closure only: the bytes of the harness roots and of the first-party modules they import, transitively. No reference edge (path, prefix, read, load) from a harness root or from a module of that closure contributes a key input; the data a run reads is not the code the test runs on.
  2. KEY_SCHEMA_VERSION bumps to 3, so every green recorded under the earlier key rules is missed once and re-proven at the new key.
  3. An edge-named file (a file a test reaches by reference edge or by manifest, decision item 3) is admitted as a key input only when it is tracked by git and is not a generated artifact (addendum 1 items 1–2: reports/, .cache/, .venv/, plans/ast-*.json, docs/CODEBASE_MAP*.md, the write-mode targets of the harness). The test's own reach (its module, fixtures, conftest chain and the first-party code it exercises) keeps every edge kind as before; only the harness's edges are withdrawn. Acceptance (increment P2a-2): an unchanged rerun keeps every module's key; the input count of each V3 module is below 200; --status over the V3 set reports reuse after one green run; --gaps shows 0 sentinel-caused repo-state rows; the six red-first tests in tests/core/threading/test_green_ledger.py are green. Provenance: Mohamed, 2026-10-14 22:54, "approved addendum 3", on the proposal recorded in plans/reports/executor-experiment-2026-10-14.md §9.2 and the P2a report §3. Measured 2026-10-14 23:45 (increment P2a-2, .orchestrata/REPORT-P2a-2.md): inputs 4,490 → 147 (reach guard), 150 (live observer), 213 (ledger reuse), 147 (anchors); an unchanged rerun kept all four keys; 0 sentinel repo-state rows. The 213 is the genuine import closure of tests/scripts/conftest.py (scripts.run_tests, semantic_graph.embedder_factory), kept by this addendum's "the test's own reach keeps every edge kind"; the "below 200" figure above was the supervisor's sizing estimate, not part of the ruling. --status still reported no reuse for a reason outside this addendum (census selection string ≠ RUN_TESTS_MARKEXPR, fixed in P2b).

Rulings cited

Specs cited

None (or no spec page exists for the ones named).