Orchestrata Ultimate › Plans › test-run-ledger-design

Test-run observability — event stream, live view, durable ledger

Source: plans/test-run-ledger-design.md at 8a2cb42ad · last changed 2026-08-20 · first committed 2026-08-20 · 1224 lines.

Prose pending.

Plan text

Test-run observability — event stream, live view, durable ledger

STATUS: requirement captured, design forks open. Written in plan mode from injected context only — every "exists" claim below is from docs/CODEBASE_MAP.md one-line entries and memory, NOT from opening the files. First action on exit is to verify the inventory.

Context — why

Asked "walk me through the last test run", the honest answer was almost entirely "not recorded". Concretely, from the artifacts injected this session:

• reports/run-tests-latest.json (~7.5h old): passed=17, failed=0, modules=0. A full run is 1454 modules / ~29.1K tests — so this is a fragment, yet two hooks (error_report.py, test_run_context.py) label it "last authoritative run". That is the project's own false-green class, live.
• No wall time recorded for it. The only full-suite timing available is a historical band in CLAUDE.md (458–486s test phase, measured 2026-07-27).
• No scope field — CORRECTED 2026-08-20. The artifact does carry a scope field ("full" or a list of test paths), read by run_tests.py::_try_reuse_report:361. Scope is recorded but never surfaced: error_report.py and test_run_context.py simply don't display it, so a 17-test fragment and a 29.1K full run look identical to every consumer while the distinguishing data sits in the file. P9 therefore surfaces an existing field rather than adding recording — materially cheaper than planned. Reuse is also NOT the fragment's origin: _try_reuse_report is conservative, requiring zero failures, ≥1 pass, matching head_commit and dirty_sig, and scope coverage. The better lead is the mid-run ABORTED state (below).
• Observed live 2026-08-20: during a healthy run, run-tests-latest.json reads ABORTED (infrastructure) — no result was produced, having read 17 passed minutes earlier. It is mutable single-slot state that passes through misleading intermediate values, not a durable record of the last completed run — a very plausible origin for the passed=17, modules=0 fragment.
• This session's real runs (461 passed 18/18 modules in 29s @ ee16d5fd; 438 passed 17/17 @ 8eca8b7a) are recorded only as prose in context.json.
• Nothing anywhere carries skips, warnings, or errors.
• Single-slot artifact → overwritten each run → no run-over-run comparison, so "did this change make the suite slower" is unanswerable.

Field evidence — authoritative full-suite run, 2026-08-20

Kernel started + gRPC-verified, all safety gates passed, so this run is authoritative. It supplied measured support for several requirements that were previously only argued.

Baseline: 34,414 passed · 1 failed · 0 errors · 0 skipped · 1665/1666 modules · 1196s (19.9m), exit 1.

The one failure, and why it matters here. test_run_tests_main_infra.py::test_main_with_real_infra_check_and_skip_gates — assert rc == 0 got 1; failed in batch and on solo retry. Re-run on an idle box: passes in 12.80s against its 30s budget (~2.4× headroom, which suite load exceeded). So: environment_dependent_failure, load-dependent — not flaky.

Cause not isolated: a nested-runner timeout and an _check_infra() abort under load both return 1 (test_main_infra_check_failure_returns_1 confirms the latter is an expected path), and the discriminating output was deleted with the temp dir. Fix applied: the test now reads capsys before asserting on rc, so a future failure is self-diagnosing. Verified 6 passed.

Diagnosing one failure required the verbatim error, the load context at the time, and a comparison run — three things no artifact currently retains.

Requirement (Mohamed, session 266)

1. Test runs emit EVENTS — the output is a stream, not inherently a file. Named by Mohamed: test started (identifying which test), test stopped, test pass, test failed, test error. Redis is the obvious carrier — it is already the project's queue/broker.
2. A durable record per run, appended, never overwritten — a fold over the event stream, carrying:
   • run id, start ts, end ts, wall duration
   • scope: full suite | --affected subset | named modules | single module, and the actual selection (module list / selector args / commit)
   • collected (Y), passed, failed, skipped, errored, xfail/xpass
   • list of errors; list of warnings
   • commit sha, env profile, worker count, whether the kernel was started+verified
3. Autonomous run — once scope is selected the run proceeds unattended.
4. Live view while running: processes/threads in flight, how many, which tests are executing now, running tallies. Must answer mid-run: "full suite, started 11:04, 6m in, 8,412 of 29,100 tests, 8,390 pass / 2 fail / 20 skip, 8 workers busy."
5. Per-test durations joined to run id → slowest/fastest, and deltas across runs so performance regressions from code changes are visible.

Event model (proposed)

The five events Mohamed named, plus the minimum extras the stated queries require. test_started/test_stopped bracket timing; outcome is a separate fact.

Per-run phase timing is required, and total wall time is not enough (observed live 2026-08-20). The evidence is a failed inference, which makes the case stronger than a successful one would have.

During this session's authoritative run, load moved 5.19 → 2.44 → 5.75 on 12 cores. At the 2.44 sample I concluded the parallel phase had ended and a serial solo-retry tail had taken over (_run_suite(..., solo_retry_cap=5)). The next sample refuted it: load recovered, so the dip was almost certainly a batch boundary — workers cycling between batches, of which there are ~48 (8 workers × 6 batches).

Run phase could not be reliably inferred from the only live signal available, and a real attempt to infer it produced a wrong answer within three minutes. That is the argument for recording phase boundaries as events rather than deriving them.

A 480s parallel phase + 40s tail and a 480s parallel phase + 450s tail have the same total wall time and completely different diagnoses. Today the only way to tell them apart is watching loadavg in real time, which survives nothing. So the run record carries phase boundaries, not just a total.

Note this is distinct from SPEC-113 FR-002, which retains per-test phases (setup/call/teardown). This is per-run phases, and nothing records it.

Two projections over one stream:

• Live view — fold from the head, or read a maintained current summary.
• Ledger record — fold at run_finished, or on discovering a dead run.

A killed run still yields a well-formed partial record, because the stream is the truth and the fold is replayable.

Payload bounding — verbatim WITHOUT unbounded events (resolves R2). A module-level import error fails every test in the module with the same traceback; 1,000 failures × ~10 KB longrepr is ~10 MB in one run, which would trim the stream mid-run and break late-joiner accuracy. So the event carries a 2 KB head plus a sha256 reference; the full text is written once to a content-addressed sidecar under reports/test-run-artifacts/<sha256> — the same pattern codemod's content-addressed parse cache already uses. Identical tracebacks across a thousand tests therefore store once, which is the common case. Verbatim is preserved addressably; MAXLEN then bounds event count, not payload size, and byte-identity (V6) is asserted against the sidecar rather than the stream.

Unmanaged runs — scope is never simply absent (resolves R7). The plugin fires on every invocation, but only the runner supplies scope on run_started. The Bash guard is a PreToolUse hook and governs my tool calls only — not volks, CI, or a shell. So a run can reach the ledger without a runner. Such a run infers what it can from pytest's own config.args and is otherwise recorded explicitly as scope: unknown, authoritative: false, trigger: unmanaged. No consumer may present an unmanaged run as authoritative — this is the 17-test fragment's exact escape route, closed by naming it rather than leaving the field empty.

Emitter architecture — one drain thread, not one client per test thread (resolves R14). Tests run threaded under free-threaded 3.14t, in a codebase with a documented heap-corruption history under load. Test threads therefore push events to a bounded in-process queue; a single drain thread owns the Redis client and does every XADD. On queue overflow the emitter drops and records the drop count in the run record — degradation is always reported, never silent. *_NUM_THREADS=1 stays pinned.

Existing assets to reuse (VERIFY FIRST — map-level claims only)

Decisions (settled by Mohamed, session 266)

1. Event transport — Redis Streams. XADD/XREAD, replayable by a late joiner, MAXLEN trim, many simultaneous readers. Rejected: pub/sub (no replay — a mid-run attach sees nothing) and list queues (consumed once — one observer starves another). Reuse gateway/ring_buffer.py if it already wraps Streams.
2. Emission point — the pytest plugin emits; the runner supplies scope. pytest_live_observer emits every event, so every invocation is recorded — bare pytest, all four runners, subagent runs. Runners pass scope/selection into run_started. One emitter; no runner can forget.
3. Durable ledger — NDJSON first, async KG sync second. reports/test-runs.ndjson is the append-only durable log (matching the existing test-durations.ndjson convention); a separate idempotent job syncs into orchkg for querying. A graph outage must never fail a test run.
4. Consumers wired in this cut — all four: claude/hooks/test_run_context.py (prompt banner), scripts/error_report.py (session red-list), the CLIs (observe_pytest.py live attach + test_durations.py comparison), and the gateway/UI (gateway/test_runner.py, handlers/test_metrics_handler.py).

Requirement block 2 (Mohamed, session 266) — discipline, enforcement, determinism

6. Exact error messages, precise language. No paraphrase, no softening. Exact text as emitted.
7. "Flake" is a defined engineering term and may not be used unless the definition is met. Same discipline for every other loose word.
8. My output is reviewed by a Prolog ruleset — the vocabulary/claims gate.
9. I am reminded of the rule list on every test run.
10. No random bash.
11. After "run tests", pytest may ONLY be invoked through the test runner, which is a versioned submodule of codemod.
12. Runner parameters include profile yes/no and trace yes/no.
13. "Tell me about the last test run" is answered deterministically from the ledger + Prolog: the time, the scope, the last full run, the last scoped run and its scope, the wall time, and whether a gate ran it.
14. Session start makes me aware of these rules at skill + context level, and they are simultaneously enforced deterministically.
15. I may fix and add features to the runner module — versioned, under codemod.
16. Context injection + natural-language query, scoped to the project — resolved from cwd, or named explicitly when several are in play. Events, queries and data are all project-aware.
17. An explicit policy for what lives in the KG vs. what lives in logs.
18. Prolog answers these questions deterministically, reaching the data via Python atoms where needed, and the answers are injected into my context and into agent context, with agents scoped to running the tests.
19. Runs are threaded, under all existing test-run rules.

The lexicon — words that must be earned

Single source: src/orchestrata_ultimate/prolog/test_lexicon.pl, generated the way prolog/intent_lexicon.gen.pl already is. A term may be used only when its predicate succeeds against the ledger.

flaky(Project, NodeId) holds only when outcomes differ across ≥2 executions identical in all six of: nodeid · source commit · env profile · scope selection · worker count · scheduling mode — with no write to any tracked file in the test's impact set (code_intel.py impact) and no change to any generated or untracked artifact in that set between them, over ≥N repetitions. Anything short of that is one of:

Why six and not four (resolves R12): two runs at the same commit with different concurrency are not identical executions, so the original four-part tuple would have classified a genuine concurrency-dependent failure as flaky — misclassification by the very definition whose purpose is precision. And "same commit" already implies no tracked-file writes, so the write clause only earns its place once it is split to cover generated and untracked artifacts, which a commit hash does not constrain.

Also gated: "hung" (use no_event_for(N seconds); distinguish stalled from dead), "slow" (requires a measured threshold and the number), "transient" (must name the condition), "intermittent", "spurious", "mostly green", "should pass". Extends the existing weasel-word ban (problematic/issue/suboptimal → failed/bug/wrong).

Error text is quoted verbatim from the event payload. Summarising an assertion message is a gate violation, not a style choice.

Enforcement — aware AND enforced, from one artifact

The banner I read and the gate that blocks me read the same rulebase, so they cannot drift (the loophole scripts/rulings_lint.py exists to close; drift gated the way check_generated_artifacts.py does).

One ruleset, two languages — resolved by generation, not by assertion (R8). The rules live as Python checks (verification/review/checks_*.py, scripts/code-review.py) and as Prolog (test_discipline.pl, test_lexicon.pl). Two implementations of one ruleset is exactly the drift this section claims to prevent, so the claim is made true rather than asserted: the Prolog is generated from the Python checks' declared metadata, the way gen_intent_lexicon.py already generates lexicon facts, with regeneration gated by check_generated_artifacts.py (RUL-030). A differential test covers the entire rule set, not just flaky/2 — every rule runs through both implementations over the shared corpus and must agree.

The claims gate is three-valued (R3). Absence claims ("no run recorded a wall time") are checked by negation-as-failure, and negation-as-failure over an incomplete KB proves absence that is not true — certifying a false claim as verified, which is the false-confidence class this plan exists to kill, reintroduced at the layer meant to stop it. So an absence claim is certified only when run_ingested(RunId) holds for the relevant runs. Otherwise the verdict is unknown, and the gate permits the claim only if it is stated as unknown. Unknown must never collapse into false.

Runner — refactor/codemod/testrun/, versioned

Lives under the SPEC-110 codemod engine and rides its existing daemon (codemod/server.py JSON-RPC over Unix socket), warm workspace, CodeIntelligence Graph (instant affected-set selection) and 5-observer framework. Exposed as a codemod verb via scripts/codemod.py. Carries __version__; I may extend it.

Parameters — the two profiling senses are separate flags, so the ambiguity is removed at the CLI rather than by convention:

Threaded, never xdist (banned in .claude/banned-libraries.txt; -n auto deadlocks ~78%) — via core/threading/test_executor.py ParallelTestExecutor. Worker events therefore carry thread ids as well as pids.

Deterministic answer contract

scripts/test_intel.py, built like code_intel.py / trace_intel.py / history_intel.py, over ground facts (Python atoms via janus; variable-bearing rules stay in .pl, only ground facts asserted):

test_run(RunId, Project, StartTs, EndTs, WallSeconds, ExitCode).
run_scope(RunId, ScopeKind, Selection).
run_totals(RunId, Collected, Passed, Failed, Skipped, Errored).
run_trigger(RunId, Trigger).        % interactive | gate(Name) | hook | ci | agent(Id)
run_commit(RunId, Sha).  run_profile(RunId, EnvProfile).
run_kernel_verified(RunId, Bool).
last_full_run(Project, RunId).  last_scoped_run(Project, RunId, ScopeKind).
test_outcome(RunId, NodeId, Outcome, DurationMs).
flaky(Project, NodeId, Evidence).   % only when the definition above is met

Every answer is provable via prolog/explain.pl — "why do you call that flaky" returns a derivation, not an opinion.

Data placement policy (req. 17)

Redis = what to watch · logs = what to grep · KG = what to join · Prolog = what to prove.

The event stream must not share fate with the system under test (resolves R4). Run events go to a dedicated Redis DB index outside the suite's per-slot allocation (the mapping asserted by tests/core/test_worker_database_mapping.py). This repo has already had the incident — a slot-0 flushdb wiped shared db0 mid-flight — and a run that can erase its own evidence loses exactly the data needed to diagnose what happened. The isolation is asserted, not assumed.

Project awareness (req. 16)

Every event, record and node id is project-qualified — extending the existing multi-project KG invariant (identity = project-qualified id, never a bare relative path). Project resolved by knowledge_graph/project_resolver.py from cwd; --project overrides; an unresolvable project raises — no default, per the no-infra-defaults rule.

Decisions block 2 (settled)

5. Parameters — split flags: --cprofile, --env-profile, --trace.
6. Claims gate blocks, on test-claim messages only. A Stop hook parses my message; if it asserts a test/run claim (numbers, outcomes, lexicon terms) it must satisfy the rulebase or the stop is denied and I revise. Prose with no test claim passes untouched.
7. Bash: extend the existing deny guards, not deny-by-default. Add a rule denying pytest invoked outside the runner. Accepted trade-off: unknown-unknown commands stay permitted until a rule names them.
8. Execution: the daemon orchestrates, the run is spawned. codemod does selection (warm CodeIntelligenceGraph), scheduling, IPC and event emission; the run executes in a spawned process with its own thread pool. A hang, segfault or env mutation cannot reach the daemon — material given the 3.14t heap-corruption history.

Deliverable 1 — the spec (authoritative contract)

This plan is the design record. The authoritative artifact is a numbered spec, because in this project specs outrank tests when resolving a design question and every gate downstream keys off requirement ids.

Number: next free. SPEC-112 is the highest I can see referenced (kimi/k3-spec-112); confirm the free number in Phase 0 rather than assuming.

Structure — Purpose · Scope and non-goals · Definitions (normative — the lexicon is part of the contract, not commentary) · FR-001..FR-0NN · NFRs · Data-placement policy · Interfaces (event schema, Prolog predicate surface, CLI surface) · Acceptance criteria · Test ids · Traceability.

Id grammar — canonical forms already enforced in this repo: REQ-NNN-MMM, stories REQ-NNN-US-NNN-MMM (canonical since the legacy-story-id migration), AC-NNN-*, TST-NNN-*, FR-NNN.

"Easy to verify" is mechanised, not asserted: every FR gets exactly one AC carrying a machine-checkable predicate with a number in it, and at least one TST-NNN-* id; the implementing symbol carries @traces(reqs=[...]). The spec is then gradeable by tooling that already exists — check_req_format.py, check_spec_authority.py (RUL-013), check_requirement_corpus.py, parse_spec_requirements.py, check-traces, trace_intel.py untraced-reqs, generate_rtm_evidence.py, verification/spec_completeness.py. An FR with no gradeable AC is a spec defect and fails the gate — which is exactly the "easy to validate" property, made enforceable.

Delivery workflow — executed BY orchestrata, via the volk pipeline

This plan is built by orchestrata itself, not by Claude subagents. It is the dogfood: the volk pipeline runs autonomously, I watch, and every defect the run exposes in orchestrata's own workflow is a finding to fix — that is a deliverable of this work, not a distraction from it.

All of it on one feature branch, merged and deleted at the end.

Escalation is the seam where I intervene. orchestrator/pipeline_escalation.py plus session pipeline respond|approve|reject — the pipeline escalates a decision rather than guessing, and I answer without stopping the run.

Observation, never blind waiting: orchestrata watch (kernel/cli_watch.py, the streaming proof consumer), kernel events, gateway pubsub → the VizInt UI, and scripts/e2e_run.py (the dogfood runner with real-time observability). Test runs inside the pipeline emit into the very event stream this spec builds — the system observes its own construction from S1 onward.

Model routing is a profile, not an open question — see the next section. The profile is created in S0 and overrides any volk's built-in model choice (the Architect's current deepseek-reasoner binding included).

The LLM profile (S0 deliverable)

Orchestrata already carries provider/model profiles — pure YAML, no hardcoded defaults (llm/config.py, tests/llm/test_config.py), routed by llm/client.py through providers/provider_manager.py + registry.py to a provider adapter. So the tiering is expressed as a profile, not as an argument.

Binding rule — the whole profile in one line: EVERY Claude model — Fable, Opus, Sonnet AND Haiku, without exception — runs on THIS Claude Code as the provider. Only a non-Claude model would use OpenRouter.

claude-* → provider claude_cli (this Claude Code account, no API key). Everything else → provider openrouter.

Adapters exist for both: providers/adapters/claude_cli_adapter.py and providers/adapters/openrouter_adapter.py.

Both routing seams are covered, and the API-key paths are forbidden. A volk can reach a model through providers/provider_manager.py or through drivers/driver_factory.py, and each side has a Claude API-key path that would take the call off this account while the profile still reads correctly:

Sonnet and Haiku are Claude models and therefore bound to claude_cli exactly as Fable and Opus are. There is no tier for which OpenRouter is the Claude route.

The build factory — Opus designs, Fable reviews, generation is deterministic, Haiku fills, Sonnet supervises:

Stage 3 is why the bar on stage 1 is so high: the diagram is not documentation, it is the source the structure is generated from. It must be complete and traced enough to generate from — which is exactly what an Opus author plus a Fable reviewer are for.

Stage 5 is the control on stage 4: Haiku is right for filling bounded bodies and wrong for judgment, so it never runs unobserved. Sonnet watches, catches, and unblocks — and escalates upward rather than guessing.

The fill → review → fix loop terminates (resolves R5). Nothing in an autonomous loop may retry without a bound, or an overnight run spins on one method body, burning quota with no progress and no alarm. So, per body: 3 attempts by Junior, then senior_dev writes it directly, then escalate to U:Cd, then to me. The question channel is the parallel escape hatch — a volk that cannot satisfy the reviewer asks rather than retrying blind. Attempt counts and escalations are recorded per packet in the ledger, so a body that consumed the full ladder is visible as a number rather than as lost time.

Volk → tier mapping:

Consequence: for this work the profile is Claude-only end to end. OpenRouter is declared as the non-Claude route and stays unexercised unless a non-Claude model is introduced. Opus review of Fable's design remains my job in-conversation, on top of U:Cd's in-pipeline review.

Second consequence — Haiku is a volk brain, and never an unsupervised one. J:Junior runs on Haiku to fill generated method bodies, which is bounded work constrained by a signature, a contract and a test. It is paired with senior_dev on Sonnet, which observes it, reviews its output, takes its escalations and unsticks it. Haiku is never given judgment work and never runs alone; that pairing is the control, not an optimisation.

Mechanically gated invariant — one predicate, trivially checkable, so this cannot silently drift:

for every volk in the profile: model.startswith("claude-") ⟺ provider == "claude_cli", and not model.startswith("claude-") ⟺ provider == "openrouter"

A Claude model routed through OpenRouter or the Anthropic API-key path fails the gate. A volk with no mapping raises — no fallback, matching the no-hardcoded-defaults rule already asserted by tests/llm/test_config.py.

Hazards to encode now, not rediscover:

• OpenRouter key shadowing — a rotated key in .env does nothing while the shell still exports the old one; use env -u OPENROUTER_API_KEY. The keys in keys.txt are 402 account-exhausted. /api/v1/key reporting limit: null is not proof of credit — probe with a real request.
• Profile selection var — tests select claude_cli via SPIKE_PROFILE; confirm the runtime selector name in S0 rather than assuming it is the same one.
• Cassette layer — llm/_cassette.py record/replay: live-record once, then replay in tests, so profile tests do not burn account quota per run.

Code-review rules are injected into every coding volk

Coding volks — U:Cd CodeDirector, J:Junior, senior_dev, U:QA — receive the code-review rules in their context at dispatch. The set is declared explicitly in config, not inferred from a volk's name.

This matters most for J:Junior on Haiku: it fills method bodies and is the tier least able to infer an unstated rule. A rule it was never shown is a rule it will break, and Sonnet supervision then spends its budget catching what injection would have prevented.

One source of truth — injected rules ≡ enforced rules. The same artifacts that gate the code are what get shown to the volk, so the two cannot drift: verification/review/checks_{correctness,quality,safety,structure}.py, scripts/code-review.py, scripts/test-review.py (T-rules), .claude/banned-libraries.txt via core/banned_libraries.py (the single reader), core/standards.py, and the CLAUDE.md Key Rules — no mocking, no pip, absolute imports only, no bare os.environ[...], no monkeypatch, function-scope fixtures, no xdist, declare every dependency, no AI attribution. Same discipline as the test rulebase: the reminder and the gate read one artifact.

Mechanism — reuse, don't invent: knowledge_graph/context_injector.py inject_for_task is already SPEC-106 selector/packer-backed and already task-type-aware, dispatching by SUBJECT_TASK_TYPE through pipeline_kg. Code-review rules become a layer selected for coding task types, assembled by context/context_assembler.py and bounded by context/context_budget.py. scripts/gen_brief_pack.py carries the same pack for swarm fan-out, and claude/hooks/task_context_hook.py does it for any Claude-side agent.

Budgeted and measured, not unbounded — per-layer, per-volk token spend is already reported by scripts/context_usage_report.py. The rules layer is measured like every other layer; if it crowds out task context, that is a number to see, not a surprise.

Every edit is reviewed by the tool, at the point of the edit

No file changed by a volk reaches the branch unreviewed. Each edit — or the file list a volk changed — is run through the review tool immediately, routed by path:

Findings block the volk's turn — it fixes and re-runs before proceeding. There is an existing seam to build on: agents/volks/_edit_review.py, plus the PostToolUse precedent (.claude/hooks/autoformat.py) for firing on write.

Reviewing at the edit rather than at the review stage is the cheap ordering: Haiku corrects its own method body while it still holds the context, instead of Sonnet discovering it three files later and spending supervision budget on a fix that injection-plus-immediate-review would have prevented. The same tool still runs at U:Cd self-review and U:QA — per-edit is the tight loop, not a replacement.

Rule class this must catch — silent swallows. The named example is a non-empty pass for exceptions: except SomeError: pass discards a failure with no handling and no signal. This is the same defect class the repo already gates in one specific place — scripts/check_graph_write_swallow.py ("a graph write failure must reach the caller"); the review rule is the general form.

Verify in S0 whether code-review.py already carries this rule. If it does not, adding it is in scope — it is a stated requirement, not an enhancement.

The rule gets the same treatment as every other: a deny corpus at least as large as the allow corpus, because a deliberate best-effort swallow is occasionally legitimate. Those carry the existing waiver convention — # gate-ok: <reason>, read by core/gate_waivers.py, the single reader — so an intentional exception is declared and reviewable, and an undeclared one fails.

Every volk gets a prepared context — and may ask, and gets a real answer

The brief pack — per volk, per run

No volk is dispatched without a pack containing everything it needs to execute. Built by scripts/gen_brief_pack.py over the SPEC-106 context pack (knowledge_graph/context_pack/{frame,selector,packer,relevance,cache}.py), task-type-aware through context_injector.inject_for_task, assembled by context/context_assembler.py, bounded by context/context_budget.py.

Minimum contents, tailored per volk:

• The requirement ids in scope and their acceptance criteria, from the spec
• The slice of the Mermaid contract diagram it implements
• The generated signatures it must fill (Junior) or review (senior_dev)
• The tests that must pass — TST-* ids and their bodies
• The code-review rules and the lexicon
• The blast radius / impact set for the files in scope, from a fresh AST
• Prior decisions bearing on this packet, so it does not relitigate them
• Its escalation target, by name

Completeness is checked, not assumed — factory/projection/completeness.py (rule-driven projection + completeness check, ADR-0006) already performs this class of check. A pack failing completeness blocks dispatch. Dispatching an under-specified brief is a defect belonging to the dispatcher, never to the confused volk.

The question channel — two tiers

Tier 1 — retrievable, so the volk pulls. tools/context_tools.py (SPEC-107 FR-007, on-demand context) gives direct access to AST, source, graph, spec and ledger. No human in the loop; most questions should resolve here.

Tier 2 — not retrievable, so the volk asks. A strictly ordered, acyclic ladder:

J:Junior (Haiku) → senior_dev (Sonnet) → U:Cd CodeDirector (Opus) → me (Opus, in-conversation) → Mohamed

The answer rule — real information, or escalate; never invent

• Every answer carries provenance — a spec id, file:line, a graph query, a ledger run id, or "human decision by <name>". An answer with no source is a guess and is rejected, by the same discipline as the claims gate.
• No level may invent. A level that does not know escalates. "Probably", "assume", "it should be" are rejected answers, not answers.
• "Unknown" is a legal answer that escalates — never one that unblocks with a default. R3's three-valued rule applied to the question channel: unknown must stay distinguishable from false.
• A question never times out into a default. A timeout escalates further. The autonomous run may stall on an unanswered question — that is correct behaviour, and the stall is visible in the live view rather than silent.
• Answers are cached into the run's pack, so parallel volks asking the same question are answered once, not once each.

Question volume is a metric, not noise

A packet generating many questions means the brief pack was under-specified — the fix is the pack, not the volk. Per-packet question counts are recorded in the ledger beside the run, turning the observe-and-improve requirement into a number instead of an impression. This also supplies R5's missing escape hatch: a volk that cannot satisfy the reviewer asks, rather than retrying blindly into an unbounded loop.

Slices — one pipeline intent each, small enough to watch

Each slice is a complete pipeline pass (intake → spec/design → build → QA → compliance → docs), submitted as one intent and observed end to end. Between slices: a retrospective on the workflow itself, and fixes land before the next slice dispatches.

Who executes S0 (resolves R10). I do, in-conversation — inventory verification, the LLM profile, the spec, and the blast-radius proof. The first slice cannot be executed by the mechanism it configures: S0 is what puts each volk on its intended model, so volks cannot run S0. Volks take over at S1, once the profile they run under exists.

Merge cadence — per slice, not once at the end (resolves R1 and R13). Every slice merges to master behind its own full authoritative run (kernel started and gRPC-verified). Two consequences, both deliberate:

• No slice ever builds on unverified work. Affected-set selection is proven in S0, and a full run at each merge is the backstop if the proof has a hole.
• A failed slice never merges. Rollback is therefore trivial: reset the worktree to master and re-dispatch the slice with an amended brief. There is no accumulated broken state for the next slice to build on.

Cost is roughly 8 minutes of full run per slice — about an hour across eight slices, which is the price of not discovering a six-slice-deep defect at the end.

S0 and S1 are the proving slices. The volk pipeline has been demonstrated end-to-end on review, analysis and documentation work types; carrying a sixteen-packet implementation autonomously is a larger claim than anything it has been measured doing. If a slice stalls, escalates wrongly, or produces work that fails review, that is the finding — recorded, fixed, and the workflow improved before S2. Success is not assumed; it is measured per slice.

Branch: feat/spec-NNN-test-run-ledger, cut from origin/master, not local HEAD — edits to files touched by unpushed commits silently no-op on a stale base. Worktree provisioned with its own venv (RUL-021 prohibits symlinking the shared one; a bare uv run re-points the shared editable install and poisons other checkouts), gated by check_worktree_envs.py. Delegated work is watched by swarm_supervisor.py heartbeat — never hand-polled.

Work packets

Each is self-contained and independently dispatchable; the numbering is dependency order, not a mandate to serialise.

Prolog is tested exactly as the Python is (P14)

The .pl rulebases this spec adds are production artifacts and get the same treatment the existing gate rulebases already get in tests/gates/ and tests/prolog/:

The negative-corpus rule is the important one: a false positive in the lexicon gate is a defect in the gate, never something to route around — and the matcher may only be relaxed with a deny corpus proving the genuine violations still deny.

Blast radius — used for selection, and proven safe (P15)

--scope affected resolves through the blast radius, not a heuristic: code_intel.py impact / rts.py (AST-based safe regression selection over the Prolog KB) and the codemod daemon's CodeIntelligenceGraph.

The failure mode that matters is under-selection — an affected set that misses a test produces a false green, which is the same class of defect that started this whole plan. So the selection is proven, not asserted, by reusing the machinery that already exists for it: factory/evidence/mutation.py (SPEC-033 B6 affected-set mutation testing). Mutate a symbol → a test in the computed affected set must die. If nothing dies, the affected set is unsafe and the run must refuse to present itself as covering that change.

Over-selection is a cost, not a correctness bug, and is reported as a number (selected vs. minimal), never silently trimmed.

Runtime evidence → traceability: ALREADY SPECIFIED, Phase 1 BUILT (verified 2026-08-20)

Mohanmed's "call-graph should always feed requirement traceability" is an existing open backlog item, not new design — plans/bdd-runtime-tracing-plan.md:

• Phase 1 DONE — attribute_traces_runtime.py: ctrace-verified coverage join, 4-tier output to docs/trace-attribution.json; gate was a pilot slice reproducing 25/33 and nominating REQ-106-063.
• Phase 3 OPEN — "EXECUTED grade in generate_rtm_evidence.py; provenance static|runtime|both on sync_traces_to_neo4j.py TRACES_TO; add_traces_from_evidence.py --runtime for RUNTIME-CANDIDATE writes." This is exactly the ask.

The safeguards proposed in review already exist in code:

Scoring: support / (log2(1+breadth) * log2(1+spread)) — deterministic, "sorted inputs in, sorted outputs out".

Blocker: T1b records that attribution cannot run today — "needs ~8min per-test coverage regen (current .coverage has no contexts)". Phase 3 begins with a contexts regeneration, not with code.

Runtime call-graph differential — the primary blast-radius validator

Different signal from the traceability work above, and this one IS new. attribute_traces_runtime.py consumes coverage contexts — node-level, which defs executed under which test. Blast-radius validation needs edges — which caller called which callee — and coverage contexts never record that. So the two ideas share the --trace run but consume different data, and nothing in bdd-runtime-tracing-plan.md covers the edge-level differential.

Static blast radius is structurally incomplete on this codebase. code_intel.py impact, rts.py and codemod's query_blast_radius all infer reachability from AST-derived edges, which cannot see dynamic dispatch, DI resolution, plugin/registry lookup, the observer mesh, driver-factory dispatch, CXP handler routing, or Prolog→Python py_call callbacks. This architecture is built from exactly those patterns, so static analysis under-approximates precisely where the system is most dynamic — the false-green source R1 names.

The --trace runtime call graph is ground truth for what executed. Validation is a differential: every caller→callee edge observed at runtime MUST appear in the static graph.

The asymmetry is the discipline and MUST NOT be inverted:

Observed = true; unobserved = unknown, never false — the same three-valued rule as R3 and SPEC-113 FR-006. Inverting it would emit a flood of bogus over-selection findings.

Complementary to, and ordered before, mutation testing:

• Call-graph differential — catches missing edges. Exhaustive over everything the suite ran; cheap.
• Mutation testing (run_mutations, limit=50) — catches semantic under-selection: edge present, test selected, but the test detects nothing. Deep, narrow, sampled.

Run the differential first: broader and cheaper, and it finds structural holes.

Validates all three implementations, and cross-validates them. code_intel.py, rts.py and codemod's query_blast_radius can each be checked against runtime ground truth and against each other — three implementations that disagree is itself a finding (R8's "one ruleset, N implementations" in another guise).

Shared-signal economy. One ORCH_RCG_DIR run pays for this and the traceability-candidate report. --trace yes stops being a rare debug flag and becomes a periodic evidence run worth scheduling.

Limit, stated: validates only code the suite exercises. Validation coverage is bounded by suite coverage.

Possibly half-built: build_runtime_callgraph.py merges runtime capture into plans/ast-ultimate.json — the static snapshot itself. If static and runtime edges already coexist there, the differential is a query over one artifact, not a join across two systems. Verify before designing.

This proof runs in S0, not S7 (resolves R1). It was originally scheduled last, which would have left S1–S6 validated by a selector whose safety was not yet established — the plan's own target defect, reproduced inside the plan. It needs nothing this project is about to build: factory/evidence/mutation.py and run_tests.py both exist today. Prove the selector before relying on it.

AST freshness — fresh before touch (P16)

Any code that is touched, and any selection computed, is based on a fresh AST — including code that was just generated. A builder emitting a new module and then computing blast radius from a pre-generation snapshot under-selects by construction.

Mechanism, reusing what exists:

• The codemod daemon already has inotify file watching and a content-addressed parse cache — riding the daemon is what makes the CIG inherently fresh, and is a second reason the runner belongs there.
• scripts/ast_live_sync.py (file save → graph updated in seconds) and scripts/ast_daemon.py for the warm path; freshness_guard.py for the sweep.
• Freshness is inference, with a three-valued verdict — the pattern tests/gates/test_freshness_inference.py already establishes. A stale KB refuses to answer; it never answers wrongly. Applied here: a stale AST blocks selection and blocks the edit, rather than yielding a plausible wrong set.
• Generated artifacts are re-ingested before the next selection, gated the way check_generated_artifacts.py (RUL-030) already gates generated output.

The daemon does not host its own mutation (resolves R11). The runner is a codemod submodule, and codemod is the warm daemon that enforces freshness over that same tree — so editing refactor/codemod/** mutates code the daemon has loaded, while the daemon is the authority on whether the tree is fresh. Rule: after any edit under refactor/codemod/**, the daemon restarts before serving the next selection, and it refuses to serve selection when its loaded module hashes differ from disk. Self-modification under a warm cache is otherwise a silent correctness hole, not a performance question.

Hazard to encode, not rediscover: two different programs answer to ast-sync on this machine — the repo venv's serializer (writes plans/ast-ultimate.json) and /usr/local/bin's graph syncer (never does). PATH resolution silently runs the wrong one and exits 0. Every refresh call site invokes by absolute venv path. Related: never launch the daemon with uv run — it holds the machine-global uv cache lock for the child's entire life.

Technical detail per packet

Phase 0 — verify the inventory (non-negotiable first step). Every "exists" claim in this plan came from docs/CODEBASE_MAP.md one-liners. Confirm via serialize_codebase_ast.py symbols/source before writing a line: pytest_live_observer, observe_pytest, test_durations, whatever writes run-tests-latest.json, gateway/ring_buffer, codemod/{server,client}, project_resolver, task_context_hook, test_run_context, error_report, flake_report/flaky_test_hunter, and whether reports/ is tracked or ignored. Correct the plan where it is wrong.

Phase 1 — event spine. Versioned event schema in refactor/codemod/testrun/events.py. Extend pytest_live_observer to XADD to test:run:{project}:{run_id}:events, keeping its existing SIGKILL-durable local write as truth-of-last-resort. If Redis is unreachable the run proceeds and the record is stamped transport_degraded: true, surfaced in the live view — reported, never swallowed. Pure pytest-free fold in fold.py; ledger append in ledger.py.

Phase 2 — runner. refactor/codemod/testrun/ with __version__; wired into codemod/server.py JSON-RPC and a scripts/codemod.py test verb. Scope via the daemon's CIG + affected_tests.py/rts.py. Threaded through core/threading/test_executor.py. Kernel started + gRPC-verified via start_test_services.py; kernel_verified recorded on run_started. A record carries authoritative: true only when scope is full and the kernel verified.

Phase 3 — Prolog + determinism. prolog/test_lexicon.pl (generated) and prolog/test_discipline.pl (the one rulebase). scripts/test_intel.py with last, last-full, last-scoped, run <id>, slowest, compare <a> <b>, flaky, explain <goal>. Ground facts only, asserted from Python atoms (janus); variable-bearing rules stay in .pl.

Phase 4 — enforcement. Stop-hook claims gate. PreToolUse(Bash) rule denying non-runner pytest. Runner prints the rulebase banner and refuses to start on banner/rulebase drift. task_context_hook.py injects ledger + rules + project scope into agent briefs.

Phase 5 — consumers. test_run_context.py reads the live stream + last-run facts. error_report.py gains scope/skips/warnings. observe_pytest.py attaches to the stream; test_durations.py gains run-over-run comparison. Gateway/UI via gateway/test_runner.py + handlers/test_metrics_handler.py.

Phase 6 — KG sync. Extend knowledge_graph/canonical_schema.py before any import (ontology-before-graph-import rule), then an idempotent sync writing one TestRun node per run; extend kg_retention.py. NDJSON is local exhaust (git-ignored); orchkg is the durable shared record that survives a fresh clone.

S0 Group B — SPEC-113 ALREADY OWNS PART OF THIS (2026-08-20)

specs/113-measurement-retention/spec.md exists (Mohamed, 2026-08-19) and is authoritative. It must be reconciled with, not duplicated.

Spec drift — WIRING VERIFIED 2026-08-20. All three FRs marked UNBUILT are in fact built, wired and tested:

The drift was already tracked in docs/architecture/measurement-retention.md (all three logged as "IN PROGRESS (uncommitted) — spec text predates this code landing"). That doc is now stale in turn: it flags tests/core/threading/fixtures/fixture_cost_module.py as untracked ?? (now committed, git status clean), cites the hook at :715 (actually 633–688), and asserts "zero hits in tests/" for pytest_fixture_setup (now exercised at test_live_observer.py:811–1029).

ONE GENUINE GAP SURVIVES — REQ-113-FR3-002. FR-003 has two requirements. The instrument (-001) is built; the report (-002) is not: "ONE gated run SHALL suffice to produce a per-fixture cost report naming the top-N fixtures by total setup seconds." The hook writes fixture-durations-{pid}.ndjson; the only reader of that glob is a test helper (test_live_observer.py:835). No reader, no report. SPEC-113 is one reader away from FR-001..FR-005 all being satisfied.

Actions: (a) build the FR3-002 reader — a natural sibling of test_durations.py and a good S0/S1 warm-up on the real codebase; (b) correct SPEC-113's status lines for FR-002/003/005; (c) refresh docs/architecture/measurement-retention.md.

FR-006 + Non-Goals change the Prolog design

FR-006 bans asserting statistical measurement — explicitly including flake rates and durations — into any GateEngine-consulted rulebase. Non-Goals list "A Prolog fact base for the observation stream" as explicitly dropped (370× slower than dicts; every proposed query is a GROUP BY, not a recursion).

The test_run/6, run_totals/6, test_outcome/4, flaky/3 contract written earlier in this plan is that fact base and contradicts the spec.

Reconciliation (satisfies both the spec and Mohamed's stated requirement): Prolog holds the rules — what makes a lexicon term legal — and reaches the measurements through py_call python atoms into deterministic Python readers (scripts/test_durations.py and successors). Nothing is asserted, so FR-006 holds; no fact base is built, so the non-goal holds; explain.pl proof trees still work. This is exactly the "or access to it through python atoms" Mohamed specified.

Three conditions make the atoms honest rather than a dodge (Mohamed, 2026-08-20 — "save the aggregate queries as python atoms that prolog can use to answer"):

1. Atoms return EVIDENCE, not booleans. FR-006 mechanism 5 is that explain.pl renders every unit clause as an identical leaf. A py_call goal is not a unit clause, so without care the proof tree dead-ends at the Python boundary — "true because an external call said so" is not a derivation. The atom returns the supporting rows (which runs, which outcomes, which commits) so explain can still answer why.
2. Atoms return a THREE-VALUED verdict. FR-006 mechanism 3 is that statistical facts are invisible to freshness — "outside its safety net, not failed by it." A reader can fix what an asserted fact cannot: it returns unknown when the sink is stale or does not cover the runs in question. R3's rule, enforced at the boundary; unknown never collapses to false.
3. Atoms MEMOIZE per run_id. The durations sink is 392 MB / 2,085,361 records (measured 2026-08-20). Re-parsing it per Prolog goal would make the gate unusable. Compute once per run; cache on the FR-001 identity key.

All five FR-006 mechanisms are satisfied: nothing enters the flat namespace, nothing alters the 142 \+ sites, nothing sits invisible to freshness, no confidence-threshold truncation, and no statistical fact masquerades as a hash-verified one in a proof tree.

FR-006 mechanism #3 independently confirms R3, and words it better: statistical facts are "INVISIBLE to the freshness discipline — outside its safety net, not failed by it."

Corrections to this plan from Group B

1. Next free spec number is 115 (113 = measurement-retention, 114 = governance rule learning). This plan previously guessed 113.
2. Specs are DIRECTORIES: specs/NNN-slug/spec.md + checklists/, governed by specs/SPEC_STANDARD.md, validated by specs/spec-review/validate_spec_standards.py, verify_all_specs.py, verify_requirements.py.
3. Requirement id format is REQ-113-FR1-001 — REQ-<spec>-FR<n>-<nnn>.
4. R7's resolution is REVERSED — the spec wins. REQ-113-FR1-001: a bare-pytest row OMITS run identity; "omission means 'not run under the executor'; writing null would claim the field was measured and found empty." Encode absence by omission, not by a scope: unknown sentinel.
5. reports/ is TRACKED (51 files); exhaust is ignored per file (test-durations.ndjson, test-phase-durations.ndjson, container-cpu.ndjson). So reports/test-runs.ndjson and reports/test-run-artifacts/ need explicit .gitignore entries, or every run dirties the tree.
6. reports/run-tests-latest.json is TRACKED and not ignored — R9's deletion is a git rm, and run results are currently committed as a practice.

S0 Group A — inventory VERIFIED (2026-08-20)

Read via docs/CODEBASE_MAP_FULL.md + AST-toolkit source. These supersede the map-level guesses above.

Consequent plan changes:

1. P1 shrinks — add a third observer, not an emitter rewrite.
2. P2 shrinks — extend RunHeader/RunSummary; the fold already exists.
3. R11 shrinks — reload() already exists on the codemod client.
4. Sync client in the drain thread — NOT an async bridge. RedisRingBuffer is async; pytest hooks are sync and must be, because pluggy's _multicall dispatch never awaits, and pytest deliberately owns no event loop so that pytest-asyncio / anyio / trio can each own loop policy. (pytest-asyncio wraps async tests from a sync hook; it does not make hooks async.) Resolution: the R14 drain thread uses a sync redis-py client and reuses serialize_for_redis, which is a module-level def in gateway/ring_buffer.py and therefore usable without the async class. A dedicated drainer gains nothing from async — blocking XADD there is correct, and under free-threaded 3.14t a blocking socket call does not impede the test threads. No event loop, no bridge, no new dependency.
5. A9 correction against my own claim — the brief pack is a ~70% extension, not a reuse.
6. A7 — wrap resolve_project_id to raise on None, per the no-default rule.
7. A4 — investigate _try_reuse_report / _supersede_module_totals as the origin of the 17-test fragment before deleting the artifact (R9).

Also discovered — code-review.py's real rule surface (the set to inject): RuffChecker, RuffFormatChecker, MyPyChecker, StyleChecker (docstrings, type hints, naming, imperative accumulator, complexity, test isolation, code patterns), ChecklistChecker, TraceabilityChecker (@traces), ConnectionStringChecker (_is_env_get_call, _collect_env_get_defaults), check_banned_dependencies, check_prolog_boundary.

Review 1 — adversarial findings (ALL RESOLVED INTO THE DESIGN ABOVE)

Self-review; treat accordingly. Ranked by likelihood of biting in execution. Each row's fix is now written into the section named in the third column — this table is the audit trail, not an outstanding to-do list.

Constraints carried from project doctrine

• No mocking; real infra (real Redis). Kernel started + gRPC-verified or the run is non-authoritative and the record must say so.
• Never run blind — the event stream is the observation channel.
• Consumers split by kind (resolves R6). One rule does not fit both:
  • Interactive — test_run_context.py fires synchronously on every UserPromptSubmit. It must not block: non-blocking XREAD with a hard timeout, degrading to "run state unknown" rather than hanging the prompt. A blocking read here would hang every prompt whenever Redis is degraded, and this repo has already shipped that bug once (a timeout test silently waiting its full 30s).
  • Streaming — the CLI observer blocks on XREAD BLOCK.
  • Neither ever spins.
• Ledger must record a run that was killed/timed out, not just clean exits.
• --affected runs are non-authoritative by construction — the scope field is what finally makes that machine-checkable instead of prose in context.json.

Verification (to fill in once designed)

Real infra, no mocking, per project doctrine. Each item is a PASS/FAIL with numbers.

The end-to-end acceptance test is this conversation: ask "tell me about the last test run" and get time, scope, wall time, trigger, last-full and last-scoped — sourced from the ledger, not reconstructed from prose.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks