Home / Projects / Orchestrata Ultimate / Decisions / ADR index / ADR-0024

ADR-0024: The assumption-matching arbiter and its independence guarantee

← ADR-0023 · ADR-0025 →

The record

• Status: Accepted
• Date: 2026-07-24
• Spec: SPEC-033 §6 (REQ-033-C1), task B13

Context

B13's core operation is: "each assumption is matched to a pinning artefact (AC / anchor / ADR), or raised as a blocking question." That match is itself a semantic judgement, and the spec never says who makes it. The three candidate arbiters all fail differently:

• The executing model self-judges ("this assumption is covered by AC-3") — this is precisely the self-report the design forbids everywhere else (§1 Why, ADR-0008). It converts B13 into a second rubber stamp with extra steps.
• An independent model tier judges — stronger, but costs an escalation-shaped resource B13's budget never allocated.
• Lexical/string matching (the repo's existing spec_completeness.py pattern) — cheap and deterministic, but weak: it will report "matched" on a loose paraphrase.

This matters more than its size suggests, because B13 is the first thing we build (Phase 0), and this choice decides whether it produces real signal.

The requirements-quality literature settles the shape of the answer. LLM judging of requirement characteristics measures precision 0.13–0.50 with recall 0.74–0.89 — systematically over-flagging, and worst precisely on the characteristics needing external knowledge. Rule/NLP checkers (Smella) reach precision ~0.59 at recall ~0.82. The consistent finding across the field: these tools are high-recall, low-precision — good hypothesis generators, poor verdict-givers.

Decision

A three-stage cascade, with the model used only where its high recall is an asset:

1. Deterministic pre-filter. Lexical/structural matching of the assumption against ACs, anchors and ADRs. A match here is provisional evidence, never sufficient on its own.
2. Independent-tier adjudication for anything not deterministically resolved — performed by a tier that is not the author of the assumption, mirroring ADR-0022's separation. Its output is matched / unpinned / needs-human, and it may not adjudicate its own enumeration.
3. Human adjudication for anything the independent tier marks needs-human, or where the two stages disagree. Adjudications are then promoted into the Project KB (ADR-0014) so the same assumption is never adjudicated twice.

The model never issues a bare verdict on its own assumptions. Where a model is used, it is asked to produce questions and candidate pinnings, not conclusions.

Alternatives rejected

• Self-judgement — reintroduces the self-report the whole design exists to eliminate.
• Human-adjudicates-everything — correct but unaffordable; the cascade reserves human attention for the residue.
• Lexical matching alone — false "matched" on paraphrase; would silently pass the very assumptions that matter.

Consequences

• B13's cost is bounded by how much the deterministic stage resolves; investment in the lexical stage directly reduces model and human load.
• Requires the independence property to be enforced structurally (different tier, no access to the author's rationale), not merely asserted.
• Introduces a dependency from Phase 0 onto ADR-0014's KB promotion, or the cascade has no memory and repeats per change.

Provenance

Round-3 Sonnet, which identified the unstated arbiter as "the one real design gap in the new material" and the choice that determines whether B13 delivers signal. Precision/recall figures from the requirements-quality research (Femmer et al., Smella; LLM requirement-QA evaluation).

Related ADRs

• Also mentioned: ADR-0008, ADR-0022, ADR-0014

Rulings cited

None.

Specs cited

• SPEC-033