Home / Projects / Orchestrata Ultimate / Decisions / ADR index / ADR-0028

ADR-0028: Completeness axes are delivered as a construction procedure, not a checklist

← ADR-0027 · ADR-0029 →

The record

• Status: Accepted
• Date: 2026-07-24
• Spec: SPEC-033 §6 (REQ-033-C1), §2 (DoR)
• Amends: ADR-0015

Context

ADR-0015 established that Definition of Ready is necessary but not sufficient, and that the gap is closed by forced-choice axes: for each requirement, every completeness axis must be answered SPECIFIED / N/A (with reason) / GAP, with silence disallowed. The reasoning was that recognition beats recall — a reviewer asked "what about timeouts?" answers better than one asked "what's missing?".

The reasoning is right. The delivery mechanism is empirically the weakest of the three known techniques.

Porter, Votta & Basili (Comparing Detection Methods for Software Requirements Inspections: A Replicated Experiment, IEEE TSE 21(6), 1995), replicated twice — with professional subjects (Porter & Votta, ESE 3(4), 1998) and independently (Fusaro, Lanubile & Visaggio, ESE 2(1), 1997) — compared ad-hoc reading, checklist reading, and scenario-based reading. The result: scenario-based reading significantly outperformed both; checklist reading was not reliably better than ad-hoc. A flat list of axes to tick is, measurably, barely better than handing the reviewer the document and asking them to look hard.

What works is a procedure that forces the reviewer to construct an artefact, with the questions attached to each construction step. Perspective-Based Reading (Shull, Rus & Basili, IEEE Computer 33(7), 2000) operationalises this: the reviewer builds the work product their perspective implies — the tester builds equivalence classes and boundary tests, the designer a high-level design — and the defects surface as a by-product of not being able to complete the construction. Cost is 0–30% more reviewer effort, much of which yields a reusable high-level test plan rather than pure overhead.

This matters more for us than for a human review process, because our reviewer is frequently a model, and a model handed a checklist will produce a plausible tick for every row. A construction step cannot be ticked — it either produces the equivalence classes or it visibly cannot.

Decision

The completeness axes are retained in full, and are delivered as a construction procedure. Forced choice applies to the residue, not to the whole list.

The procedure, in order — each step forces a subset of the axes:

1. Enumerate the inputs of the criterion. Are they all specified? Are any specified but unused?
2. Build the equivalence classes and boundaries for each input — at minimum one valid class and two invalid classes (below the low extreme, above the high extreme); for set membership, the set and its complement. Forces: input domain, empty/zero/absent, units and precision.
3. Write the expected behaviour for every class, including every invalid class. Forces: error handling, output legality.
4. Walk the state machine — every state × every input, plus the no-input case. Forces: timeouts, applicability conditions, startup/shutdown/mode transitions, ordering, concurrency.
5. Walk the failure envelope — dependency down, dependency slow, dependency partially correct, retry, restart. Forces: graceful degradation, idempotency, durability.
6. Walk the limits — rate, size, count, time. Forces: capacity, latency, data age, pagination.
7. Answer the residual axes as forced choice — measurable threshold, authorization default, feedback loop, observability, the systematically-omitted non-functionals, time/timezone, encoding and locale, domain-ambiguity probe, external completeness, divergent-interpretation probe.

The construction output is an artefact, retained as evidence — not a set of ticks. Step 2–3's output is a high-level test plan, which feeds ADR-0012's requirement-derived anchors directly.

The axes themselves carry provenance tiers, so their authority is not overstated:

• [A] derived from validated research — Leveson's completeness criteria for process-control systems (the source of: every state has a transition for every input; the disjunction of transition conditions out of a state must be a tautology; every state has a timeout behaviour; output value and timing; environmental capacity; data age; graceful degradation; feedback), PBR's defect taxonomy, ISO/IEC/IEEE 29148, Firesmith's list of systematically-omitted non-functionals.
• [B] derived from rigorous engineering literature outside RE — Waldo et al. on partial failure and concurrency; Saltzer & Schroeder's fail-safe defaults for the authorization axis.
• [C] engineering convention with no formal source — idempotency/retry, pagination, timezone/DST, encoding/locale/collation. Included because the failure mode is common; marked honestly so nobody cites a convention as a standard.

Alternatives rejected

• Flat forced-choice checklist (ADR-0015 as written) — measurably no better than ad-hoc reading, and uniquely vulnerable to a model producing a plausible tick per row.
• Scenario-based reading without the axis list — stronger on the axes the scenario touches, blind to the ones it does not; the residual forced-choice step in 7 exists to cover those.
• Drop the [C] axes for lacking a research source — they encode real, recurring failures; the honest fix is labelling their provenance, not deleting them.

Consequences

• DoR becomes more expensive per requirement: 0–30% more effort by the PBR measurement, and it produces an artefact that must be stored and versioned.
• That artefact is not waste — steps 2–3 produce exactly the example and boundary anchors ADR-0012 requires, so the cost is partly a prepayment on anchor authoring rather than an addition to it.
• ADR-0015's "forced choice" framing narrows to step 7. Its core claim — DoR is necessary, not sufficient — is unchanged.
• Requires the procedure to be executable by a model, which means each step needs a machine-checkable output shape (an enumeration, a class table), not prose.

Provenance

2026-07-24 requirements-quality research. Porter/Votta/Basili and its two replications for the checklist-vs-scenario result; Shull/Rus/Basili for the PBR procedure and its cost; Leveson (FMSP 2000) for the completeness criteria; Firesmith (JOT 6(1), 2007) for the omitted-non-functionals diagnosis.

Related ADRs

• Amends: ADR-0015
• Also mentioned: ADR-0012

Rulings cited

None.

Specs cited

• SPEC-033