Home / Projects / Orchestrata Ultimate / Decisions / ADR index / ADR-0029
ADR-0029: The lexical incompleteness detector set is tiered by measured precision
| field | value |
|---|---|
| Status | Accepted |
| Date | 2026-07-24 |
| Register | factory (docs/adr, 4-digit ids) |
| Source | docs/adr/0029-lexical-detector-set-tiered-by-measured-precision.md — permalink at bdafa201a (unchanged on master since that commit) |
| Wiki register entry | ADR-0029 (wiki status: Accepted) |
The record
- Status: Accepted
- Date: 2026-07-24
- Spec: SPEC-033 §6 (REQ-033-C3), task B13
- Relates to: ADR-0011, ADR-0024, ADR-0028
Context
REQ-033-C3 specifies "cheap lexical detectors" for incompleteness without saying which ones, and ADR-0024 makes a deterministic lexical prefilter the first stage of the assumption-matching cascade — so how good that stage is directly determines B13's model and human cost. "Lexical detectors" as a phrase is not implementable, and the obvious implementation (a hand-rolled bad-words list) has a known failure mode: it flags constantly, reviewers learn to dismiss it, and the stage becomes noise.
The field has measured this. ISO/IEC/IEEE 29148:2011 §5.2.7 enumerates the unbounded and ambiguous language classes, and Femmer et al. (Rapid quality assurance with Requirements Smells, JSS 2017) derived nine detectors exhaustively from that clause and measured each on 51 industrial artefacts / 266,432 words / 11,608 findings. The per-rule precision varies by a factor of nearly four:
| Detector | Precision | Recall |
|---|---|---|
| Subjective language | 0.96 | 0.86 (dictionary family) |
| Ambiguous adverbs/adjectives | 0.81 | 0.86 (dictionary family) |
| Loopholes / escape clauses | 0.72 | 0.86 (dictionary family) |
| Open-ended, non-verifiable terms | 0.70 | 0.86 (dictionary family) |
| Superlatives | 0.49 | 0.50 |
| Comparatives | 0.48 | 0.95 |
| Negative statements | 0.33 | 0.84 |
| Vague pronouns | 0.26 | 0.92 |
Treating a 0.26-precision rule the same as a 0.96-precision rule is the design error. The same study also diagnosed why the weak rules are weak, which is directly actionable: ~48% of comparative false positives were comparisons against absolute numbers inside a condition ("if the system takes more than 1 second to respond") — not a defect. The same holds for negation inside a condition.
Two further findings shape the design:
- The Berry Recall Condition (Tjong & Berry, REFSQ 2013): for requirements tooling, recall dominates precision, because a reviewer who cannot trust the tool to be exhaustive re-reads the whole artefact anyway and the tool saves nothing. Femmer et al. accept this with Menzies' qualifier — low precision is affordable when a false alarm is cheap to dismiss (seconds, in their experience).
- Scope ambiguity is the highest-value unserved category. Berry, Kamsties & Krieger's handbook
devotes its largest section to it; the trigger tokens are a closed set; and no mainstream requirements
tool covers it. Each hit is a genuine off-by-one:
up to 12(including 12?),byvsuntil,minimum/maximumleaving the interval open or closed,onlyposition,and+orin one sentence with no natural-language precedence.
Decision
REQ-033-C3's detector set is specified explicitly and tiered by measured precision, with each rule carrying its provenance and its measured accuracy where one exists.
- Tier 1 — blocking. Subjective language, ambiguous adverbs/adjectives, loopholes, open-ended non-verifiable terms, weak modals, unresolved placeholders (TBD/TBR/???), missing units, exact duplicates, template violations, incomplete references. These may fail a gate.
- Tier 2 — advisory. Superlatives, comparatives without a baseline, negative statements, non-atomic criteria, coordination and PP-attachment ambiguity, not-well-formed criteria, solution language. Reported, never blocking.
- Tier 3 — question-generating only. Vague pronouns/anaphora, quantifier scope, operator placement, boundary words, non-specific temporal words, under-specified head nouns, deictic references. These produce questions for the ADR-0024 cascade, never verdicts.
Three design rules apply globally, each derived from a measured failure:
- Exempt conditional clauses from the comparative and negation detectors. This is the single highest-yield precision fix available and it is measured, not guessed.
- Optimise for recall; make dismissal cheap. Findings are presented spell-checker style — inline, one-click dismissible, with per-rule disable and a persistent blacklist. A finding is an indicator, never a verdict.
- Weight by location. Findings in a rationale clause ("so that…") are ~2× denser and empirically less relevant. Down-weight; do not suppress.
No detector in any tier may issue a completeness verdict — that is ADR-0011's boundary, and the measured ceiling for automated completeness checking (coverage ≈38% at precision ≈12%) is why.
Alternatives rejected
- A single undifferentiated bad-words list — treats a 0.26-precision rule as equal to a 0.96 one; trains reviewers to dismiss everything.
- Only the high-precision rules — violates the Berry Recall Condition; a reviewer who knows the tool is not exhaustive gains nothing from it.
- Only an LLM detector — measured precision 0.13–0.50 on requirement characteristics with systematic over-flagging, and it cannot be a deterministic prefilter. It belongs in ADR-0024's cascade as a question generator.
- Add a passive-voice detector to Tier 1 — worth stating explicitly, because it is a very common assumption: passive voice is not an ISO 29148 §5.2.7 smell and not in Femmer's catalogue. It is INCOSE rule R2, which the INCOSE automation analysis classifies as not tool-automatable. It goes in Tier 3 with its weak provenance stated, or not at all.
Consequences
- REQ-033-C3 becomes implementable and testable: each rule has an expected precision to regress against, so detector quality is measurable rather than asserted.
- Tier 1 blocking means a false positive can block work. Accepted only because every Tier-1 rule measures ≥0.70 precision and dismissal is one click.
- The condition-exemption requires a clause parser, not a word list — a real implementation cost, bought with a measured ~48% reduction in comparative false positives.
- Tiers must be revisited against our own measured precision once running; the published figures come from natural-language industrial requirements, and acceptance criteria in our format may behave differently.
Provenance
2026-07-24 requirements-quality research. ISO/IEC/IEEE 29148:2011 §5.2.7 for the smell classes; Femmer, Méndez Fernández, Wagner & Eder (JSS 2017) Tables 6–8 for precision/recall/prevalence and the condition-exemption diagnosis; Tjong & Berry (REFSQ 2013) for the Berry Recall Condition; Berry, Kamsties & Krieger (2003) §5.2–5.7 for the scope lexicon; QRA's INCOSE automation analysis for the passive-voice correction.
Related ADRs
Rulings cited
None.