Orchestrata Ultimate › Plans › traceability-repair-2026-08-13

Traceability repair — session 264, 2026-08-13

Source: plans/traceability-repair-2026-08-13.md at 8a2cb42ad · last changed 2026-08-13 · first committed 2026-08-13 · 137 lines.

Prose pending.

Plan text

Traceability repair — session 264, 2026-08-13

Status: paused mid-verification. The plumbing is fixed. Traceability itself is not. Read the last section before resuming; the numbers got worse on purpose.

Commits

The through-line

Every defect this session was the same shape: a reader that could not tell "there is nothing here" from "I did not look here."

1. Two output locations. ca71ec06 moved rtm-evidence.md to reports/ but left the generator's argparse defaults and project_status.get_rtm_summary on docs/. For ten days the session status line summarised a stale copy (5,284 rows) while the fresh matrix sat unread. Neither component was wrong on its own; the split was only visible from outside both.

2. Two readers of one corpus. serialize_codebase_ast._requirement_source_files had already ratified the corpus as three shapes — checklist, spec.md, *-addendum.md — after finding a checklist-only scan made 43 real @traces look dangling. generate_rtm_evidence.load_requirements never received that fix. So trace-lint called 21 citations clean while the RTM graded them GHOST, and 68 requirement ids across 6 files were absent from the denominator — every coverage percentage was computed against a wrong total.

3. Thirteen specs with no ids at all. They expressed requirements as ### FR-001: sections and carried no REQ- id anywhere, so they contributed zero of the universe. Not UNLINKED, not GHOST — unaddressable.

The apparent "missing checklists" were a symptom of (2). Authoring one for SPEC-033 would have papered over a loader still blind to fifteen other specs. The adversarial verifiers independently refuted the fix-the-checklist dispositions in favour of the loader fix.

What changed

• load_requirements delegates to requirement_source_files(), which loads the toolkit's canonical scanner by path (the ast_report._load_tool pattern). Delegating rather than re-implementing is the point — a second copy of the scan recreates the divergence.
• requirement_ids_in() extracted as the single definition of "this text defines requirement X", shared by loader and gate.
• scripts/check_requirement_corpus.py (new gate): no src/ @traces may cite an id the RTM cannot grade. Wired into gates.py and health_check.py.
• 1,636 REQ ids assigned across 13 specs, per docs/req-id-grammar.md.

The gate's first version was worthless — worth knowing why

It compared load_requirements() against the corpus function load_requirements() itself calls. A tautology: it could never fail. That is the same false green as a checker that never looked. It now compares against @traces harvested from the AST, which knows nothing about how requirements load and can genuinely disagree. Proven both ways: 29 violations against the pre-fix loader, 0 after. A gate that has never been shown to fail is not evidence of anything.

Measured outcome

Remaining 39 ghosts are all test-side; 32 are cited only from docstrings in the RTM tooling's own fixtures (filed — the harvester counts a docstring mention as a citation).

Traceability is NOT fixed — read this

UNLINKED rose from 41% to 55% and CITED fell from 5.3% to 4.1%. Nothing regressed — we stopped hiding 1,704 requirements we cannot prove. The old percentages were computed against a corpus missing 14 specs.

Three facts matter more than the table:

• TESTED is zero. Every grade here is static. --verify, which runs the citing test modules and grades TESTED/FAILING, has never been run. No number in this matrix is execution evidence.
• 41% rests on decorators, and the generator's own docstring measures decorator claims at 55% ± 5% accurate. The largest evidence class is close to a coin flip.
• 25 of 61 specs have ≥5 requirements and not one cited by any test — 1,274 requirements with zero test naming them.

Resume here, in order

1. python scripts/generate_rtm_evidence.py --verify — converts CITED into real execution evidence. Largest honesty gain available, already built, never run.
2. Finish the adversarial verification of the grammar adoption (13/13 adopted, ~2/13 verified): Workflow({scriptPath: '~/.claude/projects/-home-mohamed-workspace-orchestrata-ultimate-architecture/1ce729ef-f537-43d8-99b5-5c32d6b06ae9/workflows/scripts/adopt-req-id-grammar-wf_a3d6c70b-2b2.js', resumeFromRunId: 'wf_a09d31d1-f98'}) — adopt agents replay from cache; only verifiers re-run. They check two things I could not fully confirm by hand: that no id imposes an obligation its spec never stated, and that the FR- → REQ- relabelling dropped nothing.
3. Fix check_staged_secrets.py — DONE, 9454e2dc. It scanned whole staged blobs, so any spec edit re-flagged pre-existing doc samples and forced --no-verify; it now scopes to the lines a commit adds. Proven both directions: an untouched pre-existing line no longer blocks, a secret added by the same commit still does, and new files plus initial commits are scanned in full. That commit went through the gate it fixes without a bypass.
4. The 1,274 uncited requirements in those 25 specs. Test-writing, not tooling.
5. Decide what DECLARED is worth at 55% accuracy — strengthen the convention until it is trustworthy, or stop counting it as evidence. This one needs Mohamed's ruling.

Judgment calls made, in case they need reversing

• reports/ is canonical for RTM evidence, not docs/ — that is where generated run output lives, what the RUL-030 gate already registered, and where ca71ec06 put the .md.
• The loader delegates rather than re-implements the corpus scan.
• REQ ids were ASSIGNED to obligations the specs already stated, never authored. Where **FR-001:** The system MUST ... became **REQ-062-011**: The system MUST ..., wording carries verbatim — which is why diffs show deletions despite this being a relabelling.
• US-110-013 was backed rather than renumbered — four shipping modules already cite it.

One correction on the record

Mid-session I reported that SPEC-062 had lost normative content, having read 12 added lines and extrapolated. It had not: all 15 named tools survive (7–22 mentions each), as do the PAM++ mapping and the 20-test/90%-coverage bars. The deletions were FR- → REQ- label swaps preserving wording exactly. I also claimed 14 specs had "missing checklists"; they had no REQ- ids at all, which is a different defect with a different fix.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks