Orchestrata Ultimate › Plans › trace-remediation-plan

Trace Remediation Program — operational runbook

Source: plans/trace-remediation-plan.md at 8a2cb42ad · last changed 2026-08-19 · first committed 2026-07-08 · 345 lines.

Prose pending.

Plan text

Trace Remediation Program — operational runbook

Session 224 (2026-07-08). Owner: orchestrator (Opus) + Sonnet swarm waves via .claude/workflows/trace-wave.js. Status ledger at the bottom is the single source of progress truth; update it at every wave boundary.

Objective: every @traces decorator in src/ carries only requirement ids whose spec text genuinely describes the decorated def; everything unmappable is an explicit, recorded spec-gap finding; downstream artifacts (RTM, Neo4j KG, Memgraph IMPLEMENTS layer) rebuilt from the cleaned traces; deterministic gates prevent regression. Ground truth: specs/*/checklists/requirements.md — 4,864 unique REQ ids across 33 specs (specs are authoritative; the orchestrator never adds/edits requirements — W9 drafts go to the user for ratification).

1. Problem statement — measured, not estimated

Census 2026-07-08 (AST parse of every decorator, saved as trace_census.json): 5,061 @traces decorators carrying 14,395 requirement-id occurrences.

Root cause: historical bulk-annotation generated traces mechanically (first-req stamp, FR-section paste, sequential enumeration). check-traces only detects dangling, so borrowed-but-real ids passed every gate. Verified empirically in W1: 43% of sampled mappings were wrong while all gates were green.

2. Invariants — bound into every agent brief (see trace-wave.js INVARIANTS)

1. A req id may be written only if it exists in the requirements index; read the requirement's FULL text (index → source file) before judging.
2. "Exists in the index" ≠ "correct" (the W1 43% lesson, quoted verbatim in briefs).
3. Never fabricate: no genuine match → untraced + unmapped:{file,symbol,reason}.
4. Only decorators + the traces import may change. Never executable code.
5. Sparse-and-accurate beats padded. Class decorator = the class's responsibility (a handful of reqs); method decorator = that method's behavior.
6. Agents never touch the git index. Orchestrator stages/commits per wave.
7. Every wave = map/fix (Sonnet) → independent adversarial verify (Sonnet), structured outputs, per-packet gates. No wave self-certifies — W6 is the program-level gate with fresh eyes.
8. Multi-spec ids in one decorator are legal (spec= names the primary). Superseded 2026-07-09: every req id must be defined by the spec the decorator claims (trace-lint spec-mismatch, membership-based). Mixed provenance = one stacked @traces per spec.

2.5 Judgment rubric — what "genuinely describes" means

Match classes, from the W2a keeps/strips that survived adversarial re-audit:

• A. Verbatim/mechanism match — KEEP. The requirement names the exact artifact or mechanism the code implements. Ex: REQ-020-019 "annotated git tag in format session-NNN-YYYY-MM-DD-HHMM" on _create_session_tag building that literal string; REQ-021-004 naming KnowledgeGraphContext.get_file_context() with the caller proven in opencode/plugins/orchestrata.ts.
• B. Semantic match — KEEP. The requirement describes this def's specific behavior in different words (REQ-001-US-004-023 "Clean up session resources" on a method that deletes session state). Must be about THIS operation, not the subsystem in general.
• C. Subsystem-adjacent — STRIP (or move). Right spec, wrong requirement: the req describes a different method's behavior (REQ-006-US-007-005 "Session list can be queried" already correct on all_sessions() does NOT also cover session_count()). Plausible-but-wrong is wrong.
• D. Name-echo / keyword bait — STRIP. Shares a keyword only (REQ-008-034 "Audit trail maintained…" on an int counter with no persistence/actor/time).
• Evidence hierarchy for keeps: named artifact/format/method in the req text > described mechanism > described outcome. A keep justified only by package affinity or keyword overlap fails verification.
• Replacement discipline: when stripping, look for the correct id only where it is obvious from the def's body + the index (class D3/D4 waves: the right id usually exists nearby in the same FR section). Otherwise strip and record unmapped — W5/W9 own completion.

Business-facing vs infrastructure code (user ruling, 2026-07-09, during W6.5). Traceability is two-lane:

• Business-facing code (volk behaviors, handlers, pipeline stages, policy decisions, protocol semantics) implements specific FRs → trace to them.
• Infrastructure / cross-cutting code (buffers, caches, pools, observers, serializers, retry helpers, connection factories) serves many consumers. Trace it ONLY to an explicit NFR/architectural requirement it genuinely implements; otherwise it is deliberately untraced with unmapped reason infrastructure — serves many consumers; business reqs live on the consumers. That end state is CORRECT, not a spec gap: W9 triage must separate untraced-infra (by design) from untraced-business (real spec gap). The business req belongs on the consumer def; stamping a utility with every consumer's req is the D3 fan-out defect. (Canonical bad example: RingBuffer.peek() carrying "Implement autonomy verification".)

3. Wave DAG

4. Wave procedures

Common per-wave loop (the wave protocol):

1. ENTRY: prior wave committed; python scripts/run_tests.py green; no other src-writing agents in flight.
2. SIZE: run python scripts/serialize_codebase_ast.py trace-audit --json <out> (post-W0); derive the packet file (orchestrator, in scratchpad; ~60 sites or one coherent file-set per packet; packetize by FILE so no two agents edit one decorator).
3. BRIEF: write the wave brief file (defect class, judgment prior, rubric pointer, req-index path, packet-specific notes).
4. LAUNCH: Workflow({name:'trace-wave'}, {briefPath, packetsPath, packetIds}).
5. TRIAGE the aggregate: every failed/unclean packet gets read (agent transcript), then a targeted fix packet or an inline orchestrator fix.
6. EXIT GATES (exact commands):
   • python scripts/serialize_codebase_ast.py check-traces → 0 dangling/malformed
   • python scripts/serialize_codebase_ast.py trace-lint → 0 errors (post-W0)
   • wave-specific metric (below)
   • uv run ruff check src/ tests/ + diff-scan: touched files show only decorator/import lines (git diff -- <files> | grep '^[-+]' | grep -v traces… eyeball the residue)
   • python scripts/run_tests.py → 0 failed
7. COMMIT (template): traces(Wn): <defect class> — <headline numbers> body: packets, fixed/stripped/kept counts, verifier stats, unmapped delta, link to plan §. No AI attribution. Update the ledger (§8) + session CLI log.
8. Refresh census snapshot (trace-audit --json) → next wave sizes from CURRENT state, not stale data.

W0 — instrument (RUNNING: builder)

trace-audit (census/analytics: family_matrix, shotgun, fanout, sequential, convention stats; --json) + trace-lint (deterministic gate: dangling, placeholder, shotgun>25=error 11–25=warn, unknown-spec, malformed; exit≠0 on error) as toolkit subcommands + tests/scripts/test_ast_trace_audit.py. DoD: both green on the live tree; neighbors' tests unaffected; committed with the pending attr-refs toolkit change.

W1 — dangling + untraced (DONE, run wf_330bec66-67b)

575 traced · 924 dangling resolved (468→real, 458 stripped) · 760 spec gaps · verify 242/565 wrong → 216 corrected · 4 failed-verdict packets → spawned W2a.

W2a — wrong-family 020/021/022 (DONE, run wf_6a6e8cce-914)

363 stripped · 32 decorators removed · 12 keeps (all upheld: rubric class A) · c04 re-map traced 50/85 protocol methods with honest unmapped reasons · 1 reporting-integrity finding (r2-session-a1's fixer under-reported ~19 correct-but-undisclosed REQ-100 cleanups — verifier spot-checks passed; W6 covers).

CP1 — checkpoint (IN PROGRESS)

Full suite running; then two commits: (a) seam phase-5 migration (38 files, commit_seam_files.txt), (b) traces W1+W2a sweep (~262 files, decorator-only,

• plan doc + trace-wave.js). Toolkit files held for W0's commit.

W2b — wrong-family purge, all remaining families

• Inventory (Appendix B, upper bound pre-refresh): 48 outlier cells, ~993 occurrences. Prior differs from W2a: SUSPECT, not presumed wrong — the <5% heuristic also catches legitimate cross-spec references (agents×032 "ConciergeVolk registered with DI" is about agents). Expected keep-rate is materially higher; the rubric decides.
• Packets: ~12 (by package, ~80 sites each). Brief lists each cell's family with the spec's title so agents judge with the domain in view.
• Wave metric: every surviving outlier cell justified in kept_genuine.

W3 — fan-out + shotgun re-judgment (the big wave)

• Inventory (Appendix A, refresh at entry): 24 fan-out ids (2,148 occ) ∪ 199 shotgun decorators = 2,299 decorators across 325 files → ~39 packets.
• Brief mechanics: for a fan-out id the agent reads the requirement text ONCE, then judges every carrier def in its files: keep on the def(s) that actually implement it (typically 1–3 tree-wide), strip elsewhere; replace only when the correct id is obvious (usually the same FR section). For shotgun decorators: rebuild reqs from the def's actual behavior; the FR-section paste dies.
• Wave metric (trace-audit after): no id >5 uses without a recorded justification; 0 decorators >25 reqs; ≥10 only with justification.
• Est. 45–70 map+verify agent pairs, 15–25M tokens, 2–3 h pipelined.

W4 — convention normalization (mechanical, NO agents)

One reviewed script pass: spec="SPEC-XXX" → spec="XXX" (majority form); story formats inventoried by trace-audit and normalized to the dominant per-package form; black; separate pure-mechanical commit; then trace-lint enforces the single form (add the check in the same commit).

W5 — coverage completion

find_untraced re-run + the 11 defs skipped as in-flight during W1. Scope ruling: public defs only (private helpers/dunders inherit their class/module trace) — recorded in the W8 conventions doc. Standard trace-wave run, prior = "map if a genuine req exists, else record unmapped".

W6 — statistical acceptance (fresh eyes; the program gate)

1. trace-audit --json → population census.
2. Orchestrator draws a stratified random sample: strata = package, proportional allocation, n=400 (±4.9% at 95% for N≈5k), seeded RNG, sample saved to plans/trace-acceptance-sample.json (reproducible).
3. Fresh verify agents (never used in W1–W5) judge each sampled decorator in full — every id vs the rubric. Packets of ~25.
4. Gate: ≥95% of sampled decorators fully correct. Per-stratum: any package <90% loops back into a targeted W3-style wave, then that stratum re-samples.
5. Report → plans/trace-remediation-report.md (accuracy point estimate + CI, per-package table, before/after defect-class counts).

W7 — downstream rebuild + reconciliation

1. python scripts/generate_rtm.py (+ generate_detailed_rtm.py) → fresh docs/rtm.{json,md}; record the per-spec link-count delta in the report (the RTM will shrink — that is fictional links dying, documented not mysterious).
2. python scripts/sync_ast_to_neo4j.py → Neo4j KG code-intel re-sync (stale since before the seam migration).
3. python scripts/enrich_semantic_graph_specs.py → Memgraph IMPLEMENTS layer refresh (label-isolated per the shared-cluster rules; verify replica health after, per HA runbook).
4. Reconciliation gate: for 3 sampled specs, traces-in-code count == RTM rows == graph IMPLEMENTS edges. Mismatch fails the wave.

W8 — prevention

trace-lint wired into: pre-commit (beside check-deps/standards), CI (validate_ci.py inventory), review pipeline (comprehensive_review.py), and test-review.py rule list. Conventions section in AGENTS.md: format, placement, class-vs-method semantics, sparse-and-accurate, unmapped-is-a-finding, public- defs-only scope. Memory file updated. DoD: introducing a REQ-100-001 or a 30-req decorator fails pre-commit demonstrably.

W9 — spec-gap triage (USER decision point)

Input: all unmapped outputs (760 from W1 + W2/W3/W5 additions). An auditor agent clusters into: (a) spec-worthy → drafted requirement text per spec, delivered as plans/spec-gap-triage.md for USER ratification (never self-ratified); (b) not-trace-worthy (HTML-parser internals, thin delegations, registry plumbing) → named in the W8 conventions; (c) dead-code candidates (e.g. context_injector's 7 retired handlers) → deletion backlog items with evidence. DoD: triage doc delivered; backlog updated; user decision requested ONCE with the full document (not drip-fed).

5. Budget (W1/W2a actuals → estimates)

6. Risks & mitigations

• Verifier blind spots (W1: gates green while 43% of sample wrong) → W6 fresh-eyes statistical gate; no wave self-certifies.
• Over-stripping → verifiers re-judge keeps AND strips both directions; W6 detects systematic over-strip as inaccuracy; every strip of a class-A/B candidate must be argued in the verifier transcript.
• Fixer scope creep / under-reporting (seen in r2-session-a1) → verify step 3 (diff-vs-report reconciliation) is mandatory; misreport ⇒ verdict failed ⇒ orchestrator reads the transcript before accepting.
• Shared-tree collisions → waves serialized; packets partition files; the wave protocol's ENTRY step forbids concurrent src-writers.
• Suite breakage via decorator removal (wiring tests may assert @traces) → wave exit runs the full suite; failures route back into the wave; never fixed by weakening tests (LEVEL0 rule).
• RTM shrinkage shock → W7 quantifies per-spec before/after.
• Cost overrun on W3 → packets are independent; the wave can stop cleanly at any packet boundary and resume (resumeFromRunId replays finished packets from cache).

7. Definition of done (program)

1. check-traces: 0 dangling / 0 malformed. trace-lint: 0 errors.
2. trace-audit: no unjustified fan-out >5, no shotgun >25, sequential fingerprint only where justified, single spec= convention.
3. W6 report: ≥95% sampled accuracy (95% CI), all strata ≥90%.
4. RTM + Neo4j KG + Memgraph rebuilt; reconciliation counts match.
5. Prevention gates live in pre-commit/CI/review; conventions documented.
6. Spec-gap triage delivered to the user; backlog updated.
7. Full suite green; every wave committed with its ledger entry.

8. Status ledger

• 2026-07-09 evening: PROGRAM FROZEN AT PREVENTION-ONLY (user strategy review). The wave strategy did not demonstrate convergence: W1–W5 verify metrics reported 0.6–12.5% wrong, but the independent W6 acceptance sample measured 55.0% fully correct (224/407, gate 95%) — FAIL. Wave-level QA measured the packets each wave touched, not the corpus; ~15M tokens of fixes had no corpus-level measurement until W6. The 95% gate was also set without measuring inter-judge agreement (the instrument's noise floor) — it may be statistically unreachable. Ruling: W6.5 (154-packet exhaustive wave, paused at ~1/3 by user order, resumable via resumeFromRunId=wf_d37360aa-3ba) stays stopped; no further LLM remediation spend without a NAMED CONSUMER of the corrected corpus + a measured inter-judge ceiling (n=100 double-judged, ~300k tokens) clearing the bar. Corpus accuracy caveat: 55% ± 5% — the RTM and KG IMPLEMENTS edges are not ground truth. What shipped instead (deterministic, ~0 tokens): trace-lint gained the spec-mismatch rule (membership-based: every id must be defined by the claimed spec; the deterministic signature found on 55% of all W6-judged-wrong decorators), a one-shot normalizer rewrote 15 spec fields and SPLIT 103 mixed-provenance decorators into stacked per-spec decorators (34 files; ids untouched), trace-lint entered pre-commit, and the two-lane + alias + stacked-decorator conventions landed in AGENTS.md. Post-pass tree: trace-lint 0 errors.
• 2026-07-08 morning: W1 launched after ground-truth staging (4,864-req index).
• 2026-07-08: W1 DONE (see §4). Census v1: 5,061 decorators / 14,395 ids.
• 2026-07-08: W2a DONE (363 stripped, 12 keeps upheld, c04 re-map 50 traced). Tree-wide check-traces: 0 dangling / 0 malformed (was 924 sites).
• 2026-07-08: W0 builder launched. .claude/workflows/trace-wave.js engine created. Plan upgraded to operational runbook. CP1 suite running.
• 2026-07-08: W0 DONE — trace-audit (census cross-validated: exact match on D3/D4 fingerprints) + trace-lint (live tree: 373 findings / 267 errors) + 36 tests, 100% diff coverage. Ground-truth data-quality findings (→ W9 doc, user ratification; W4 entry rulings): (a) 60 REQ ids defined in TWO specs (30× 001-kernel↔008-knowledge-graph, 30× 001-kernel↔010-memory-engine) — lookup ambiguity for every wave; W9 drafts the dedup, waves meanwhile must check BOTH definitions when judging a colliding id. (b) spec="007b" on 196 decorators names specs/035-verification, whose checklist ids are 009-family — "007b" is a legitimate label with no own req family; trace-lint's unknown-spec rule needs an alias map ({"007b": "035-verification"}) at W4, not a purge.

Appendix A — W3 inventory (census v1; refresh with trace-audit at entry)

24 fan-out ids ≥20 uses (top: REQ-001-US-002-001×370, REQ-001-001×317, REQ-001-US-005-001×298, REQ-001-US-004-001×206, REQ-008-001×180, REQ-097-001×92, REQ-001-US-007-001×91, REQ-011-001×81, REQ-002-US-001-001×80, REQ-010-001×52…) totalling 2,148 occurrences; 199 shotgun decorators (≥10 reqs; max 103 on code_parser.py:create_file_node; six ×73 in postman/config.py). Union: 2,299 decorators / 325 files / ~39 packets. Machine-readable: scratchpad wave_sizing.json.

Appendix B — W2b inventory (census v1)

48 outlier cells (<5% of package ids, ≥5 uses, family not legit-homed), ~993 occurrences. Top: kernel×008 (91), kernel×010 (85), kernel×004 (76), knowledge_graph×014 (55), agents×032 (54), agents×019 (51), agents×026 (43), knowledge_graph×106 (38)… Legit-home map (ruling): 020→session, 021→session, 022→gateway; extend at wave entry with evidence. Machine-readable: wave_sizing.json.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks