Orchestrata Ultimate › Plans › bdd-runtime-tracing-plan

BDD runtime-tracing plan — execution-evidence tier for the RTM

Source: plans/bdd-runtime-tracing-plan.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-21 · 100 lines.

Prose pending.

Plan text

BDD runtime-tracing plan — execution-evidence tier for the RTM

Session 244, 2026-07-21. Execution plan derived from reports/research-bdd-runtime-tracing-rtm.md (verdict: PARTIAL YES — runtime attribution fixes the PRECISION half of the frozen-at-55% trace problem; recall moves only as test-side citations grow). Read the report for the measured evidence; this file is the actionable plan of record. Related: plans/trace-remediation-plan.md (the freeze ruling this plan operates under), scripts/sync_traces_to_neo4j.py (the TRACES_TO writer, committed 4c5d5615 and deployed live 2026-07-21 — 6,298 edges).

Goal

Make the RTM truthful, graded, and monotonically improvable by adding a deterministic EXECUTED evidence tier: per-test coverage attribution joins (test → requirement citation) × (test → executed def) into (def → requirement) edges with provenance — zero LLM tokens, no judge noise floor. Plus the wrong-trace detector: a static @traces id whose citing tests all pass but never execute the decorated def is CONTRADICTED by the only behavioral evidence there is.

Non-negotiable constraints (from the report's measurements)

• COVERAGE_CORE=ctrace is load-bearing. The 3.14t default core (sysmon) silently drops per-context re-recording of repeated lines — the pilot attributed only 2/11 tests under it. Any attribution consumer MUST refuse sysmon-produced data (report §3.2).
• Separate cadence, never the default lane. +70% wall worst case; plan 17–22 min for the full suite (report §3.3).
• Attribution-exempt paths: subprocess/kernel-served execution cannot be per-test attributed (report §3.4); their traces stay on static/citation evidence.
• CONTRADICTED flags, never auto-strips (pending user decision D3).
• BDD form: NO Gherkin/pytest-bdd (installed, zero usage, adds nothing). The citation form is @traces decorators on test defs + docstring citations — both already in the wild (392 decorators / 42 files / 96 ids).

Phases

Phase 0 — harvester completeness (0.5 day, unblocked)

• Extend harvest_test_citations (scripts/generate_rtm_evidence.py) to read @traces decorators on test defs — +82 unique ids of authored evidence currently invisible.
• Retire or re-point the self-contradicting legacy docs/rtm.md generator (reports "Coverage 100.0%" alongside 2,876 orphans and 0 verified).
• Gate: harvested-id count ≥ current 146 + 82; evidence RTM regenerates clean.

Phase 1 — scripts/attribute_traces_runtime.py (1–2 days, unblocked)

• Input: combined .coverage DB (hard-verify ctrace core; refuse sysmon), specs index, Phase-0 citation harvest.
• Join + scoring per report §5.2: score = support / (log2(1+breadth) · log2(1+spread)); hard filters — public defs only, infra lane excluded, ubiquity cap 0.5, |run phase only, top-N=3 proposals per requirement.
• Output: docs/trace-attribution.json with four tiers — CONFIRMED / RUNTIME-CANDIDATE / STATIC-UNCONFIRMED / STATIC-UNTESTABLE — and per-edge provenance {tests, support, breadth, spread, score}.
• Skeleton exists: the pilot join (~150 lines) is specified in report §5.
• Gate: pilot slice reproduces the session-244 result (25/33 confirmed, REQ-106-063 nominated on relevance.py:rescore_by_tier).

Phase 2 — runner integration (0.5–1 day)

• run_tests.py --attribute: inject --cov --cov-context=test, pin COVERAGE_CORE=ctrace in the batch env, run Phase 1 post-suite. test_executor.py already plumbs per-batch COVERAGE_FILE + combine; contexts survive combine.
• Cadence: on-demand + nightly/weekly (user decision D4); never default.
• Gate: full attribution run completes; kernel REQUIRED rule untouched.

Phase 3 — consumers (1 day)

• (a) generate_rtm_evidence.py: new grade EXECUTED between TESTED and CITED, read from docs/trace-attribution.json.
• (b) sync_traces_to_neo4j.py (already live): add provenance property on TRACES_TO (static | runtime | both); accept the attribution file as a second harvest source beside collect_traces. Same writer, same MERGE, same canonical RequirementItem targets — no parallel edge type.
• (c) add_traces_from_evidence.py --runtime <file>: RUNTIME-CANDIDATE decorator writes through the existing AST-precise writer, post review gate.
• Gate: SPEC-106 selector's trace_degree unaffected for static edges; provenance visible in the graph; trace-lint green.

Phase 4 — recall program (open-ended; BLOCKED on user decision D2)

• Citation-expansion waves over the ~745 test modules already carrying structured compliance headers, spec by spec. Each citation is new evidence, not new judgment — but it is net-new annotation work adjacent to the freeze ruling, so it ships only with explicit sign-off: scope, authorship (human vs LLM-assisted), budget.
• Honest ceiling: adjudicable occurrences grow from 6.6% toward a plausible 40–60%; "95% corpus correctness" stays unfundable and unpromised.

Total Phases 0–3: ~3–4.5 builder days, ~0 LLM tokens at runtime.

User decisions (report §9 — all OPEN)

• D1 Adopt graded RTM + EXECUTED tier as THE RTM (honest shrinkage of the headline number; legacy 100.0% fiction retired).
• D2 Fund Phase 4 citation expansion (scope/authorship/budget).
• D3 CONTRADICTED handling: flag-only (recommended) vs auto-strip vs human-strip queue.
• D4 Cadence + whether STATIC-UNCONFIRMED becomes a CI ratchet.
• D5 Bless @traces-on-tests as the canonical citation form in AGENTS.md.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks