Orchestrata Ultimate › Sessions › Session 262

Session 262

‹ previous · next › · timeline

Prose pending.

Purpose

Second adversarial review of plans/suite-runtime-and-graph-defects-plan.md + research record; corrections applied

Work log

1. Second adversarial review of the suite-runtime plan: measurement core reproduced exactly, 4 inferences corrected
   • Independent re-derivation (fresh ndjson segmentation, fresh _prepare_batches replay, AST-toolkit reads of every cited site)
   • Reproduced exactly: 2312.0s call CPU, 289.0s floor at 8 lanes, 51 batches, 3104.6s cache mass / 792.6s bucket, 8.4/35.9/53.3/85.3% distribution, 3/24/48/189 histogram, module CPU table, D3 32.7-of-34.1s, 75.75s backoff sum
   • CORRECTED 1: '581 tests in the 2-60s band = 52%' -> that band is 264 tests / 1261.4s / 54.6%; 581 is the >=1.0s cut (1678.1s, 72.6%)
   • CORRECTED 2: D7 classifier is a disjunction (TransientError type OR contention markers); fix needs permanent-marker short-circuit ordering, not just a shared marker list
   • CORRECTED 3: D1 fix inverted needle/haystack (predicate is $chunk_text CONTAINS e.id) -> no property index can serve it; 'tens of ms' target withdrawn, batched-neighbor query carries the win
   • CORRECTED 4: resident-worker regression is 587.4s (wt-RESIDENT-WORKERS artifact), which records 0 reruns / 0 solo retries -> '4 recoveries' and the cross-batch-state cause withdrawn
   • WEAK: 289s floor is a 289-358s range (three consecutive 08-11 runs: 2312/2417/2862s call CPU)
   • New findings appended as plan section 8: production defects hold 1.16% + 3.35% of suite call CPU yet run first; no real-scale regression guard for any of the four (CIG test modules total 0.1s vs a 340.37s real call); D2 invalidation machinery unnecessary (CIG has no mutators); BlastRadius.impact_score vs CodeIntelligenceGraph.impact_score name collision
2. Folded the remaining second-review findings into the plan body (8-1..8-6, R1-R4)
   • S4 execution order reversed: D3 -> suite-side S3 -> D7 -> D1 -> D2+D2b, with the 1.16%/3.35% call-CPU rationale inline
   • S2 D2: lazy per-instance memoization replaces 'compute per build + invalidate on mutation' (CIG has no mutators, 22 symbols checked); BlastRadius.impact_score -> reach_fraction rename shipped with the cache
   • S2 D2b: k=256 latency derived (340.37 x 256/38369 = 2.27s) but top-20 fidelity marked an assumption to measure against the existing exact run; real-CIG benchmark now required; fidelity check moved off fixture graphs onto the real CIG
   • S2 D1: scale-fixture regression guard (>=10k entities, round-trip assertion)
   • S3: target definition corrected to the >=1.0s cut (581 tests, 1678.1s, 72.6%)
   • S4 step 6: report call CPU alongside wall (24% run-to-run variance exceeds several levers); RUL-008 reconciliation paragraph added (gates per item pre-merge; --no-verify is not a review waiver)
   • S7 recommendation updated; S8 converted from an open list to a disposition record
3. Swarm executed all 8 work items; merged into master db0609f9
   • WI-1 retry guard (permanent-marker ordering), WI-2 retrieve batching (16.19s/10,964 round-trips -> ~2s/<=3), WI-3 pagerank lazy cache (0.7s->0.003ms) + approximate centrality (340s->0.66s, 95% top-20 overlap), WI-4 fact-share (module call CPU 16.95s->0.78s), WI-5 agentcheck-once (15.3->8.6s, 20.8->8.2s), WI-6 retry-budget DI (13.0->0.27s, 10.5->1.1s), WI-7 ast-share (34->11s, 32->16s), WI-8 search fixtures (share 48.8%->11.2%). 7 branches merged --no-ff; purity-gate skip violation fixed as scale-up (db0609f9).
4. Pushed master db0609f9; measure-first checklist completed and disarmed
   • Final gate: 32498/0/0/0, 1611 modules, 455s wall (+2% vs 446s), call CPU 2250s (-2.7% vs 2312s) — both inside the 24% noise band; suite wall improvement NOT PROVEN, recorded as such. Production per-call wins verified at scale (retrieve ~8x, centrality ~515x). Push from wt-D7-RETRY-GUARD after ff to master (pre-push hook gated the full suite).
5. Suite objective recorded PROVEN FAILURE; suite-side merges reverted
   • User-directed rollback: reverted WI-4..WI-8 merge commits (c7043047 3097834c f44c64a0 b55077e3 8bafe038). Kept production fixes (429fd08c, 5d3ed30c, db0609f9) + docs. Plan doc marked PROVEN FAILURE with the rule: standalone module benchmarks are non-evidence for suite wall claims.
6. Rollback pushed (6ceef2ef); failure documented
   • Suite-side merges reverted per user directive, post-mortem committed (plans/suite-runtime-failure-postmortem-2026-08-12.md), affected gate 898/898 green pre-push, pushed db0609f9..6ceef2ef. Kept: production defect fixes + docs. All worktrees removed; only primary checkout remains.
7. D2b centrality: shipped default returned wrong rankings — fixed, guarded, recorded
   • Measured on the live 38,380-node repo CIG vs exact top-20: strided samples=256 (the shipped default) agreed on 3/20 with top score 47,150 vs exact 6,574 (7.2x biased); 1024 -> 12/20; 4096 -> 16/20; 8192 -> 17/20; random pivots 256 -> 7/20, 4096 -> 15/20. The >=90% acceptance criterion was never met at any budget
   • Root cause of zero coverage: all four centrality tests ran on graphs smaller than the 256-pivot sample count, so the implementation's own 'if exact or n <= samples' guard sent every one down the exact branch; the sampled path shipped untested
   • Real win of the rewrite, kept: array-based _brandes took EXACT from 340.37s to 88.29s (3.9x)
   • Fix: exact=True is now the default and the exact vector is cached per instance (88.3s first call, ~0ms after — same discipline as the D2 pagerank cache, sound because the graph has no mutators); sampling stays behind exact=False with the measured error table in the docstring
   • Guards added in tests/refactor/codemod/test_analysis_passes.py on a 600-node graph (larger than the sample count, so the branch is reachable): default-is-exact, sampled-diverges-from-exact, exact-vector-cached-per-instance. Red-first proof: under the old default the first guard fails, and on a 600-chain whose max betweenness is the midpoint by construction the sampled default names node 293 instead of 299/300
   • Verified: tests/refactor/codemod/ 19/19 modules, 0 failed, 43s; ruff clean; black clean. Fixed two pre-existing pyright unused-binding hints in the same file
8. Authoritative full-suite run on post-rollback master + D2b fix: 32,499 passed / 0 failed, 426.6s
   • scripts/run_tests.py --force, observed, nothing skipped. Infra complete incl. the required kernel (CXP transport OK on 50053). All safety gates pass. Config identical to the baseline: 1611 modules, 8 workers x 6 batches/worker, 420s timeout
   • Result: 32,499 passed / 0 failed / 0 errors / 0 skipped, 1611/1611 modules, 426.6s wall, head 6ceef2ef with the uncommitted D2b fix in tree (dirty_sig recorded)
   • Call CPU re-derived from the durations sink: 2,202.4s (floor 275.3s at 8 lanes) vs the 2,312.0s baseline — 4.7% lower; wall 426.6s vs 445.7s — 4.3% lower. Both inside the 24% noise band: post-rollback master is no slower than the pre-swarm baseline
   • Distribution held: top10 8.8%, top100 37.0%, top250 54.8%, top1000 86.3%; tests >=1.0s = 532 holding 1,579.5s = 71.7% of call CPU (the >=1s cut remains the only mass worth targeting)
   • No test exceeded 60s; slow_tests and acknowledged_slow both empty
   • One solo-retry recovery: tests/session/test_session_facts.py (masked flake, logged to .flake_recoveries.ndjson with its 33 batch peers)
   • Side effect: reports/run-tests-latest.json now holds a real authoritative full run again, replacing the 47-module subset that had overwritten it
9. Fixed workflow-default-sonnet hook: its advertised spread remedy was rejected, and prose 'model:' was accepted
   • Two defects in .claude/hooks/workflow-default-sonnet.sh, both surfaced by the hook blocking its own documented remedy twice. (1) The block message advertises hoisting {model:'opus'} into a const and spreading it, but the check required a literal 'model:' inside the agent() span, so that form was rejected. (2) The 'model:' check ran on the raw span with no code mask, so a PROMPT containing the words 'model: opus' satisfied the guard - a false-negative admitting a silent inherit. Added object_bindings() (brace-balanced, code-masked) plus names_a_model() with transitive spread resolution and a cycle guard; 'model:' now counts only at code positions.
   • Its test corpus was neutered: the 'spread default' case carried a redundant model:'sonnet' next to the spread, so it passed without ever exercising spread resolution. Replaced with a real spread case plus spread-of-spread and declared-after-use, and added 4 BLOCK cases (spread naming no tier, unbound spread, prose-only model, mutually recursive spreads).
   • 9/9 direct hook probes correct. pytest module run deferred: machine at load 5.9 under an in-flight builder.
10. Derived-artifact hygiene: deleted 142MB of leaked artifacts, corrected the fixture-bucket contradiction in the runtime plan

• Research verification exposed live leaks. Deleted (all untracked, all verified partial/duplicate first): plans/ast-ultimate.json.warmcache.1088354.tmp (61.2MB) and .1554581.tmp (57.0MB) - truncated partial writes from PIDs confirmed dead, 9 days old, while the real 159MB warmcache was intact; reports/session-logs/dur_tail.ndjson (exactly 30,000,000 bytes, a head -c slice of the durations sink left by a throwaway research script). plans 462M->349M, reports 366M->337M.
• Corrected plans/suite-runtime-2-plan.md sect 1b: it assigned the whole ~837 lane-s residual to fixture setup/teardown, contradicting sect 3c's independently measured 516.7 lane-s for the same bucket in the same document. Split into 516.7s MEASURED plus ~321s UNATTRIBUTED, and flagged two further open defects: the 516.7 lane-s -> 279 core-s conversion is underived, and the 336 core-s AST-daemon claim EXCEEDS the 279 core-s figure for the entire fixture bucket, so nothing may be spent against it until a paired n=3 core-second run settles it.

11. Fixed a suite-blocking stdlib shadow: scripts/secrets.py masked stdlib secrets, breaking Redis + Gateway preflight

• A new scripts/secrets.py (from the secrets-encryption packet I dispatched — my brief named the file) shadowed the stdlib secrets module, because scripts/ is sys.path[0] whenever a script there runs directly. redis-py's ConnectionPool calls secrets.token_hex at construction, so infra preflight reported 'Redis: port open but query failed: module secrets has no attribute token_hex' AND 'Gateway: health check failed' - the suite could not start. Renamed to scripts/secrets_cli.py (+ tests/scripts/test_secrets_cli.py), updated references in pyproject.toml and scripts/provision_worktree_env.py. Preflight verified green: all four services ready.
• An agent had independently hardened tests/plugins/services.py to load start_test_services.py by file path via importlib instead of sys.path.insert - kept, since that fix holds regardless of filename.

12. Volk-dispatch readiness measured: NOT ready, 3 blockers; fixed blocker 1 (src->scripts import) and re-fixed the stdlib shadow

• 7-agent assessment + 3 live probes: 0 volk invocations, 0 LLM calls, 0 artifacts. Corrected two of my own claims: (a) 'the dispatch loop is complete' is FALSE - orchestrata respond/approve/reject are functional no-ops, sending subjects human-response/human-approval/human-rejection to a thread_id target while CxpHandler.handle_send only forwards subject in {intent,response} AND target in concierge aliases; they broadcast a pubsub packet and return status=sent, never reaching PipelineRunner.respond(). (b) 'packaging is a manifest not a build-out' is FALSE - the seam died at import and the pipeline deadlocks before stage 1.
• BLOCKER 1 FIXED: src/orchestrata_ultimate/verification/spec_completeness.py:21 imported parse_requirements from the repo-root scripts/ shim instead of its canonical home; repointed to orchestrata_ultimate.openspec.requirements_parser. Verified: session pipeline run --help now imports clean, and who-imports scripts.parse_spec_requirements = 0 sites.
• BLOCKER 2 (dispatched): knowledge_graph/synchronizer.py ensure_fresh rglobs the whole repo per stage, fired unconditionally by kernel/pipeline_kg.py _pre_task_sync whose except Exception makes a hang look like success. BLOCKER 3 (deferred, off critical path): ToolExecutor._run_shell timeout=30 + stdout truncated to 2000 chars, so a volk cannot run this project's own test command.
• FALSE CONFIDENCE: tests/orchestrator/test_pipeline_runner.py passes 19/19 while no test calls PipelineRunner.run(), every stage handler is a local stub, and the volk is a fabricated alias TEST:Flaky. Also SPEC-020 (status Implemented) has 11 of 21 hooks undeployed to the target its own Overview names.
• REFRAME worth keeping: under the active claude_cli profile every volk is provider=claude_code -> ClaudeCodeDriver -> 'claude --print --dangerously-skip-permissions'. So 'volks replace Claude subagents' means orchestrata SCHEDULES/GOVERNS headless Claude Code processes; orchestrata contributes orchestration, not building.

13. external-tools gate landed green; adopted the red prolog-gate the agent declined (RUL-024)

• Agent finished green: 74 tests (57->74), 100% coverage, gate REGISTERED and RUN in scripts/gates.py (--only external-tools passes; --all shows it alongside check-deps/ruff/mypy). Declared incus>=6.0.0, swipl>=9.0.0, node>=20.0.0 (live 7.3/10.0.2/26.4.0). Console-script names and vendored-dir exclusions DERIVED live from pyproject ([project.scripts], [tool.ruff] extend-exclude), not hardcoded - proven with synthetic-pyproject tests. All 23 flagged invocations accounted for; exit 0.
• Agent reported 'gate.py run FAIL - 2 violations tests/scripts/conftest.py:309/316, pre-existing, out of scope'. RUL-024: pre-existing is a diagnosis, not a disposition. Took it.
• Diagnosis: session_ast_daemon fixture (added 2026-08-11, commit 4b5db16a) sets AST_DAEMON_SOCKET process-wide so CLI subprocesses inherit it. Sized the env= seam alternative via AST toolkit: 422 subprocess spawn sites across 115 files in tests/scripts. Disproportionate, and it defeats the fixture design (session-scoped autouse, transparent, correctness-independent - CLI local fallback is authoritative).
• core/gate_waivers.py names this exact case verbatim as the merits exception: 'a fixture that saves and restores an env var around a subprocess it must configure'. Added 2 reasoned '# gate-ok:' markers (WAIVER_REACH=1, marker directly above each mutation). VERIFIED BY AST not by eye: both env-mutation assign sites (now 314, 322) return waived=True with non-empty reasons.
• Whole-tree gate verdict blocked: freshness-as-inference returned INVALID rc=2 because the KG-sync agent is concurrently editing knowledge_graph/config.py. Ran ast-sync once (rc=0) to clear my own staleness. Did NOT touch config.py - another agent's open file.

14. Ran the volk-dispatch measurement; probe exposed a false-green in my own blocker-1 fix

• Measurement design stated BEFORE the first run (post-mortem binding rule): acceptance = exits 0 with non-empty review artifact; cost = wall + child CPU + LLM tokens; risk = failure mode and rate; noise = report median and full spread of n=3, claim no difference smaller than it. Harness pins ORCHESTRATA_ENV_PROFILE=test explicitly because a bare .env is the PROD base layer here - an unpinned dispatch silently hits production.
• Preconditions verified: all 4 services ready incl. the REQUIRED kernel (Redis PONG, Graph RETURN 1 engine=memgraph database=orchkg, Gateway ping->pong, Kernel CXP transport OK). Box load 1.50; dispatch wall is LLM-latency-dominated so local contention does not invalidate it.
• PROBE RESULT (n=1): rc=1, wall=0.9s, user=1.82s, sys=0.08s, maxrss=143724KB. It does not reach stage 1 - it dies at import. ModuleNotFoundError: No module named 'scripts', from verification/spec_completeness.py:22 'from scripts.scan_implementation import match_requirements_to_evidence'.
• MY ERROR, stated plainly: I reported blocker 1 as cleared earlier this session. It was not. spec_completeness.py had TWO scripts.* imports; I repointed line 21 (parse_requirements) and missed line 22 (match_requirements_to_evidence). I verified with 'session pipeline run --help', which returns rc=0 because _get_runner imports lazily INSIDE the function body - so --help never executes the failing import. Textbook false-green: the verification command did not reach the code under test. The probe is what caught it.
• Fix dispatched to a builder (inline, no worktree - worktree isolation branches off origin/master and would drop uncommitted working-tree fixes). Design is not new: mirror commit aa026165, which moved parse_requirements into orchestrata_ultimate.openspec.requirements_parser and left scripts/parse_spec_requirements.py as a main()-only shim. RUL-022 hook blocked the first dispatch for having no ratified design; re-dispatched with an explicit DESIGN_WAIVER since this is a mechanical relocation of existing functions.

15. KG-sync bounding landed; fixed the deadline overshoot it flagged (my brief caused it)

• Agent result: the hang was real and is fixed. Root cause was NOT enumeration (31K files hash in ~7s) but ensure_fresh doing a REAL per-file network graph write serially over all ~31K eligible files on a cold cache, with zero output until done. Fix: honour max_files, add a wall-clock deadline that degrades visibly, add ORCHESTRATA_KG_PRESTAGE_SYNC kill-switch read at call time, and stop swallowing the exception (now logs with exc_info before the print).
• FIRST FULL PIPELINE COMPLETION ever recorded. sync ON: 1m13.9s, EXIT=0, U:Qa 26170ms + U:Cp 46704ms, 1 artifact. sync OFF (ORCHESTRATA_KG_PRESTAGE_SYNC=0): 0m56.3s, EXIT=0, U:Qa 5954ms + U:Cp 49068ms, 1 artifact. Both obtained with a PYTHONPATH workaround for blocker 1, NOT the real fix - so they are indicative, not the measurement.
• The agent independently hit the SAME blocker-1 defect I had wrongly reported cleared, confirming it blocks ALL CLI use of session pipeline run, not just review.
• MY DEFECT, fixed: the agent flagged 24.2s actual vs 10.0s configured deadline and correctly called it a tuning decision rather than fixing it unasked. The cause was MY brief, which specified checking the deadline once per batch_size. With batch_size=50 and ~0.48s per real network write, worst case overshoots by a full batch. time.monotonic() costs tens of ns against a ~480ms loop body, so the batching saved nothing. Moved the check to EVERY file; overshoot is now bounded by one file, the floor without cancelling an in-flight write. batch_size had exactly 1 reference in the package (that check), so it is gone from the loop. Verified: ruff clean, pyright 0 errors, 44 tests pass (test_config 7 + test_knowledge_sync 37) in 2s, including test_ensure_fresh_stops_early_on_wall_clock_deadline.
• Agent owned a deviation unprompted: ran 'uv run ast-sync --help', which ignores --help and executes the full serialize+deep-compare pipeline. Worth knowing - that CLI has no safe --help probe.

16. Designed + dispatched the fix for name-matched CALLS/INHERITS edges (RUL-022 via a real contract, not a waiver)

• Traced the defect end to end. Producers: CodeParser.resolve_calls (code_parser.py:373) emits caller_id/callee_id/confidence with scope-attenuated confidence (same file > same package > global); resolve_inheritance (code_parser.py:454) emits child_id/parent_id. sync_project_with_resolution (synchronizer.py:~507) computes BOTH, then hands them to _persist_resolution_edges which ignores them - so the name index over every function and class is built and thrown away every sync.
• KEY DESIGN FINDING - Tier 2 must NOT simply be deleted. The Python resolver's name index is built only from all_analyses, i.e. the CURRENT BATCH. A call into a file already in the graph but absent from this batch has no candidate and yields no edge. Tier-2 Cypher is the only thing that resolves those cross-batch edges. Deleting it silently loses them. So the fix is PRECEDENCE + PROVENANCE, not removal: scoped resolution wins and is persisted by node id; name-matching may only touch pairs the scoped resolver could not reach at all; every edge records resolution='scoped' or 'name-match' and the name-matched count is logged.
• To make the narrowing possible, call_name must travel on the CALLS rows and base_name on the INHERITS rows - a small addition in code_parser.py.
• TWO MORE DEFECTS found while designing, folded into the contract: (1) resolve_calls' docstring promises a call_count key that _call_edges_for never emits - contract drift; (2) the returned edge count sums len(method_edges) + Cypher count(r) + len(import_rows), but MERGE counts matched-or-created, so pre-existing edges inflate it and any Tier1/Tier2 overlap double-counts.
• Wrote plans/contracts/kg-edge-resolution-provenance.contract.json (evidence, design, 6 acceptance tests, explicit out-of-scope) and dispatched a builder naming it. Deliberately did NOT take the DESIGN_WAIVER route - this changes behaviour and edge semantics, so a waiver would have been waiver abuse. Builder is required to produce a RED test first: two files each defining run(), caller resolved to the same-file one, assert exactly ONE outgoing CALLS edge. If that test does not fail against current code, it must stop and report - that would falsify my diagnosis.

17. Class map now injected into every session: the 'what do I extend' tier

• ROOT CAUSE of today's repeated design errors, measured: the @import tier carried package purposes + a 1614-line test inventory but ZERO class-level facts. Of six abstractions needed to design one change correctly, FIVE existed only in the non-injected docs/CODEBASE_MAP_FULL.md. Consequence: I designed a third parallel 'walk code and find things' implementation instead of extending CodebaseAnalyzer and the factory/detectors rule engine that already existed.
• THE SPLIT WAS CUT AT THE WRONG SEAM - by FILE (header vs full) instead of by SYMBOL KIND. Measured: classes = 1731 symbols / 35K tokens and carry nearly all the extension signal; methods = 6537 symbols and carry almost none. The 2026-07-06 two-tier split protected subagents from 404K, and threw away the 35K that mattered along with the 370K that did not.
• IMPLEMENTED in scripts/gen_codebase_map.py: _class_map() COMPOSES the existing toolkit (list_symbols, daemon-accelerated 2.3s) rather than writing a fourth AST walk - the lesson of the day applied to its own fix. Loads serialize_codebase_ast BY FILE PATH, never by import name, to avoid the scripts/ stdlib-shadowing class that took Redis down. Scope widened from src/orchestrata_ultimate to ALL of src/, because CodebaseAnalyzer lives in src/orchestrata_ast and was the single most important omission.
• MEASURED RESULT: 1729 classes indexed with bases, 118 marked as extension points via star-N (BaseTool(ABC) star-25 = how to add a tool; BaseAgent(BaseModel) star-11 = how to add a volk). @import tier 1842 -> 3647 lines, ~50K -> ~87K tokens (~3% more of a 1M window). Byte-identical across regenerations. Hook rc=0 end to end.
• TESTS: tests/scripts/test_gen_codebase_map.py, 17 tests, NONE existed before for a generator that feeds every session's context. The regression guard asserts all six previously-missing symbols are present. The resilience test CAUGHT A REAL BUG IN MY OWN CODE: _load_ast_toolkit claimed to return None on failure, but spec_from_file_location does not raise on a missing file - exec_module does, with FileNotFoundError - so the session-start hook would have CRASHED rather than degraded. Fixed with a logged except per AGENTS.md Hard Rule 3.
• Full gate battery: ALL 14 GATES PASS. ruff/mypy/pyright clean. Hook header comment corrected from the now-false ~25K figure.

18. Implementation scanner rebuilt on the AST + the measured registry pattern; two of my own claims proven false

• Landed: implementation_scanner.py rewritten (497 lines) mirroring factory/detectors - EvidenceFn Callable alias (no ABC), ALL_DETECTORS dict built by merged registry builders, _symbol_detector closure factory mirroring _lexicon_rule, one _RegistrationKeyVisitor(ast.NodeVisitor) mirroring NestingVisitor, scan_project(codebase, *, detectors=ALL_DETECTORS) with the registry INJECTED. 36 tests, 100% coverage, all 14 gates pass, E2E still Status: resolved rc=0.
• MY CLAIM 1 WAS FALSE: the contract asserted this would be the FIRST orchestrata_ultimate -> orchestrata_ast edge, based on 'who-imports orchestrata_ast' returning 0 sites. Wrong query. 'who-imports orchestrata_ast.analyzer' returns 3 - gates/fact_extractor.py and tools/ast_pipeline.py already imported it. LESSON: who-imports matches the EXACT module string, never the package prefix; always query the submodule.
• MY CLAIM 2 WAS FALSE: the contract called evaluate_spec_completeness 'the QA volk's production entrypoint'. It has ZERO production call sites - only verification/init.py and tests.
• REAL BUG FOUND AND FIXED by the builder: _SPECS_DIR used SIX os.path.dirname hops instead of four, resolving to /home/mohamed/specs which never exists. So _find_spec_checklist's isdir check always failed and evaluate_spec_completeness silently returned {} for EVERY real call since the module was written. Combined with zero call sites: a spec-coverage feature that never ran, called by nothing. Natural production seam identified but deliberately NOT wired (QaVolk._handle_review via _run_review_gate) - whether low spec_coverage should block or advise is a product decision.

19. Arm A measured (n=3) — and the volk pipeline returns no retrievable output

• ARM A, idle box (load 0.08), profile pinned to test, all rc=0 / Status: resolved: run1 64.6s wall / 54.35s user CPU; run2 58.5s / 48.57s; run3 63.5s / 53.93s. Median 63.5s, spread 6.1s. U:Qa near-deterministic at 13.59-13.66s; essentially all variance is U:Cp (43.87-49.84s).
• CORRECTION to my own earlier reasoning: I argued local load would not invalidate the measurement because dispatch is LLM-latency-dominated. Measured ~54s user CPU against a ~64s wall = ~85% local CPU. That claim was wrong.
• MY DEADLINE FIX VERIFIED IN PRODUCTION CONDITIONS: 'kg sync budget exceeded after 10.1s' against a configured 10.0s budget - the 24.2s overshoot is closed to 0.1s by moving the check from once-per-batch to every file.
• NO TOKEN ACCOUNTING for the volk arm: Redis has no cost:* or budget keys. CostTracker is constructed at provider_manager.py:101 but record_cost no-ops unless set_redis was called, which nothing on the dispatch path does. So the arm cannot be compared on tokens, only on wall/CPU.
• ARTIFACT FINDING (filed as a high work item): the produced artifact is retrievable by no path - CLI prints a count only, ArtifactStore is in-memory with no output_dir, no artifacts.jsonl on disk, and the KG Artifact node has no content property with file_path='' for every run.

20. Measurement complete (both arms) + fixed 5 real defects the comparison arm found in flake_report.py

• ARM A volk pipeline, n=3 idle box: 64.6/58.5/63.5s wall, median 63.5s, ~54s user CPU (~85% local, NOT LLM-latency-bound as I claimed). All rc=0, Status: resolved. Output DISCARDED - CLI prints a count only, ArtifactStore in-memory with no output_dir, no artifacts.jsonl on disk, KG Artifact node has NO content property and file_path='' for every run. Tokens not instrumented (record_cost no-ops, set_redis never called).
• ARM B Claude subagent, n=3 sequential, identical task string: 456.5s total = ~152s/run, 420161 tokens = ~140K/run. Produced full reviews with live reproduction commands.
• VERDICT: volk is 2.4x faster in wall time and returns NOTHING; the subagent is slower and token-expensive and returns the work product. Quality cannot be compared because one side has no retrievable output. The honest framing is completeness, not speed.
• PAYOFF: all three Arm-B runs independently found REAL defects in scripts/flake_report.py. I reproduced all of them myself before acting: parse_line's bare isinstance(ts, (int,float)) admits (a) NaN, (b) Infinity/-Infinity - json.loads accepts these non-standard tokens by default, (c) unbounded ints (400-digit -> OverflowError inside float(ts) itself), (d) finite-but-out-of-range floats like 1e20, and (e) bool, since bool subclasses int, silently reported as 1970-01-01. Cases a-d crashed the whole report from format_date's datetime.fromtimestamp with an uncaught traceback and rc=1, breaking process()'s documented 'Always returns 0'.
• FIXED in scripts/flake_report.py: reject bool explicitly, wrap float(ts) for OverflowError, and validate by making the SAME datetime.fromtimestamp call the report will make rather than guessing a platform range. Verified: all 5 inputs now rc=0, counted as skipped, report still prints. Added tests/scripts/test_flake_report.py::TestNumericTimestampsJsonAcceptsButDatetimeCannot - 6 regression tests incl. a process-level test asserting the return-0 contract.

21. Read specs/docs/wiki end-to-end; corrected 5 false wiki claims

• Read AGENTS.md (1064L), SPEC_INVENTORY, docs/README, architecture-overview, volks-and-communication, context-injection, SPEC-020/101 heads, and 12 wiki pages incl. the 2026-07-28 VolkDriver integration design
• Found+fixed: 5 claude/ wiki pages declared PEP 758 'except A, B:' a SyntaxError; one told readers the file 'cannot be imported as-is'. All 8 cited files ast.parse clean on CPython 3.14.6; pyproject requires >=3.14,<3.15 and black target-version py314 strips the parens back
• Wiki was self-contradictory: runbook-environment-replication already called them PEP 758 correctly

22. VolkDriver Phase 0: premise check refuted the contract's gateway seam; corrected to rev 2

• 4 opus refuters, 938K tokens, 473s. R1 supersession CONFIRMED (orchestrata.event = 0 symbols; design 8 days older than shipped bf8cf695). R2 fixed-dict trap CONFIRMED + refined. R3 no-watch CONFIRMED. R4 gateway seam REFUTED — build halted before any code
• R4: gateway.subscribe is NOT registered (73 methods; gateway.* = ping/info/clients/replay only). Real key is the bare 'subscribe' under '# Legacy short names'. Unknown method -> RPCError -32601. Re-verified independently off _build_method_registry
• Root cause: the literal 'gateway.subscribe' appears exactly ONCE in the repo — a stale docstring in gateway/models.py::MessageType that calls itself 'a published contract'. Corrected in place
• Also corrected: cli_client._rpc_call is single-shot and structurally cannot stream (contract had said reuse it); and the gateway/server.py decode edit P0-A demanded is UNNECESSARY (_decode_pipeline_event is a generic pass-through)

23. Fixed forbidden-doubles-stop.sh: it was committing the artifact-vs-description defect its own header cites

• Fired on 'was any green faked' — the name of a review lens that DETECTS weakened tests. Same class as the 2026-08-06 pair, one layer up: those fired on the name of the ENFORCING CODE (MOCK_IMPORT, _check_no_mocks), this on the name of the DEFECT ('fake the test green' is AGENTS.md's own phrase)
• Fix = two asymmetric signals. ARTIFACT: a double written into a .py file this turn blocks unconditionally, naming the file; scans every assistant message since the last user message (a turn writes early, summarizes last); scoped to .py so contracts/specs/wiki may quote doubles freely. DESCRIPTION: prose keeps the closed exemptions plus defect-name, requiring the OUTCOME word (green/pass) so 'fake the gateway' still fires
• Two subtleties: exemption is SPAN-scoped not sentence-scoped (unlike enforcement tokens) so a real double in the same sentence still fires; and containment had to become OVERLAP because _DOUBLE's leading [\w.-]* swallows hyphenated prefixes — on 'no-faked-green' the hit starts one char before the defect span
• 33 passed 0 failed 4.82s, observed, real subprocess. All 6 original real-double shapes still fire; laundering case still fires

Artifacts

• plans/suite-runtime-and-graph-defects-plan.md
• plans/suite-runtime-research-2026-08-12.md
• src/orchestrata_ultimate/refactor/codemod/_cig.py
• plans/suite-runtime-failure-postmortem-2026-08-12.md
• plans/contracts/volkdriver-phase0.contract.json - VolkDriver Phase 0 contract: supersession ruling, 2 deliverables, 9 constraints (C1 the fixed-dict wire trap), 7 acceptance tests, 6 out-of-scope items
• plans/contracts/volkdriver-phase0.mmd - Phase 0 contract diagram: shipped vs deliverable vs trap vs later-phase

Decisions

• Suite-side program runs before the four production defects; D1/D2/D2b/D7 each land with a real-scale regression guard: Measured: tests/semantic_graph = 26.8s (1.16%) and tests/refactor/codemod = 77.4s (3.35%) of the 2312.0s baseline call CPU, and the two CIG-specific test modules total 0.1s against a 340.37s real-CIG centrality call. The production defects move the 446s wall by approximately nothing, and the suite cannot catch a regression of any of their fixes at real scale.
• VolkDriver Phase 0: keep the shipped PipelineEvent taxonomy, do not build the design's orchestrata.event/1 envelope: The 2026-07-28 integration design's Phase 0 is 'orchestrata.event/1 schema + Redis publisher in kernel'. Measured 2026-08-12: the taxonomy AND publisher shipped 2026-08-05 (bf8cf695, acceptance 164/164) as PipelineEventType(13)/PipelineEvent/to_payload==asdict + PipelineEventBus.emit->Redis. Building the design's separate envelope would be a THIRD producer shape, re-opening the exact two-producer fork that commit closed. Phase 0's INTENT (one versioned stream every surface renders) is already met; what is genuinely ABSENT is (a) the watch consumer -- orchestrata --help has 18 commands, none is watch -- and (b) driver provenance on the event. Phase 0 therefore delivers those two. Plan fidelity honored in substance (RUL-015); parallel-implementation ruling honored (user, 2026-08-12).
• PipelineEventBus.emit() stamps driver='inprocess' by MUTATING the caller's event object — keep as-is: Surfaced to Mohamed 2026-08-12 with a live probe (driver reads '' before emit, 'inprocess' after) and ruled: keep. Rationale: emit() is the single choke point, so stamping there makes attribution unmissable; the alternative (stamping inside the six build_*_event factories) leaves emit() pure but misses any event not built through a factory. Accepted trade-off: mutating an argument is an observable side effect, and one event object emitted to two buses gets first-writer-wins. Documented in the emit() docstring; a caller-supplied driver (Phase 2 headless-harness producers) is never overridden.
• PipelineEvent.driver default becomes 'inprocess'; delete the emit() stamp: Three verifier lenses found PipelineKernelClient._emit_event is a SECOND producer bypassing PipelineEventBus.emit(), broadcasting driver='' forever (measured 2/2 unattributed on the real wire). Stamping in to_payload() was REJECTED: test_to_payload_equals_asdict pins to_payload()==asdict(self), the invariant bf8cf695 established to close the original two-producer fork. Making 'inprocess' the field default covers BOTH producers with zero stamping code, covers any future third producer, preserves to_payload()==asdict, and stops emit() mutating caller-owned state -- which un-forks _GOLDEN_EVENT_JSON/_GOLDEN_EVENT_JSON_VIA_BUS and restores the cross-path invariant that fork silently retired. Verified safe: the only conditional branch on driver in the whole tree is the stamp itself (pipeline_events.py:252); nothing constructs PipelineEvent from wire data, so no historical backlog entry can be mislabelled. Supersedes the earlier 'keep the mutation as-is' ruling by REMOVING the mutation rather than defending it.
• SUPERSEDES the earlier driver-default choice: per-producer driver identity, resolved non-mutatingly at serialization: The 'driver: str = inprocess default' decision I logged earlier was REFUTED by an adversarial pass before any code was written. Three blockers: (1) it repeals contract clause :61 and acceptance A1 ('default "" when unset') with no revision-3 amendment; (2) driver=='' on the wire is the ONLY observable signature of an unattributed producer -- the exact signal that FOUND this defect -- so making 'inprocess' the default means a future third producer is MISLABELED for free and undetectably: the bug class is made invisible, not deleted; (3) it ships a red suite (tests/kernel/test_pipeline_events.py L46/L54/L55 pin driver=='' and were never in the step list). My stated measured fact 'nothing treats "" as a sentinel' was FALSE -- cli_watch.py:78-80 branches on falsy driver to decide rendering; I had scoped refs to one file and grepped three that excluded it. CORRECTED DESIGN: keep driver='' meaning unattributed; give PipelineEventBus and PipelineKernelClient each a driver='inprocess' constructor identity; delete the emit() mutation and resolve on the OUTPUT dict (event_dict['driver'] = event_dict['driver'] or self._driver); apply the same one-line resolve at pipeline_kernel.py:510 (the site that measured 2/2 UNATTRIBUTED); _forward_to_ui carries the resolved value; ADD a guard that every broadcast/stream payload has a non-empty driver so a third producer FAILS loudly; ADD the cross-producer test that does not exist today (nothing drives PipelineKernelClient._emit_event). to_payload()==asdict preserved.

Achievements

• (auto-closed by new session)

Handoff

• Key decisions: Suite runtime thread CLOSED at db0609f9 (pushed). Wall objective not proven within noise; per-module and production per-call wins verified. Next suite-time lever would need the unmeasured gaps closed: fixture-vs-collection split instrumented run, per-lane occupancy logging in test_executor.
• Blockers: None
• In-progress: None
• Important files: None
• Commands: None
• Warnings: 7 swarm worktrees + wt-RESIDENT-WORKERS + wt-AST-DRIFT still exist, all branches merged into master db0609f9 — gc with .venv/bin/python scripts/worktrees.py gc --apply (wt-AST-DRIFT has 1 dirty file, needs --force or hand cleanup). knowledge-wiki repo still has unpushed local commit cee073f (expired credential helper).

Related

• Session log
• Session timeline
• Orchestrata Ultimate — project index
• Workflow