Knowledge Wiki Orchestrata Ultimate

Orchestrata Ultimate › Sessions › Session 264

Session 264

Field Value
Session 264
Date 2026-08-12 20:23
Model Claude Opus 4.6
Length 23h 43m
Tag session-264-2026-08-12-2023
Source archive (context.history.md)

‹ previous · next › · timeline

Prose pending.

Purpose

Document driver-provenance and parallel-safety (Gate 0.4) contracts as specification

Objectives

Work log

  1. Documented driver-provenance contract (SPEC-001 addendum, REQ-006-034..037)
    • PipelineEvent.driver: default empty is legal/unattributed, never an error
    • resolve-not-mutate: payload['driver'] = payload['driver'] or self._driver on the OUTPUT dict only, event.driver never mutated
    • exactly two production producers (PipelineEventBus.emit, PipelineKernelClient._emit_event) enforced by an AST structural guard over every to_payload() call site under src/
    • cli_watch.py render_event_line omits driver/volk when empty rather than rendering a placeholder
    • grounded every citation via serialize_codebase_ast.py source/refs + a live 91/91 pytest run across the 4 referencing test modules; caught and corrected a residue-mode line-arithmetic error against my own first draft
  2. Created SPEC-097 addendum for Gate 0.4 parallel-safety (REQ-097-101..105)
    • Why the gate exists: tests/conftest.py redis_client fixture has no db=, lands on REDIS_DB=0 (env/test.env), the same DB the live test-fleet kernel/gateway use; teardown only deletes test_id:* keys
    • What Gate 0.4 flags: AST Call whose verb is in REDIS_WRITE_VERBS (67 verbs) and whose receiver resolves Redis-tainted and whose key is not test_id-derived, per gate_04_redis_keys
    • The two dispositions: (A) rewrite the key test_id-derived, (B) a reasoned # gate-ok: waiver read via core.gate_waivers (bare marker waives nothing)
    • Per-call-site requirement: WAIVER_REACH=1 covers only the marker's own line + one below, never a function/class/file — the exact shape of the defect the old presence-check rule had (test_id anywhere in the enclosing function)
    • Verified rc=0/PASS via scripts/gates.py --only parallel-safety --all, and 76/76 on the gate's own test suite this session
  3. Fixed the concurrent-write defect that shredded plans/ast-ultimate.json and took the review battery down at gate 1
    • Battery run 1 died at ast-refresh: the serializer WROTE the 73MB snapshot then could not json.loads the file it had just written — 'Extra data' at char 73,507,696.
    • Measured shape: 75,563,054 chars total; valid JSON document ends at 73,507,683; followed by 2,055,371 bytes of a LONGER previous run's tail.
    • Cause: _cmd_serialize used open(output,'w') + streaming json.dump. 'w' truncates at OPEN, then the doc streams across many writes holding an independent offset, so two concurrent serializers (the battery fans out; the AST daemon re-serializes) interleave — shorter doc at 0..N, longer doc's tail surviving past it.
    • RED-FIRST: a 3MB two-process control reproduced 0/12 and proved NOTHING — too small, the write completes inside the stdio buffer. I had asserted the mechanism before that control ran. Rebuilt at real scale (100MB docs, barrier-released): RED 2/5 rounds produced 'Extra data' with the identical prefix+tail signature; GREEN (atomic temp + os.replace) 0/8.
    • The identical fix already existed IN THE SAME FILE for the pickle sidecar, with a comment describing this exact incident — applied to the cache, not to the JSON artifact every consumer reads.
    • Converted all 4 json.dump sites (_cmd_serialize, _cmd_compare, _cmd_deep_compare, _cmd_trace_audit) to one sink. AST invariant now holds: 1 json.dump in the file, 0 outside it.
    • A swarm agent independently found the same defect 7 min later and wrote a shared src/orchestrata_ultimate/core/atomic_json.py (mkstemp in dest dir, mode preserved, fsync before replace, BaseException cleanup) with 8 passing tests — strictly better than my PID-unique-temp version, which could collide across threads in one process. Deleted mine, delegated to theirs, staged both.
  4. Closed both real suite failures: the RTM golden-number pin and an untested gate check that silently broke 7 worktree fixtures
    • Full observed suite (kernel STARTED and gRPC-verified, so authoritative): 32,821 passed / 8 failed / 0 errors / 0 skipped, 1616/1618 modules, 858s. Solo-retry: 4 batch failures -> 2 recovered as load/order flakes, 2 genuinely failed.
    • FAILURE 1 tests/scripts/test_derive_rtm.py: assert 786 == 777. The test carries an explicit protocol in its own comments — identify the cause, re-validate the structural invariants, and READ the numbers from derive_rtm's output, never compute them — with a recorded incident where someone computed a pin and left it 2 low for days. My first reading blamed the two new atomic_json modules because +9 matched a runtime_calls delta; that was a DERIVATION and it was REFUTED by measurement: neither module carries @traces, so both attribute zero. Real cause, measured: src/orchestrata_ultimate/kernel/cli_watch.py had ZERO @traces at HEAD and now carries @traces(spec='000', story='US-006', reqs=['REQ-000-097']), pulling its module closure into spec 000 — exactly 9 symbols. Pins now 786/580; invariants re-validated unchanged (48 specs with anchors, spec_006 36, three named anchors present, no scanner leakage). Verified: 12 passed.
    • FAILURE 2 tests/scripts/test_check_worktree_envs.py, 7 failures: scripts/check_worktree_envs.py gained a NEW check_env_missing (HEAD had 0 mentions of .env, worktree has 3) as part of the SOPS migration, enforcing RUL-021's full-env requirement. The gate is CORRECT; the 7 tests asserted the pre-change contract by building worktrees with only .venv. Fixed the fixture (_add_worktree now provisions .env, with the reason recorded) rather than weakening the gate.
    • SECOND DEFECT found underneath FAILURE 2: check_env_missing shipped with NO test of its own — every env-named test in that module is about .venv. That is why a new check silently broke 7 unrelated fixtures instead of being caught by its own coverage. Added 3 tests (violation when missing, None when present, None when no branch). They FAILED red first on my own errors — I invented WorktreeInfo(path,branch,head) when it requires 4 fields including detached, and asserted Violation.kind/.detail when the fields are check/message — corrected against the real dataclasses. Module now 37 passed, rc=0.
  5. Clean full-suite verification at HEAD 5eb80769: 32857 passed / 0 failed — the first suite result that actually describes the delivered branch
    • Ran ast-sync -> gate battery -> full observed suite sequentially on a quiet machine (load 1.52, one claude process, tree clean). ASTSYNC_RC=0, GATES_RC=0, SUITE_RC=0. 32,857 passed / 0 failed / 0 errors / 0 skipped, 1619/1619 modules, 472s. Kernel started and gRPC-verified, so authoritative. git-sentinel mutations 0, GIL assertions 0.
    • This was needed because my PREVIOUS suite run was invalid and I caused it: I ran the full suite concurrently with a 7-agent opus review workflow in the same live repo at load 21 on 6 cores. Four failures carried 'LIVE-REPO MUTATION DETECTED (git-sentinel)' and one was 'assert not sys._is_gil_enabled()' -> assert not True. Those were collision artifacts, not regressions. Clean run is 406s faster (878s -> 472s), which measures the contention cost.
    • The swarm's own verify phase independently reached the same conclusion as majors #4 and #6: reports/run-tests-latest.json recorded head_commit 5a34c8ce while 5eb80769 afterwards changed core/env_bootstrap.py and core/infra_config.py, so the 32853-passed figure quoted in the promotion report did not certify the delivered branch.
    • MASKED FLAKE to track: tests/agents/test_runtime_real.py failed in batch and recovered on solo retry. Same class as the existing masked-flake backlog items (conversation consumer flush, test_lsp_transport, test_session_facts).
    • A green suite does NOT clear either blocker. Blocker 1 is deployment wiring — the suite starts its own services, so it cannot observe that the DEPLOYED gateway fell back to the in-process client. Blocker 2's regression tests were deleted, so no test exists to fail.
  6. Blocker 1 fixed: gateway↔kernel wiring made ordered, observable, live-checked and detectable
    • Verified the blocker independently before changing anything: 58 fallback events vs 3 successful connects on the deployed fleet, current boot in fallback, both units active. Found it is THREE defects, not one — (a) boot race: kernel unit is Type=simple so After= orders SPAWN not READINESS, gateway probes 385ms early; (b) _init_kernel has one call site and one attempt, so the fallback is permanent; (c) on fallback self._kernel_client is set to the PIPELINE client, so '_kernel_client is not None' is true in BOTH states and nothing in the process could tell them apart. handle_info also advertised 'cxp' in capabilities unconditionally. A detector was impossible to write, not merely absent.
    • Dispatched 3 builders against a contract I pinned first (kernel block in gateway.info). ALL THREE came back sound=false from their opus verifiers — 11 defects, every one reproduced by command. Two were blocking: a helper named _fake_gateway_with_info tripped audit_test_purity's 'fake' prefix rule and turned the RUL-008 purity gate red repo-wide (9 violations, rc=1 -> rc=0 after rename); and GatewayConfig is a frozen dataclass so two new retry fields broke the generated ordered-param interface test, which the builder never ran (incomplete-scope green).
    • THE DEFECT THAT WOULD HAVE RECREATED THE BLOCKER: _kernel_mode was latched at boot and never refreshed, so gateway.info would answer wired:true forever after a kernel death and the detector passed on that stale value. Extended the contract rather than pretend it covered this: kernel block now carries live (probed at request time, bounded 2s) and wired = mode=='grpc' AND live. Detector fails with 'wired at boot but does NOT answer now'; a build omitting 'wired' is UNKNOWN, not OK.
    • Also closed: tautological 'assert rc == 0 or rc != 0'; a non-discriminating assertion using the REAL configured endpoint 10.93.53.13:50053 as its sentinel (now sentinel-kernel-host:59999); untested unit wiring (verifier proved deleting ExecStartPre AND TimeoutStartSec left the module 8/8 green — the same hole that cost this repo a boot failure on 2026-08-11) — added 5 guards asserting the probe is wired, fatal (no '-' prefix), and present on disk; the untested exit-0 path (fatal ExecStartPre, so a defect there stops the gateway starting) — added a real DirectCXPServer success test with a paired negative control; and stop() calling disconnect() on a PipelineKernelClient that only has close(), AttributeError swallowed by a blanket handler whose message named nothing.
    • MY OWN ERRORS, all caught by targeted runs: prepended 'import pathlib' above 'from future'; invented a fixture parameter that does not exist; used '--timeout' when the probe takes a positional; put StartLimitIntervalSec/StartLimitBurst in [Service] where systemd ignores them (moved to [Unit], systemd-analyze verify silent); and introduced a REAL regression — gating the first attempt on room for a worst-case 3x attempt_timeout made a short deadline run ZERO attempts and report 'not ready' without ever looking. Retry budget now uses MEASURED attempt cost: closed port polls the full budget (1.93s of 2.0s), a wedged endpoint's own 3s measurement stops the loop. Default timeout 55->45 so 45 + 3x3 = 54 <= the 60s outer wrapper; the documented '5s margin' was arithmetically negative before.
    • Targeted verification: 5 modules, 170 tests, rc=0. Full battery + full suite running now; deployment and live link proof still to come.
  7. Blocker 2 closed: non-Python evidence restored AND made non-silent
    • Restored .js/.jsx/.mjs/.ts/.tsx/.html/.css evidence lost in 29bbf088, via _non_python_scan.py + serialize_js_ts_ast.py
    • Parity vs master on master's own fixture: 10 facts vs 8 — strict superset (AST also finds methods render/connect the regex missed)
    • All 3 swarm packets FAILED adversarial verification first: js_ts_diagnostic was write-only (attr-refs: 0 production readers in src/ and scripts/)
    • Two proof tests stayed green with node deliberately broken — fixture basenames spelled the identifiers, so only the basename tier was load-bearing; renamed both fixtures to panel.tsx
    • Fixed 3 defective tests: one asserted stdout=='' + rc==0 as correct, one named '_with_facts_exits_zero' never asserted a fact, one used a brittle 'Found' not in stdout substring that false-fails on class FoundationModel
  8. Two live Prolog defects found and fixed while proving the blocker-2 consumer
    • JOIN KEY: match_requirements_to_evidence emitted implemented(Spec, Description) but spec_completeness.pl unifies implemented(Spec, ReqId) — they can never unify, so implementation_coverage/3 returned implemented_count=0 for EVERY spec in EVERY language regardless of evidence. Measured: exact-matching fact -> {'implemented_count': 0, 'spec_complete': False}. Fixed on the producer side (contract forbids touching the .pl).
    • ACCUMULATION: _run_prolog_evaluation asserted dynamic facts without retracting, so re-evaluating one 1-requirement spec in one process reported total_count=2 — the denominator grew with how many times the process had been asked. Fixed with a scoped retractall.
    • The guarding test asserted only isinstance(result, dict), so it stayed green through both. Replaced with 3 real regression tests.
  9. RUL-030 mechanized: generated artifacts belong in git
    • scripts/check_generated_artifacts.py + 14 tests against a real temp git repo (no monkeypatch; repo_root/artifacts injection seams)
    • Wired as the 'generated-artifacts' gate in scripts/gates.py, fingerprinted on .gitignore + its own registry rather than _SRC — index state is not a content tree, so caching on _SRC would let one .gitignore line drop an artifact silently
    • Live violation found and fixed: docs/CODEBASE_MAP_FULL.md was gitignored (.gitignore:223, 'too big to track/import') — the exclusion conflated tracking it with @importing it; 1.6 MB, now tracked
    • Ruling published as wiki page 573; rulings_lint now checks 30 rulings, all mechanisms verified
  10. Unified the RTM-evidence artifact on reports/ (commit 735eec58)
  1. CORRECTION to the RTM-denominator work item: no checklist GENERATOR exists
  1. Fixed all 21 production dangling @traces at the root: the RTM loader read a narrower corpus than the repo's ratified one
  1. Fixed the staged-secrets gate that forced --no-verify on every spec edit (9454e2dc)
  1. ship-final failure triage: all 3 failing modules fixed
  1. runtime_input split in the RUL-030 registry
  1. plugin-drift gate wired into gates.py battery
  1. installer hardening from the migration audit
  1. swarm builders complete; composed selection previewed at 13 modules
  1. Adversarial verifier refuted 2 of 8 claims on the selection fixes
  1. Re-verifier REFUTED round 1 on 3 findings; builder resumed for round 2
  1. Builder round 2 closed all 3 re-verify findings; 136 tests green; verify pass 3 dispatched
  1. Plugin cutover complete: orchestrata plugin is the live hook delivery path, enforced current on every deploy
  1. Ship day closure: 16 commits landed via 3 promotes; bot unit convention violation found by verification battery and fixed

Artifacts

Decisions

Achievements

Upcoming

Handoff