Orchestrata Ultimate › Sessions › Session 254

Session 254

‹ previous · next › · timeline

Prose pending.

Purpose

Session 254 — post-SPEC-023 follow-ups: B5 per-gate wiring wave, ft-grpcio teardown SIGSEGV, kg-schema gaps #2/#3/#6/#7/#8, B4a.2 byte-race hardening

Work log

1. Article XII (Plan Fidelity) added to the constitution: constitutional.pl plan_fidelity/2 + constitutional_check(12), ARTICLE_COUNT 11→12, ARTICLES registry entry, enforce/permit suite coverage; B17 meta-lint PASS, 153+136 tests green
   • Human-ratified by Mohamed 2026-07-24: an active ratified plan is executed as written; divergence only via a human-raised issue. Facts active_plan/1, plan_step/2, human_issue_raised/2 asserted ground from Python. Also fixed pre-existing FUNCTION_TOO_LONG on ConstitutionalEnforcer.check_article (extracted _article_result).
2. Wiki diagram saturation task started: inventory found 208 pages, 52 with diagrams, ~120 real pages without; plan = 9 parallel Sonnet batches author 8.8.2-safe mermaid fences, then render→upload→bulk-import→index pipeline + adversarial accuracy review
3. Diagram wave: 9 authoring batches done (98 fences, all 8.8.2-validated); 4 standalone adversarial reviewers found 38 real defects -> 3 fixer agents dispatched; visual-AST swarm (20 agents, 5 projects) + 5 review cycles running as workflow after user directives (visual AST, 20-agent swarm, 5 cycles)
4. write: /home/mohamed/.config/opencode/opencode.json
5. write: /home/mohamed/.config/opencode/opencode.json
6. edit: /home/mohamed/.config/opencode/opencode.json
7. edit: /home/mohamed/workspace/orchestrata-ultimate-architecture/.opencode/opencode.json
8. Thread ontology backfill APPLIED to live KG
   • Fixed builder CLI inversion (dry-run now default, --apply explicit; 83c9bc9e; 20/20 tests). Apply matched contract exactly: 9950 Thread/9950 HAS_THREAD/26107 IN_THREAD/12588 BY_VOLK/15603 TO_VOLK; 0 isolated CXPPacket/Message/ToolCall/Artifact; idempotency proven by graph recount after 2nd apply. +202 pre-existing AuditEntry BY_VOLK literals formalized. Wiki page updated with final numbers.
9. Boot health FAIL triage
   • cachyos-rate-mirrors boot DNS race — restarted OK, 0 failed units
   • neo4j-backup boot catch-up failed before container up (03:00 run fine, fresh 198M dump); added bolt-port ExecStartPre guard + ordering
   • Retention timer first 04:00 run verified clean, flags honored
   • xhci post-resume HC death = dock transient
10. Root-connectivity invariant achieved: every KG node leads to the root Project (UNREACHABLE=0)

• Census: 1 Project root, 2,655 WCC components, 6,211 islanded nodes decomposed into 6 root causes
• User-ratified purge D1-D6: 80,790 -> 75,824 nodes (spec/pattern husks 180, Intent/PipelineRun exhaust 4,685, architecture fragment 45, foreign-slug dogfood tree 31, straggler husks 15, test/dupe volks 10; 269 EVALUATED_BY edges rewired to canon aliases, 202 junk v1 BY_VOLK edges dropped)
• Backfills: 322 HAS_SCENARIO, 9 HAS_VOLK (new canonical vocab), 26 code-root CONTAINS, ADR-0037 Decision BY_VOLK->U:Ar
• Sweeper deep fingerprints extended to spec_id/name/thread_id (the evasion properties); 100/100 tests
• Builder in flight: 5 writer fixes + check_kg_connectivity gate wired into health_check

11. Connectivity gate + writer fixes landed (c33e6bdd): 66,160/66,160 reachable, gate exit 0, wired into health_check

• Builder: HAS_VOLK at kernel boot, HAS_SCENARIO in harness history, volk-alias canonicalization, audit writer no longer mints Volks from unresolved ids; 178 tests
• Sweeper thread_id infix forms swept 7,337 more exhaust
• FLAGGED DEBT: kernel/audit AuditLogger subsystem has no production caller — needs wiring decision

12. KG root-connectivity: fixed the reference-data seeder

• seed_reference_data() wrote 5 WAFPillar + 30 CloudService with no edge at all; now requires a project slug and MERGEs (:Project)-[:REFERENCES]->(node). Live graph repaired (35 linked); gate 67453/67453 reachable, 0 unreachable. Sweeper also fingerprints degree-0 FactoryBuildResult husks.

13. Migration Step 1.3: orchgraph backup + daily timer, restore-drilled

• backup_orchgraph.sh resolves MAIN from SHOW INSTANCES, CREATE SNAPSHOT per database, pulls root + per-uuid snapshot dirs, writes manifest.json, prunes to 14. User units orchgraph-backup.{service,timer} enabled; real systemd run pulled memgraph 494M + orchkg schema. Restore drill: scratch container from the snapshot alone came up with 71 constraints + 76 indexes matching source.

14. wiki-kimi frontmatter + vocabulary defects fixed

• 21 rulings pages had invalid YAML (bare ': ' in description); parse_frontmatter swallowed the error and misreported it as missing keys. Added strict= + _yaml_scalar quoting + taxonomy.md-derived controlled tags; 13 new tests; pages republished.

15. SKELETON_FLOW structural escape: task.py

• STRUCTURAL: 2 escape(s) A.init: method deleted A.m: method deleted

16. SKELETON_FLOW structural escape: task.py

• STRUCTURAL: 2 escape(s) A.init: method deleted A.m: method deleted

17. SKELETON_FLOW structural escape: task.py

• STRUCTURAL: 2 escape(s) A.init: method deleted A.m: method deleted

18. Migrated the three worker Neo4j instances to the canonical schema

• Six suite modules failed with ConstraintValidationFailed on Directory(path)='.'. Root cause: .13/.14/.19 still carried legacy directory_path/file_path constraints on the bare path property while the primary .11 had been migrated to project-qualified id keys. A uniqueness constraint on bare path makes it impossible for two projects to both have path='.' — exactly the multi-project invariant the canonical migration exists to satisfy. Ran scripts/migrate_neo4j_canonical.py per worker: .13 purged 57 / backfilled 19719, .14 46 / 19717, .19 140 / 429; each dropped 7 wrong-key constraints and applied 78 canonical DDL statements. All four instances now carry only directory_id/file_id.

19. Fixed two order-dependent false-confidence governance tests

• test_gateway_governance_handler and test_publisher_wiring both asserted a deny that came from an UNDEFINED PREDICATE, not a policy. janus state is process-global, so with no module having loaded governance.pl the may_execute/2 query errored and the non-strict error branch produced the deny. With the rules loaded, governance.pl:848's permissive fallback correctly ALLOWS an unregistered actor and both tests flip. Reproduced deterministically by running test_policy_gate first. Both now load the rulebase and drive the deny via governance:agent_action_blocked/2 on uuid-scoped pairs; a mirror test pins the permissive ALLOW. Commits 7c013903, b4f44a59.

20. Fixed the shared-Memgraph read-after-write visibility flake and a self-contradicting gate

• add_chunks() returning is not proof the write is queryable: the vector index is populated behind the write transaction. Under load this failed test_retrieve_with_zero_graph_depth in the batch round and test_retrieve_matches_entity_by_property in the solo-retry round while passing standalone. Added _retrieve_until_visible (re-query with existing backoff until the chunk id appears, returning the last context so a chunk that never lands still yields the real diff). Separately, test-review.py's T4 flagged a session-scoped fixture that carries a reasoned '# gate-ok:' waiver which gate_parallel_safety.py already honours; unified the waiver vocabulary (bare marker still does not waive; class scope still unconditional) with 3 new tests.

21. SKELETON_FLOW structural escape: task.py

• STRUCTURAL: 2 escape(s) A.init: method deleted A.m: method deleted

22. SKELETON_FLOW structural escape: task.py

• STRUCTURAL: 2 escape(s) A.init: method deleted A.m: method deleted

23. SKELETON_FLOW structural escape: task.py

• STRUCTURAL: 2 escape(s) A.init: method deleted A.m: method deleted

24. Dedicated test Memgraph + schema-leak closure (commit 5cd0f43f)

• Routed suite at standalone prod-identical Memgraph 10.92.53.167 (Terraform-modeled, binary sha256 == prod build)
• Fixed 2 architecture-twin fixtures leaking uuid-scoped chunk-label schema
• Added test-review rule T17 + scripts/purge_memgraph_test_schema.py, both with tests
• Purged 823 leaked labels from prod: 484->207 indexes, 898->75 constraints, all 5 prod vector indexes intact
• Fixed test_mvcc_retry negative-control false negative via a start barrier (5/5 green)
• Verified prod unchanged (+0/+0/+0) after a full semantic-graph run; 498 passed across the affected set

25. SKELETON_FLOW structural escape: task.py

• STRUCTURAL: 2 escape(s) A.init: method deleted A.m: method deleted

26. SKELETON_FLOW structural escape: task.py

• STRUCTURAL: 2 escape(s) A.init: method deleted A.m: method deleted

27. SKELETON_FLOW structural escape: task.py

• STRUCTURAL: 2 escape(s) A.init: method deleted A.m: method deleted

28. Authoritative suite GREEN (run 14)

• 29,045 passed, 0 failed, 0 errors, 0 skipped; 1451/1451 modules in 471s; kernel started + gRPC-verified (port 50053)
• 2 modules flaked in batch and recovered solo (test_audit_logger_enforcement_proof, test_kg_schema)
• Fixed 9 failing modules across runs 12-13: orchkg database topology on the test node, dotenv override precedence, graph_browser toString-on-list bug, audit poll budget

Artifacts

• /home/mohamed/.config/opencode/opencode.json
• /home/mohamed/.config/opencode/opencode.json

Decisions

• Article XII priority=1 (Phase 1), but CRITICAL_ARTICLES stays [1,3,7]: CRITICAL_ARTICLES is the pinned critical-SAFETY fast path (test_governance.py asserts [1,3,7]); plan fidelity is a human-authority rule, priority-1 like Article I, enforced via check_all/check_article
• Root-connectivity invariant ratified: every node must lead to the root Project node: User ruling session 254; no floating subgraphs — shared entities (Volk, corpus) get hub edges rather than whitelist exceptions; garbage is deleted not linked; enforced by BFS gate (check_kg_connectivity, building)
• Migration Phase 0 ratified: dedicated orchgraph-kg cluster (0.1), NO Bolt auth (0.2, user ruling), KG_* env seam with NEO4J_* alias: User ratified 2026-07-26; auth-less matches live semantic cluster posture on the lab network; execution is GO — swarm execution end-to-end per plan
• Migration re-ratified: operational KG = multi-tenant database 'orchkg' on the EXISTING orchgraph cluster (user ruling), dedicated-cluster decision superseded, 5 new containers destroyed: Live proof 2026-07-26: CREATE DATABASE isolation + driver database= param + HA replication of second database all verified on the shared cluster; label collision eliminated by per-database storage; user accepts process/failover coupling
• Did NOT change governance.pl's default-ALLOW posture; pinned it with a test and surfaced it as Mohamed's decision: governance.pl:848 allows any agent that is not a registered volk and is not explicitly blocked. Fixing the two false-confidence tests exposed this, but flipping a production governance posture to default-deny is a behaviour decision with system-wide blast radius, not a test fix — and the deny-posture question is already recorded as the user's open call. The posture is now asserted explicitly by test_handle_evaluate_with_injected_engine_allows so it can never flip silently, and reported as an open decision rather than changed unilaterally.
• Did NOT fold kimi/k1-vizint-failures into the master merge: 6 unmerged commits, 444 insertions across 15 files including production gateway code (handlers/cxp.py +142, rpc.py, protocol.py, server.py). git merge-tree reports 6 conflict regions, at least one in tests/semantic_graph/test_memgraph_adapter.py which this session also edited. Outside the stated scope (merge feat/arch-canvas to master), already on origin so 'push everything' is satisfied, and folding it in silently would put unreviewed gateway changes on master. Recorded as task #18 for an explicit decision.

Upcoming

• Hub-link remaining runtime KG writers

Handoff

• Key decisions: None
• Blockers: kimi/k1-vizint-failures (6 commits, prod gateway code, 6 conflict regions) is unmerged and needs Mohamed's decision, ingest_orchestrata_ultimate.py stalls at exactly 4096 chunks on a fresh Memgraph node — not root-caused
• In-progress: Hub-link remaining runtime KG writers (760 unreachable nodes), Migration phases 3.2 query-parity harness, data migration run, Phase 4 cutover, then Neo4j node deletion
• Important files: tests/core/test_test_memgraph_routing.py, scripts/purge_memgraph_test_schema.py, incus/orchgraph-test-node-deploy.sh, infra/terraform/memgraph_test.tf
• Commands: incus/orchgraph-test-node-deploy.sh # re-provision the test Memgraph, python3 scripts/purge_memgraph_test_schema.py --uri <uri> # dry-run leaked-label purge
• Warnings: feat/arch-canvas MERGED to master + pushed (57f48f3f, 58 commits). Suite GREEN: 29,045 passed / 0 failed / 0 errors / 0 skipped, 1451/1451 modules, kernel gRPC-verified on 50053., The suite no longer writes to the production orchgraph cluster — it talks to memgraph-test 10.92.53.167. Do NOT unset TEST_MEMGRAPH_URI without reading docs.

Related

• Session log
• Session timeline
• Orchestrata Ultimate — project index
• Workflow