Orchestrata Ultimate › Plans › swarm-execution-plan-2026-08-12

Swarm execution plan — suite runtime + graph defects (2026-08-12)

Source: plans/swarm-execution-plan-2026-08-12.md at 8a2cb42ad · last changed 2026-08-12 · first committed 2026-08-12 · 88 lines.

Prose pending.

Plan text

Swarm execution plan — suite runtime + graph defects (2026-08-12)

Executes plans/suite-runtime-and-graph-defects-plan.md (with §6b/§8 second-review corrections) per the user sign-off: both scopes, suite-side first, executed in parallel by a swarm of coder sub-agents, each in its own wt-* worktree.

Research backing: plans/suite-runtime-research-2026-08-12.md. Baseline: 445.7s wall / 2,312s call CPU @ cf59d5c8. Floor range 289–358s. Final gate reports wall and call CPU (§8-6: call CPU varies ~24% run-to-run; a wall-only comparison cannot prove a win).

Hard rules for every work item

• Worktree per item: cd /home/mohamed/workspace/orchestrata-ultimate-architecture && .venv/bin/python scripts/worktrees.py create <NAME> — never hand-run git worktree add. Work with cd /home/mohamed/workspace/wt-<NAME> && env -u PYTHONPATH .venv/bin/python ....
• Tests ONLY via env -u PYTHONPATH .venv/bin/python scripts/run_tests.py <targets> --skip-gates --skip-infra --force. Plain pytest is banned; xdist is banned.
• Commits in the worktree with --no-verify, message naming the work item. Do NOT merge, do NOT push, do NOT touch master — merges happen centrally after all items land (git-state mutations during a suite run trip git_sentinel).
• No mocks/stubs/fakes/monkeypatch in tests (purity gate reads fake_* names as stubs; use DI seams and _recording_* names).
• Do not edit shared conftest files (tests/conftest.py, tests/scripts/conftest.py, any conftest outside your module). If your fix needs one, implement it inside your own module instead; if truly impossible, stop and report the need.
• Match surrounding code style; update stale docstrings; minimal diffs. No AI attribution in commits.
• Final report must include: files changed (+ diffstat), verification command output (PASS/FAIL with numbers), per-module before/after wall where the item targets runtime, and anything explicitly NOT done.

Work items

WI-1 — D7: permanent errors must not enter the memgraph retry storm

• Branch/worktree: wt-D7-RETRY-GUARD (branch feat/d7-retry-guard).
• Files: src/orchestrata_ultimate/semantic_graph/adapters/memgraph.py (_execute_with_reconnect, ~line 899), src/orchestrata_ultimate/core/graph/retry.py (marker list — extend only if needed), tests/semantic_graph/test_memgraph_adapter.py.
• Measured defect: adapter classifies on type(exc).__name__ == "TransientError" alone; a 50N42 "unknown database" error burns 75.75s of backoff PER operation (12 attempts) and connect() to a nonexistent db never failed within 250s. NOTE (§6b-2): the classifier is a disjunction — the fix needs short-circuit ORDERING (permanent check before contention check), not just a shared marker list.
• Do: classify the unknown-database / 50N42 class (and any clearly-permanent gql status) as permanent → fail in < 1s; share retry.py's _PERMANENT_MARKERS rather than duplicating the list.
• Verify: new test pointing the adapter at a provably nonexistent database asserts failure in < 2s (real adapter, real cluster — set -a; . ./env/test.env; . ./.env; set +a pattern for any standalone probe); run_tests.py tests/semantic_graph/test_memgraph_adapter.py --skip-gates --skip-infra --force green.

WI-2 — D1: MemgraphSemanticGraph.retrieve batched graph-side expansion

• Same worktree as WI-1 (wt-D7-RETRY-GUARD), implemented AFTER WI-1 lands there (same file).
• Measured defect: retrieve(top_k=5, depth=1) = 16.19s with 10,964 individual get_neighbors Bolt round-trips (real 13,747 entities + real-code chunks); _all_entities() full-graph fetch 0.90s per call.
• Do: (a) replace per-entity get_neighbors calls with ONE batched expansion query (MATCH (e) WHERE e.id IN $ids + variable-depth rel traversal) — this carries the win; (b) candidate selection: §6b-3 — do NOT invert needle/haystack ($chunk_text CONTAINS e.id is unindexable). First DECIDE matching semantics (document the decision in the plan file §6b-3 note): keep the substring heuristic with the scan pushed into ONE Cypher query (measure the server-side scan cost on the real 13,747-entity set before and after), or replace with exact-token matching if measurement shows the server-side scan is still > 1s. The "tens of ms" target is withdrawn; report the measured before/after.
• Verify: seeded scratch-db fixture (create sem_test_review_<pid> via CREATE DATABASE, seed real entities read from the configured db + real-code chunks, measure, drop): assert round-trips ≤ 3 per retrieve and wall < 50% of the pre-fix measured 16.19s on the same seed; adapter module tests green; scratch db dropped at the end.
• §6b-3 decision (recorded 2026-08-12 by the WI-1/WI-2 worker): KEEP the substring heuristic with the scan pushed into the ONE Cypher query. Measured server-side scan cost on the real 13,747-entity set (5 real-code chunk texts, any(text IN $texts WHERE text CONTAINS start.id OR any-string-property)): 0.833s — under the 1s threshold, so exact-token matching was NOT adopted. Same-seed before/after (13,747 real entities / 38,534 rels / 5 real-code chunks): old per-entity fan-out 21.74s, 14,733 round-trips; new batched match+expansion 1.93s, 2 round-trips.

WI-3 — D2+D2b: CIG query caching + approximate betweenness

• Worktree: wt-CIG-CACHE (branch feat/cig-query-cache).
• Files: src/orchestrata_ultimate/refactor/codemod/_cig.py, refactor/codemod/server.py, scripts/codemod.py, tests/refactor/codemod/ (analysis/IPC/CLI modules).
• D2 measured: 0.66–0.72s full-graph PageRank per SINGLE-node query. Do (§8-3): LAZY memoization of the rank vector per instance (no mutation API exists — _nodes/_csr are build-time only; do NOT build invalidation machinery, do NOT compute at build time). NOTE (§8-4): BlastRadius.impact_score (|reached|/|nodes|, _cig.py:95) vs CodeIntelligenceGraph.impact_score (PageRank, _cig.py:401) collide on the same JSON-RPC surface — rename the PageRank one (e.g. pagerank_score) shipped with the cache; use scripts/serialize_codebase_ast.py rename for the symbol + refs first for blast radius; keep the RPC method name backward-compatible or bump it explicitly with a note.
• D2b measured: exact Brandes centrality = 340.37s on the real CIG (38,369 nodes). Do: pivot-sampled approximate betweenness (k≈256 sources; arrays/lists indexed by node id, not per-source dict-of-dicts) as the default for query_centrality/CLI, exact mode opt-in flag. Acceptance: approximate call < 5s on the real CIG; top-20 overlap vs exact ≥ 90% on a mid-size fixture graph (the existing fixture_package is fine).
• Verify: per-instance cache check (second impact_score call ~0ms); benchmark artifact on the real CIG written to /tmp and its numbers pasted in the report; run_tests.py tests/refactor/codemod/ --skip-gates --skip-infra --force green.

WI-4 — D3: shared fact-extraction + consult across gate tests

• Worktree: wt-FACT-SHARE (branch feat/fact-share).
• Files: tests/gates/test_fact_extractor.py, tests/gates/test_freshness_inference.py (only these two).
• Measured: live extract 2.07s + consult 4.04s per test; 5 live-snapshot tests hold 32.7s (96%) of the module's 34.1s call CPU.
• Do: session-scoped shared live extraction + engine consult using the repo's T4 immutable-module-scope pattern (module-scope fixture produces an immutable artifact; function-scope wrappers hand out fresh reads — precedent _async_multi_results_json in tests/core/threading/test_test_executor.py). Per-test synthetic-snapshot tests must stay per-test. Careful with janus global engine state: consulting the same base twice appends — design so the shared consult happens exactly once per session per process (worker process); document the mechanism in the fixture docstring.
• Verify: run_tests.py tests/gates/test_fact_extractor.py tests/gates/test_freshness_inference.py --skip-gates --skip-infra --force green; module wall before/after in the report.

WI-5 — Pipeline-once fixtures for agent_check

• Worktree: wt-AGENTCHECK-ONCE (branch feat/agentcheck-once).
• Files: tests/scripts/test_agent_check.py, tests/scripts/test_agent_check_e2e.py (only these two).
• Measured: e2e module 39.6s across 4 tests (§6b-7), agent_check 27.9s; each test re-runs real ruff/mypy/pytest pipelines + time.sleep(1.5) handshake waits.
• Do: run each pipeline ONCE in a module-scope fixture, freeze the result (T4 immutable pattern), assert per-test on the frozen artifact; replace fixed sleeps with event/polling waits bounded by the actual condition.
• Verify: both modules green via run_tests; per-module wall before/after in the report; no behavior change to what is asserted.

WI-6 — Retry-budget DI for timeout-path tests

• Worktree: wt-RETRY-BUDGET-DI (branch feat/retry-budget-di).
• Files: tests/knowledge_graph/test_graph_client_coverage.py, src/orchestrata_ultimate/knowledge_graph/graph_client.py (DI seam only), tests/refactor/rename/test_lsp_transport.py, src/orchestrata_ultimate/refactor/rename/lsp_client.py (grace/escalation timeouts DI seam only).
• Measured: test_execute_retries_transient_error_then_raises 13.0s (real backoff), test_close_escalates_terminate_then_kill_on_unresponsive_process 10.5s (real SIGTERM-grace escalation).
• Do: add optional constructor/keyword injection for retry budgets and shutdown grace periods (defaults unchanged — production behavior identical); tests inject short budgets. Precedent: test_session_graph_store.py 2026-08-11 (42.7→4.24s). Do NOT touch core/graph/retry.py constants (owned by WI-1's scope boundary — if a change there is truly needed, stop and report).
• Verify: the named tests < 3s each; both modules green via run_tests; production defaults asserted unchanged by an explicit test.

WI-7 — AST/index sharing for query-heavy modules

• Worktree: wt-AST-SHARE (branch feat/ast-share).
• Files: tests/scripts/test_gen_brief_pack.py, tests/test_orchestrata_tools.py, tests/scripts/test_evaluate_semantic_graph_qa.py (only these three).
• Measured: 41.2s / 18.5s (single test) / 24.8s call CPU respectively; costs are repeated real index/AST queries per test.
• Do: extend the AST-daemon session-fixture pattern (shipped 2026-08-11 in tests/scripts/conftest.py — READ how tests/scripts/conftest.py uses it first; you may USE the existing session fixture but must NOT edit shared conftest files) to these modules: build/query the index once per session, immutable snapshots per test. If a module cannot use the existing fixture without a conftest change, implement the session-scope sharing INSIDE that module.
• Verify: three modules green via run_tests; per-module wall before/after in the report.

WI-8 — Fixture sharing for test_search (highest measured fixture share)

• Worktree: wt-SEARCH-FIXTURES (branch feat/search-fixtures).
• Files: tests/tools/test_search.py only.
• Measured: 51% of the module's junit time is fixture setup/teardown (10.5s of 20.7s across 80 tests ≈ 130ms/test) — the highest share in the falsify probe.
• Do: identify what the per-test fixtures repeat (read the module first: likely index/client/session construction); move repeatable construction to module/session scope with the T4 immutable pattern ONLY where semantics allow (tests that mutate shared state keep per-test fixtures — say which and why in the report).
• Verify: module green via run_tests; wall before/after; fixture share recomputed with the probe method (--junitxml + observer durations) and reported.

Central merge order (after all items report)

1. Suite-side: WI-4, WI-5, WI-6, WI-7, WI-8 (disjoint files; any order within group)
2. WI-3 (codemod subtree)
3. WI-1+WI-2 (same worktree, memgraph adapter) Each merge: --no-ff into master from the primary checkout, targeted affected-tests gate (scripts/run_tests.py --affected origin/master...HEAD --force) before push, push from a worktree (git push origin master:master).

Final gate (central, after all merges)

scripts/run_tests.py --force full suite. Report: wall vs 445.7s baseline AND call CPU vs 2,312s (§8-6), PASS/FAIL with numbers. Update the checklist implement/verify items from the result.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks