Orchestrata Ultimate › Plans › fireflies-slice-full-suite-remediation

Full-suite remediation plan — fireflies slice worktree

Source: plans/fireflies-slice-full-suite-remediation.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-12 · 559 lines.

Prose pending.

Plan text

Full-suite remediation plan — fireflies slice worktree

Generated from the full-suite run python scripts/run_tests.py --skip-gates executed in /home/mohamed/workspace/orchestrata-kimi-wt on 2026-07-12. Baseline: 24,785 passed, 29 failed, 0 errors. One failure (tests/session/test_session_fireflies_tools.py::test_returns_four_tools) was caused by the fireflies slice and is already fixed. This plan covers the remaining 28 failures across 22 modules.

After applying this plan's fixes, the full suite improved from 24,785 passed / 29 failed to 24,916 passed / 5 failed. All 5 remaining failures are in tests/scripts/test_benchmark_pipeline.py and are caused by the default SPIKE_PROFILE=default routing to DeepSeek, which returns HTTP 402 (Payment Required). Running the module with SPIKE_PROFILE=kimi yields 10 passed.

Status: fixes applied in worktree.

Executive summary

The failures cluster into five root-cause buckets. Only bucket 1 is worktree-environment specific; buckets 2–5 are latent defects exposed by running the suite from a git worktree with a fresh working directory.

Bucket A — Missing worktree environment (8 modules)

Affected modules

• tests/scripts/test_gen_level0.py
• tests/scripts/test_enrich_semantic_graph_specs.py
• tests/scripts/test_semantic_search.py
• tests/scripts/test_start_test_services.py
• tests/scripts/test_sync_architecture_to_neo4j.py
• tests/scripts/test_benchmark_pipeline.py
• tests/session/test_session_meta.py
• tests/agents/test_tool_registry.py

Root cause

.env is gitignored, and vendor/wheels/grpcio-1.82.1-cp314-cp314t-linux_x86_64.whl is untracked. git worktree add copies neither. The worktree therefore has:

• no .env → every script that resolves PROJECT_ROOT / ".env" fails to load keys;
• no vendored free-threaded grpcio wheel → uv run fails when resolving grpcio = { path = "vendor/wheels/grpcio-..." }.

Fix

Copy the two artifacts from the main checkout into the worktree:

cp /home/mohamed/workspace/orchestrata-ultimate-architecture/.env \
   /home/mohamed/workspace/orchestrata-kimi-wt/.env

mkdir -p /home/mohamed/workspace/orchestrata-kimi-wt/vendor/wheels
cp /home/mohamed/workspace/orchestrata-ultimate-architecture/vendor/wheels/grpcio-1.82.1-cp314-cp314t-linux_x86_64.whl \
   /home/mohamed/workspace/orchestrata-kimi-wt/vendor/wheels/

# Re-sync uv in the worktree so it sees the wheel
cd /home/mohamed/workspace/orchestrata-kimi-wt
uv cache clean grpcio
uv sync --dev

Verification

cd /home/mohamed/workspace/orchestrata-kimi-wt
uv run pytest tests/session/test_session_meta.py -v
uv run pytest tests/agents/test_tool_registry.py -v

Long-term hardening

Add a worktree-bootstrap step to the delegation plan or CI so future worktrees are provisioned with .env and vendor/wheels/ automatically.

Bucket B — Stale signature pins (4 modules)

Affected modules

• tests/generated_gateway/test_gateway_handlers_session_generated.py
• tests/generated_gateway/test_gateway_handlers_volk_generated.py
• tests/generated_gateway/test_gateway_pubsub_generated.py
• tests/postman/test_postman_signatures.py

Root cause

The worktree branch kimi/fireflies-slice is at 4bc8180e, which predates master commit d024b289 ("test(pins): update signature pins to wave-B public-API truth"). Intentional source changes in 4162961f and 1c9e71b4 added parameters that the generated pin tests now assert are missing:

Fix

Cherry-pick the already-reviewed pin update into the worktree:

cd /home/mohamed/workspace/orchestrata-kimi-wt
git cherry-pick d024b289

This updates all four pin files at once. If cherry-pick conflicts, resolve only the four test files listed above.

Verification

cd /home/mohamed/workspace/orchestrata-kimi-wt
PYTHONPATH=src:/home/mohamed/workspace/orchestrata-kimi-wt \
  /home/mohamed/workspace/orchestrata-ultimate-architecture/.venv/bin/python \
  -m pytest tests/generated_gateway/ tests/postman/test_postman_signatures.py -v

Note on SubscriptionManager.subscribe

One subagent reported an additional mismatch for SubscriptionManager.subscribe (filters parameter). That parameter is present in the main checkout's staged source but not in the worktree source. Because the full-suite run used the main checkout's editable .venv, the test saw the newer source. Once the worktree source is the object under test, only the four pins in d024b289 are required. If the worktree is later rebased onto master, re-run the pin tests and regenerate any newly stale pins.

Bucket C — Batched-runner marker handling (6 modules)

Affected modules

• tests/e2e/test_full_pipeline.py (llm_integration)
• tests/llm/test_qwen_minimax_e2e.py (llm_integration)
• tests/orchestrator/test_pipeline_e2e_analysis.py (llm_integration)
• tests/orchestrator/test_pipeline_e2e_docs_session.py (llm_integration)
• tests/orchestrator/test_pipeline_e2e_review.py (llm_integration)
• tests/session/test_session_coverage_gates.py (slow)

Root cause

Two independent defects combine:

1. tests/conftest.py collapses pytest exit code 5 to 1. When a module is fully deselected by the default marker filter (-m "not slow and not llm_integration and not incus_live"), pytest legitimately exits with code 5 (NO_TESTS_COLLECTED). The gRPC shutdown workaround at tests/conftest.py:682 rewrites every non-zero status to 1:

   os._exit(0 if _session_exit_status == 0 else 1)
   

   The batched executor already recognizes exit 5 as a no-op (_pytest_no_tests_collected), but it receives 1 instead, so it treats the module as failed.

2. scripts/run_tests.py does not override the default marker filter. The runner claims to run the "full suite", but it inherits the pyproject.toml addopts that exclude slow and llm_integration. The slow tests therefore never run in the default lane, and fully excluded modules trigger defect C1.

Fix

C1 — Preserve pytest exit status in tests/conftest.py

Change tests/conftest.py:682 from:

os._exit(0 if _session_exit_status == 0 else 1)

to:

os._exit(0 if _session_exit_status == 0 else _session_exit_status)

This keeps the hard-exit behavior but lets exit 5 propagate to the executor.

C2 — Override marker filter in scripts/run_tests.py

Around line 396 in scripts/run_tests.py, change:

pytest_defaults = ["-q", "--no-header", "--timeout=120"]
pytest_args = pytest_defaults + args.pytest_arg

to:

pytest_defaults = ["-q", "--no-header", "--timeout=120"]
# run_tests.py is the full-suite runner: override pyproject.toml's default
# `-m "not slow and not llm_integration and not incus_live"` so slow tests
# run, while llm_integration/incus_live remain opt-in special lanes.
pytest_args = (
    pytest_defaults
    + ["-m", "not llm_integration and not incus_live"]
    + args.pytest_arg
)

Pytest uses the last non-empty -m expression, so this overrides the addopts filter.

Verification

cd /home/mohamed/workspace/orchestrata-kimi-wt
PYTHONPATH=src:/home/mohamed/workspace/orchestrata-kimi-wt \
  /home/mohamed/workspace/orchestrata-ultimate-architecture/.venv/bin/python \
  scripts/run_tests.py \
    tests/e2e/test_full_pipeline.py \
    tests/llm/test_qwen_minimax_e2e.py \
    tests/orchestrator/test_pipeline_e2e_analysis.py \
    tests/orchestrator/test_pipeline_e2e_docs_session.py \
    tests/orchestrator/test_pipeline_e2e_review.py \
    tests/session/test_session_coverage_gates.py \
    --skip-gates --skip-infra

Expected: all six modules pass (the llm_integration modules as no-tests, test_session_coverage_gates.py with its 5 slow tests actually running).

Bucket D — Memgraph storage contention (2 modules)

Affected modules

• tests/scripts/test_ingest_orchestrata_ultimate.py
• tests/semantic_graph/test_memgraph_adapter.py

Root cause

Multiple parallel workers run CREATE VECTOR INDEX DDL on the shared Memgraph instance. The adapter's retry window (max_storage_retries=8, capped backoff ~6 s, total ~26 s) is shorter than the lock-hold time when snapshots and concurrent writers contend for storage access.

Fix

Increase the retry budget specifically for vector-index creation in src/orchestrata_ultimate/semantic_graph/adapters/memgraph.py.

At line ~369, change:

await self._execute_with_reconnect(_create)

to:

await self._execute_with_reconnect(_create, max_storage_retries=20)

Optionally raise the global default and backoff cap:

# line ~774
max_storage_retries: int = 12

# line ~803
backoff = min(0.25 * (2 ** (storage_attempts - 1)), 10.0)

Verification

cd /home/mohamed/workspace/orchestrata-kimi-wt
PYTHONPATH=src:/home/mohamed/workspace/orchestrata-kimi-wt \
  /home/mohamed/workspace/orchestrata-ultimate-architecture/.venv/bin/python \
  scripts/run_tests.py \
    tests/scripts/test_ingest_orchestrata_ultimate.py \
    tests/semantic_graph/test_memgraph_adapter.py \
    --skip-gates

Bucket E — Latent script/test defects (3 modules)

E1. tests/scripts/test_serialize_js_ts_ast.py — unpinned TypeScript

Root cause

scripts/serialize_js_ts_ast.js installs TypeScript with an unpinned version:

execSync("npm install typescript --no-save --prefix " + tmpDir, ...)

npm now resolves to TypeScript 7.0.2, which is ESM-only. The script loads it with CommonJS require(), so the Node subprocess fails with MODULE_NOT_FOUND.

Fix

Pin to a CJS-compatible version and clear the corrupted cache:

execSync("npm install typescript@5.5.4 --no-save --prefix " + tmpDir, ...)

rm -rf /home/mohamed/workspace/orchestrata-kimi-wt/.ts-tmp

Verification

cd /home/mohamed/workspace/orchestrata-kimi-wt
python -m pytest tests/scripts/test_serialize_js_ts_ast.py -v

E2. tests/scripts/test_bootstrap_graphs.py — .env-coupled test

Root cause

test_effective_env_fills_gaps_from_dotenv expects NEO4J_URI to be backfilled from the real repo .env. The worktree has no .env, so the assertion fails. This is a test-isolation defect independent of the fireflies slice.

Fix

Make _effective_env accept an injectable dotenv_path and rewrite the test to use tmp_path.

In scripts/bootstrap_graphs.py:

def _effective_env(
    env: Mapping[str, str] | None,
    dotenv_path: Path | None = None,
) -> dict[str, str]:
    base: dict[str, str] = dict(env) if env is not None else dict(os.environ)
    dotenv_path = dotenv_path if dotenv_path is not None else REPO_ROOT / ".env"
    if dotenv_path.exists():
        for key, value in dotenv_values(dotenv_path).items():
            if value is not None and key not in base:
                base[key] = value
    return base

In tests/scripts/test_bootstrap_graphs.py:

def test_effective_env_fills_gaps_from_dotenv(cli, tmp_path):
    env_file = tmp_path / ".env"
    env_file.write_text("NEO4J_URI=bolt://dotenv-test:7687\n")
    result = cli._effective_env({"SOME_UNRELATED_KEY": "x"}, dotenv_path=env_file)
    assert "SOME_UNRELATED_KEY" in result
    assert result["NEO4J_URI"] == "bolt://dotenv-test:7687"

Verification

cd /home/mohamed/workspace/orchestrata-kimi-wt
python -m pytest tests/scripts/test_bootstrap_graphs.py -v

E3. tests/scripts/test_benchmark_pipeline.py — LLM provider 402

Root cause

Even with .env present, the default SPIKE_PROFILE=default routes the Concierge volk to deepseek-v4-pro. DeepSeek returns HTTP 402 (Payment Required) with no fallback provider, so call_llm fails after all attempts.

Fix options

1. Operational: Top up the DeepSeek account.
2. Config: Switch the default profile to an OpenRouter-backed profile that currently has credits, e.g.:

   SPIKE_PROFILE=kimi
   

3. Persistent code change: In configs/spike-profiles.yaml, change the default profile's model from DeepSeek to an OpenRouter-backed model. This is a production decision; it changes the default provider for all volks.

Verification

cd /home/mohamed/workspace/orchestrata-kimi-wt
SPIKE_PROFILE=kimi \
PYTHONPATH=src:/home/mohamed/workspace/orchestrata-kimi-wt \
  /home/mohamed/workspace/orchestrata-ultimate-architecture/.venv/bin/python \
  -m pytest tests/scripts/test_benchmark_pipeline.py -v --timeout=180

Recommended implementation order

1. Environment — copy .env and vendor/wheels/grpcio into the worktree.
2. Pins — git cherry-pick d024b289.
3. Runner/conftest — apply C1 and C2 marker-handling fixes.
4. Memgraph — raise vector-index retry budget.
5. Scripts — pin TypeScript, fix bootstrap_graphs isolation, decide on benchmark profile.
6. Full suite — re-run python scripts/run_tests.py --skip-gates from the worktree to confirm green.

Files to modify

Risks and decisions needed

1. Cherry-picking d024b289 adds a test-only commit to the fireflies slice branch. If the plan is to keep the slice branch minimal, the alternative is to rebase the worktree onto master first.
2. configs/spike-profiles.yaml change affects production routing. Confirm with the owner before changing the default profile.
3. Memgraph retry increase slows vector-index failures but does not fix underlying parallel DDL contention. If the suite grows, consider per-worker Memgraph clones (already documented in docs/development-and-testing.md).
4. run_tests.py marker override changes the default suite semantics to include slow tests. Verify CI pipelines expect this.

Applied fixes and verification

All fixes below were applied to the worktree ~/workspace/orchestrata-kimi-wt.

Remaining known issues

• tests/scripts/test_benchmark_pipeline.py::test_main_cli_wires_argv_to_run_all still requires a live LLM provider. With the current SPIKE_PROFILE=default it routes to DeepSeek and receives HTTP 402 (Payment Required). Passing SPIKE_PROFILE=kimi makes the test green. No code change was applied because the fix is operational/config (provider credits or profile default), not a defect in the fireflies slice.

• scripts/code-review.py reports one pre-existing false-positive HARDCODED_URI ERROR on scripts/bootstrap_graphs.py:262:

  uri = env.get("MEMGRAPH_URI", "bolt://localhost:7687")
  

  This is the exact pattern AGENTS.md requires (env lookup with localhost fallback). The rule was not triggered before because the file was unmodified; editing _effective_env caused the file to be re-analyzed. No code change was made because the fallback is correct and required.

Commands run during verification

cd /home/mohamed/workspace/orchestrata-kimi-wt

# Environment
cp /home/mohamed/workspace/orchestrata-ultimate-architecture/.env .env
mkdir -p vendor/wheels
cp /home/mohamed/workspace/orchestrata-ultimate-architecture/vendor/wheels/grpcio-1.82.1-cp314-cp314t-linux_x86_64.whl vendor/wheels/
uv cache clean grpcio
uv sync --dev

# Pins
git cherry-pick d024b289

# Marker handling + slow tests
# (edits to tests/conftest.py and scripts/run_tests.py applied)

# Memgraph contention
# (edits to src/orchestrata_ultimate/semantic_graph/adapters/memgraph.py applied)

# TypeScript pin
rm -rf .ts-tmp
# (edit to scripts/serialize_js_ts_ast.js applied)

# bootstrap_graphs isolation
# (edits to scripts/bootstrap_graphs.py and tests/scripts/test_bootstrap_graphs.py applied)

# Targeted verification
uv run pytest tests/generated_gateway/ tests/postman/test_postman_signatures.py -v
uv run pytest tests/session/test_session_meta.py -m slow -v
uv run pytest tests/scripts/test_gen_level0.py tests/scripts/test_enrich_semantic_graph_specs.py tests/scripts/test_semantic_search.py tests/scripts/test_start_test_services.py tests/scripts/test_sync_architecture_to_neo4j.py tests/scripts/test_serialize_js_ts_ast.py tests/scripts/test_bootstrap_graphs.py -v
uv run pytest tests/semantic_graph/test_memgraph_adapter.py tests/scripts/test_ingest_orchestrata_ultimate.py -v
uv run pytest tests/agents/test_tool_registry.py -v

Full-suite follow-up

A final python scripts/run_tests.py --skip-gates from the worktree was run after all fixes were applied:

[run_tests] ❌ 24,916 tests passed, 5 failed, 0 errors, 0 skipped · 1238/1239 modules in 545s
[run_tests] ❌ FAILED tests/scripts/test_benchmark_pipeline.py

All 5 failures are the DeepSeek 402 issue in test_benchmark_pipeline.py. Verified green with an OpenRouter-backed profile:

SPIKE_PROFILE=kimi uv run pytest tests/scripts/test_benchmark_pipeline.py -v --timeout=180
# 10 passed in 107.49s

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks