Home / Projects / Orchestrata Ultimate / Decisions / ADR index / ADR-0074

ADR-0074: The prod promote re-embeds only changed semantic chunks and probes the embedding provider before the tree moves

← ADR-0073 · ADR-0075 →

The record

Status: Accepted (ruling, Mohamed, 2026-10-07: "adr 0074 agreed") · Date: 2026-10-07

Context

promote_env.py promote --to prod ends with a graphs stage (src/orchestrata_ultimate/promotion/stages_graphs.py:25-82) that runs bootstrap_graphs.py --all from the deployed tree: the operational-KG chain (schema, AST full sync, architecture sync, spec enrichment) and then the Memgraph step ingest_orchestrata_ultimate.py, which chunks the source tree and embeds every chunk through OpenRouter (mistralai/codestral-embed-2505, dim 1536, label CodestralChunk). The stage is part of the promote's success gate: a FAIL rolls the tree back and restarts the units on the previous commit (promotion/runner.py).

Measured on 2026-10-07 (plans/ast-daemon-deployment-plan.md § 3b, rows ab189ccc7 and c8c4ddd23; logs p6a_ship3.log, p6a_ship4.log):

• The ingester is not incremental. SemanticGraphIngester.ingest_directory (semantic_graph/ingestion.py:271-313) chunks the whole tree and calls embedder.embed on every batch, every run. Chunk ids are positional (CodeChunker.chunk_file, ingestion.py:134-199: _chunk_id(prefix, path, start, end)), so an unchanged file produces the same ids and is re-embedded anyway; no content hash is stored on the node, so nothing can tell "same text" from "changed text".
• One promote re-embeds 5,740 *.py chunks plus the *.pyi/*.md/other patterns: 570 embedding POSTs, 5 min 20 s of the stage's 835.6 s on the ab189ccc7 promote. The source changes those promotes carried were two shell scripts and plan rows; the embedded text was identical to the run before.
• On the c8c4ddd23 promote the 58th embedding POST returned HTTP 402 Payment Required; the stage failed at 449.8 s, AFTER services had restarted the four prod units on the new tree and verify had passed, so the promoter rolled back and restarted them again on ab189ccc7. The OpenRouter account read total_credits 1148, total_usage 1148.22 (balance exhausted) from the prod key immediately afterwards; usage_daily 1.39.
• bootstrap_graphs.py already distinguishes --kg, --memgraph, --all; the ingester already has a free --backend local (SentenceTransformer) seam. The promoter uses neither: --all is a literal in stages_graphs.py.
• The infra stage probes Redis and the graph cluster before the tree moves. It does not probe the embedding provider, which the promote depends on just as hard.

The forcing fact is the last one combined with the first: a metered external API that the promote calls ~570 times per ship, on text that did not change, decides at the very end whether a deploy that has already passed verify stays or rolls back.

Decision (proposed)

1. The semantic ingest becomes incremental by content hash. Every chunk node carries content_sha256 (hex of the chunk text). ingest_directory reads the existing (id, content_sha256) pairs for the project before embedding, embeds only chunks whose id is new or whose hash differs, and deletes chunk nodes under the project whose id no longer exists in the tree. A promote whose ship changed no chunked file performs zero embedding calls. The first run after Accepted re-embeds everything once to record the hashes.
2. The infra stage probes the embedding provider before the tree moves. With the same key the deployed tree will use, it reads the provider's credit/limit endpoint (for OpenRouter GET /api/v1/credits) or, where no such endpoint exists, embeds one short string; a 401/402/403, or a balance that cannot cover the estimated run (changed_chunks × per-call cost, zero when nothing changed), is [FAIL] infra embedding provider: <status> and the promote is refused before any unit is restarted. No rollback is needed because nothing moved.
3. graphs keeps --all. "Promoted to prod" continues to mean both graphs are derived from the deployed commit; the stage simply stops paying to re-derive text it already holds.

Alternatives rejected

• Take the Memgraph ingest out of the promote's success gate (promote runs --kg; a timer re-embeds later). Removes the dependency but also the guarantee: the semantic graph would lag the deployed code between timer runs and nothing would refuse a promote whose chunks never land. The reproducibility plan's R4 rule ("derive graph state from source, never hand-build") is only enforced if the promote checks it.
• --backend local in prod. Free, but a different embedding space and dimension from the 1,536-d Codestral vectors every reader (semantic_search.py, the context-pack retriever, wiki-rag) indexes today; it is a retrieval-quality decision with no measurement behind it, not a cost fix.
• A --skip-semantic flag on ship.sh. A step people skip stops being a step (tests/deploy/test_ship_master.py, the --no-promote rationale). --skip-ingest already exists for the 31-project workspace ingest and is justified there by its 9-minute serial cost, not by an external dependency.
• Top up credits and change nothing. 570 paid calls and 5+ minutes per promote for identical text, and the next lapse again fails a verified deploy at the end and rolls it back. The balance has to be topped up for the next promote regardless of this ADR (item 2 will refuse until it is); that is an operator action, not a design.

Consequences

On Accepted: packet P6c of plans/ast-daemon-deployment-plan.md builds items 1–2 against semantic_graph/ingestion.py, semantic_graph/adapters/memgraph.py (read existing hashes, delete stale ids), promotion/stages_resolve.probe_endpoints + promotion/env_profile.probe_infra (provider probe) and promotion/stages_graphs.py (unchanged --all), with real-Memgraph tests on the test fleet (10.53.2.10) proving: an unchanged tree embeds nothing; a one-line edit re-embeds exactly the chunks covering that line; a removed file's chunks are gone; a 402 from a local HTTP stand-in is refused at infra with no unit restarted. A no-change promote's graphs stage drops by ~5.5 min; a credit lapse surfaces in seconds at infra with the instruction to top up, instead of after a restart-and-rollback cycle. The Definition of Done (ADR-0069) applies: @traces, RTM, docs (docs/ship-and-deploy.md), wiki, ≥90% coverage, one ship. Until Accepted, prod stays at ab189ccc7 and the next promote needs credits on the account; nothing in /opt/orchestrata needs the two commits master is ahead by (both are ship-side).

Provenance

Orchestrator finding F-P6a-2, 2026-10-07, from the c8c4ddd23 ship (p6a_ship4.log: [FAIL] graphs 449.8s bootstrap failed: … VectorSearchError: OpenRouter embeddings HTTP error 402; [OK ] rollback returned to ab189ccc722f and restarted 4 unit(s)), the ab189ccc7 ship (p6a_ship3.log: 570 embedding POSTs 06:22:11–06:27:40, [OK ] graphs 835.6s), and the OpenRouter /api/v1/credits reading taken with the prod key right after the failure (total_credits 1148, total_usage 1148.22191807). Code: src/orchestrata_ultimate/semantic_graph/ingestion.py:134-199, 271-313; src/orchestrata_ultimate/promotion/stages_graphs.py:25-82; scripts/bootstrap_graphs.py --help; scripts/ingest_orchestrata_ultimate.py --help. Awaiting Mohamed's ruling; only he rules.

• Ruled Accepted by Mohamed on 2026-10-07, verbatim: "adr 0074 agreed". Execution: packet P6c of plans/ast-daemon-deployment-plan.md builds items 1–2 (incremental semantic embed by content_sha256, embedding-provider probe at infra) with the Consequences' real-Memgraph tests. Prod stays at ab189ccc7 until a promote runs; the OpenRouter top-up remains an operator action, not a design change.

Related ADRs

• Superseded by: ADR-0076
• Also mentioned: ADR-0069

Rulings cited

None.

Specs cited

None (or no spec page exists for the ones named).