Orchestrata Ultimate › Plans › dogfood-notes-2026-07-06

Dogfood notes — Wave 7 (started 2026-07-06, session 204)

Source: plans/dogfood-notes-2026-07-06.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-06 · 260 lines.

Prose pending.

Plan text

Dogfood notes — Wave 7 (started 2026-07-06, session 204)

W7.1 — Working topology (recorded 2026-07-06 22:35)

All endpoints come from .env (no hardcoded addresses in source; per-worker Neo4j fleet keys NEO4J_URI_GW<N> map batched-runner slots to instances).

Preflight command: uv run python scripts/start_test_services.py → ✅ ALL SERVICES READY with all four rows green (kernel no longer skipped).

Supporting incus fleet also running (not part of the app topology): prolog-node-1..3 (.16/.17/.18), janus-test (.15).

W7.2 — Per-module end-to-end runs

(One module at a time; full trace captured under pipeline_output/traces; bugs fixed per fix-code-not-test doctrine; design docs updated where reality diverges.)

Run log

Run 1 — text_slug utility module (2026-07-07 02:36, 445.5s, DeepSeek v4-pro)

Artifacts: logs/e2e-runs/20260707-023640-w72-run1-text-slug/ (app.jsonl 15,372 lines, conversation.jsonl, cxp.jsonl, summary.json). Status reported: completed, final subject implementation_ready.

What worked end-to-end: Cx clarification (asked 1 scoped question, got the scripted path), BA requirements, PM plan, Ux pass-through, Ar full design (architecture/requirements/api_spec/file_tree/class_diagram all written to design/), Junior design review ("VERDICT: READY"), project scaffolding (pyproject, tests/ tree, git init, per-project .venv), CXP packet flow + KG audit writes (CXPPacket nodes + ROUTED_TO edges), session lifecycle.

Findings (ranked):

1. F1 — Junior codegen dead on arrival (root cause): deepseek_coder_driver → call_llm(...) with no volk kwarg (3 sites: :275, :386, :441) → get_volk_model(None) raises LLMConfigError: no volk configured ... get_volk_model has no default fallback. This is exactly backlog #29 / SPEC-028 REQ-028-021/028 / decision D10 ("sentinel implemented, profiles authoritative" — ruled but never executed). ZERO source files were generated.
2. F2 — Fail-open completion: despite F1's hard driver failure, U:Cd's self-review looped 10× on "Let me start by reading all the source and test files..." over an EMPTY file set and the pipeline emitted implementation_ready → run reports completed. A stage whose codegen driver errored + files_created == [] must fail loudly (feed the runner's SPEC-003 retry/escalation classification), never report ready.
3. F3 — U:Ar produced 8 empty responses before the 17,790-char design (8 wasted paid round-trips; retry loop works but the empty-response cause needs a look — likely reasoning-content parsing on deepseek-reasoner).
4. F4 — conversation.jsonl capture records system/user as empty for every exchange (docstring promises full prompts) — capture defect in llm.client._capture_exchange or the runner's env wiring.
5. F5 — Session auto-push warning: session.controller._git attempts a git push with the embedded expired GITLAB_TOKEN → HTTP Basic: Access denied (non-fatal WARNING). Known-bad credential path (glab helper is the working route); auto-push is silently non-functional.

Disposition: F1+F2 dispatched to a builder same-session (D10 is ruled; fail-loud is doctrine). F3/F4/F5 filed as backlog items.

Run 2 — same text_slug intent, post-F1/F2 pipeline (2026-07-07 03:08, 403.5s)

Artifacts: logs/e2e-runs/20260707-030855-w72-run2-text-slug/. Zero ERROR lines in app.jsonl. F1 confirmed fixed: deepseek_coder_driver generated 4 real files (src/core/text_slug.py, tests, package inits) — first run ever to produce implementation output. Generated slugify is competent (NFKD transliteration, collapse, strip, TypeError guard) and its tests pass.

New findings (ranked, all root-caused from conversation.jsonl + design/):

1. F3a — silent tool-budget truncation: U:Ar's design call burned all 10 _call_llm_with_tools turns on file-reading tool calls; the loop returned the truncated mid-thought fragment ("Now let me check if there are any existing source files…", 101 chars) as "the design", no warning.
2. F3b — blank design accepted: _parse_design on that fragment yielded ALL-BLANK sections; _write_design_docs wrote them without erroring.
3. F3c — ungrounded iteration hallucinated: Junior review correctly said QUESTIONS ("sections are all blank") — but _iterate_design (plain call_reasoner, no tools, weak grounding) answered with a canned books-inventory Express/TypeScript API design — a completely different system. All 5 design docs on disk are the books design.
4. F3d — review lacks intent-fidelity: Junior re-reviewed the books design and returned VERDICT: READY for a slugify task — structure was checked, task-consistency was not.
5. F6 — max_length requirement dropped: generated code has no max_length param (spec said 64 + word-boundary truncation). Run 1's BA requirements captured it perfectly (FR-01/FR-05), so the loss is the poisoned/blank design chain, not BA. Expected to heal with F3a-d.
6. F7 — generated pyproject.toml invalid: scaffold uses the raw task SENTENCE as the package name → uv run fails inside the generated project (Not a valid package or extra name). Needs a slugified identifier — ironically the very function under construction.

Disposition: F3a-d dispatched to a builder (tool-budget finalization call, blank-design fail-loud retry, tool-loop-grounded iteration, intent-fidelity review instruction). F7 to backlog. Run 4 (same intent) after the fix lands; requirement-fidelity (F6) re-checked there.

Run 3 — analysis-only intent, kernel relay (2026-07-07 03:21, 460s)

Intent: Redis streams vs pub/sub trade-off analysis for the volk event mesh, "no code changes". Artifacts: logs/e2e-runs/20260707-032136-w72-run3-analysis-streams/.

F8 — work-type misrouting (harness-level): the explicit analysis intent ran the FULL FEATURE chain — BA said PROJECT_TYPE: refactor, Ux invented "DocumentViewer" screens, Junior scaffolded conftest/tests, final subject implementation_ready — for a task that forbade code. Root: scripts/ e2e_run.py drives the kernel PipelineRunner relay, which has no work-type concept (the D1 dual-runner state); the work-type router (WorkType.ANALYSIS → Cx→BA→Cp) lives in the orchestrator PipelineRunner. Upside: Ar's 20K-char output after 7 grounded tool-read turns (no truncation — budget sufficed this time) contains genuinely codebase-aware analysis material.

Fix applied same-session: e2e_run.py --work-type — non-feature values drive the orchestrator runner + PipelineRouter (mirrors the LLM-lane e2e tests); artifacts written per stage to the run dir. Run 3b re-submitted the same intent via --work-type analysis. The deeper NL-intent→work-type classification in the LIVE path remains the Wave-3/E consolidation item (orchestrator intent path, 0 prod construction sites).

Runs 3b/3c — analysis route (2026-07-07 03:32 / 03:35)

• 3b (29s, ESCALATED): correct-by-design suspension — Concierge asked a clarification and the run checkpointed ESCALATED after U:Cx (D1-addendum-2 semantics, matches backlog #71). Harness lacked the answer step; added an escalation respond() loop to the routed path (committed).
• 3c (163s, RESOLVED): full route Cx→BA→Cp green, 1 escalation round answered mid-pipeline, per-stage artifacts captured. Cx's requirements spec for the analysis doc is outstanding (structure, audience, delivery-semantics criteria, assumptions section, quality gates).
• F9 — work-type semantics gap: analysis = requirements decomposition; it never AUTHORS the requested analysis document (no deliverable file). Deliverable-shaped asks must route to documentation (U:Do). Backlogged with the Wave-3/E intent-classification consolidation; BA's PROJECT_TYPE also lacks analysis/doc labels (said refactor twice).
• F10 — artifact extractor drift: task_summary (Cx) and user_stories (BA) artifacts extracted EMPTY from 4.8K/5K-char stage responses — section-format drift between extractor patterns and current volk output formats. Backlogged.
• 3d dispatched: same analysis ask via --work-type documentation (Cx→Do→Cp) — the route whose deliverable IS an authored document.

Run 3d — documentation route (2026-07-07 03:39, 157s, FAILED — correctly)

Cx resolved (another excellent requirements capture), U:Do returned 10 empty-content exchanges — the same F3a tool-budget truncation (Documentor uses _call_llm_with_tools, volk.py:112) — and _handle_documentation RESOLVED over an empty deliverable (F3b-class fail-open, second volk). U:Cp then FAILED the run — the compliance net caught the empty deliverable. First run where a downstream stage correctly refused a hollow upstream result.

Disposition: confirmed the truncation/fail-open is a defect CLASS, not an Architect one-off. In-flight F3 builder's base.py finalization fix covers U:Do's truncation automatically (shared loop); its brief EXTENDED same-session with the Documentor empty-output fail-loud guard + test. Re-run the doc route after merge.

F11 — no web-research capability (by construction): tool_registry has no web_search/web_fetch tool and run_shell BLOCKS curl/wget. Tasks needing current external facts run on training data alone, silently. Backlogged as a decision (governed web-fetch tool vs documented limitation); behavioral probe queued post-merge.

Runs 3e + 4 — post-F3-merge verification (2026-07-07 03:56, parallel)

• 3e (doc route, 228s, FAILED-correctly): the D1 budget warning FIRED and the finalization call ran — but DeepSeek v4 ignored tool_choice="none" with the tools schema still in the request and emitted raw DSML tool-call markup as text (<｜｜DSML｜｜invoke name="read_file">…), which passed the "non-empty" guard; U:Cp failed the run again. F12: OpenAI-compatible adapters must omit tools+tool_choice entirely when tool_choice=="none" (anthropic_adapter already does) + DSML-markup sanitizer backstop in the finalization path. Builder dispatched.
• 4 (feature re-run, 358s): fail-loud chain WORKED — no books design, no hollow implementation_ready; the failure surfaced to HUMAN. But Ar again couldn't compose within budget: 6 exploration turns, then the finalization call returned another mid-thought fragment (same F12 class). F13: max_turns=10 is tight for grounded design (real exploration eats 6-9 turns) — made configurable, Ar design call gets 16 (folded into the F12 builder).
• max_length fidelity (F6) still unverified — needs Run 5 after F12/F13 land.

Runs 3f + 5/5b — full-stack verification (2026-07-07 04:41-04:52)

• 3f (doc route, 122s): breakthrough — U:Do AUTHORED the full 16,640-char analysis (5 tool turns then compose; finalization/adapter stack works). Remaining break = F14: _parse_output extracts one-line metadata and DISCARDS the CONTENT body; zero write_file calls; nothing persists the doc; Cp fails the missing deliverable. Builder dispatched. (F10 "empty artifacts" partially RETRACTED — wc -l illusion on no-trailing-newline files; doc_output really carried its 427-byte summary.)
• 5 (feature, concurrent with 3f) + 5b (solo): identical crash both times: [CRASH] U:Ar.process() failed: call_with_tools(volk=U:Ar) failed after 6 attempts: No eligible provider found for request after ~9 heavy design turns → kernel run shutdown. F16 root-caused live: ProviderConfig.rate_limit_tpm defaults to 100_000; DeepSeek's registration never overrides it; Ar's tool loop resends the accumulated conversation (persona + graph context + monorepo tree + tool results) each turn → TPM sliding window exhausts → route() → None → opaque ProviderError → crash. Same loop with small prompts succeeds end-to-end (probe: 6 turns + finalization → complete 13K design — D1/F12/F13 all observed working: budget warning fired, finalization composed). Breaker is NOT fed by refusals (checked) — pure throttling surfaced as a crash. Builder dispatched: explicit TPM limits (deepseek/openrouter 1M, anthropic 400K), limiter refusals wait-for-window-reset once instead of raising, and reasoned refusal messages ("rate limited on deepseek — resets in Ns").
• F15 (side-find): kernel session controller writes sessions/context.json INSIDE src/orchestrata_ultimate/kernel/ — runtime state in the package tree. Backlogged.

Runs 3g/3h/3i — doc-route convergence (2026-07-07 05:04-05:26)

• 3g: doc_output artifact now carries the full document (10K) — but the write was refused: F17 — harness exported ORCHESTRATA_WORKSPACE as a cwd-RELATIVE path; _persist_doc_content compared resolved target vs unresolved base → every write "escapes the workspace". Fixed inline both sides (resolve before prefix check; harness exports absolute) + regression test.
• 3h: the deliverable exists — 25,452-byte real technical analysis persisted to workspace/docs/redis-event-mesh-analysis.md (metadata table, structured sections). Cp still failed silently → F18 root-caused: CAPABILITY_SUBJECTS[(DOCUMENTATION, TRACEABILITY)] was "documentation_complete" — a subject ComplianceVolk never implemented (handles only compliance_check; returns None otherwise) → silent stage failure by construction on every doc run. Fixed inline: mapping → compliance_check + runner WARNING naming stage/subject on any None response (kills the silent-failure class).
• 3i: doc persisted again (23,561B); stage-retry classification fully observable now (attempt warnings + backoff). U:Do stage FAILED on attempts 2-3 with the F16 TPM starvation ("No eligible provider") — full-conversation retries drain the 100K default TPM window. All roads now lead to the in-flight F16 fix (explicit TPM, wait-for-window, reasoned refusals). Doc route re-verifies after its merge.

Post-F3-merge run matrix (user-directed campaign, 2026-07-07)

All feature/Ar-dependent runs blocked on the F3a-d merge. Planned:

1. Run 4 — text_slug feature re-run: design-chain heal check + F6 (max_length fidelity) verification.
2. Doc route re-run — the Redis streams analysis via documentation (3d retry; U:Do authored deliverable expected).
3. Refactor — --work-type refactor (Ar→Cd→QA→Cp) on seeded existing code via WORKSPACE_DIR (volk workspace = env at harness import).
4. Web app/middleware/backend — small FastAPI service + auth middleware
   • persistence (feature route, multi-file).
5. Web-research probe — a task requiring current external API facts; expected to expose F11 behaviorally (hallucinate vs flag inability).
6. Build-and-use a tool — Redis client wrapper built then consumed by a feature in the same intent.
7. Tests-to-90% — --work-type test (Cd→QA) against seeded module.
8. Self-hosting — an orchestrata spec cluster as the intent (candidate: a small SPEC-014 honest-gap REQ), the "can it build itself" probe.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks