Orchestrata Ultimate › Plans › e2e-deepseek-test-plan

E2E DeepSeek-v4-pro Dogfood Test Plan

Source: plans/e2e-deepseek-test-plan.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-05-27 · 148 lines.

Prose pending.

Plan text

E2E DeepSeek-v4-pro Dogfood Test Plan

1. Objective

Prove the full intent → orchestration → output loop works on a real LLM with real infrastructure (Redis + Neo4j), and that we have provenance of exactly what happened — no simulation, no hidden fabrication (the lesson behind the VizInt provenance gates). Every LLM exchange, every CXP packet, every gateway message, and every error must be captured to durable, tail-able files in real time.

2. Model configuration (DONE + live-verified)

• All volks pinned to deepseek-v4-pro for this campaign via env override (ORCHESTRATA_LLM_PROVIDER=deepseek, ORCHESTRATA_LLM_MODEL/_REASONING_MODEL=deepseek-v4-pro).
• The DeepSeek API key (DEEPSEEK_API_KEY) is present and serves exactly deepseek-v4-pro + deepseek-v4-flash (verified via /v1/models).
• Stale deepseek-chat/deepseek-reasoner ids (now dead) were replaced with the v4 ids in llm/client.py (adapter registration), llm/config.py and core/infra_config.py (PROVIDER_MODELS, defaults). Default tiers corrected to standard=deepseek-v4-flash, reasoning=deepseek-v4-pro (per SPEC-102 FR-003).
• Live round-trip through the real call_llm path confirmed: deepseek-v4-pro → OK in ~3s.

3. Architecture under test

intent ──> PipelineRunner.submit_intent()
              │   (Postman.route_sync — synchronous CXP routing)
              ├─> U:Cx Concierge ─ clarify intent (human-in-the-loop phase)
              ├─> U:BA / U:PM / U:Ar / U:Cd / U:Qa / J:Junior / U:Ux / U:Cp
              │     each volk: call_llm(...) -> deepseek-v4-pro
              └─> drivers (deepseek_coder / aider / claude_code) write files
       CXP packets ──> Neo4j (CXPAuditor) + Postman.message_log + Redis thread audit

Two-phase human model: an intent/clarify phase (Concierge asks questions; the harness plays the human with scripted answers) followed by autonomous execution watched through the logs (and, in gateway mode, the VizInt portal).

4. Observability design (the core deliverable)

scripts/e2e_run.py creates a per-run directory logs/e2e-runs/<ts>-<slug>/:

Unbuffered / real-time: PYTHONUNBUFFERED=1; Python's StreamHandler/FileHandler flush per record; _capture_exchange and _Tee flush on every write. Tail live: tail -f logs/e2e-runs/<run>/conversation.jsonl | jq .

Reconciling with SPEC-100: the spec wants non-blocking/async logging for high-throughput production. This is a watched dogfood test, so live visibility wins; full prompt/response capture is opt-in via env (off by default → respects FR-009 "no PII without consent"). CXP audit to Neo4j (CXPAuditor) and gateway audit (AuditWriter) already exist and are reused, not reinvented.

Gateway messages: the in-process driver does not traverse the WebSocket gateway. To capture gateway traffic (SPEC-014), run the same intent through kernel serve and connect the VizInt portal / a JSON-RPC client; gateway/server.py already logs inbound messages + RPC timing and audits connections. Gateway mode is phase 2 of this plan.

5. Tool augmentation for Coder + Architect (context-injection)

Decision: context-injection first (no agentic tool-loop yet — none exists in the volks). Before a Coder (U:Cd) or Architect (U:Ar) LLM call, the runtime gathers real codebase facts and injects them into the prompt as a <codebase_context> block:

• RLM: load the fresh plans/ast-ultimate.json (regenerated 2026-05-27: 518 modules, 1066 classes, 4261 methods, 143K LOC) and query it in Python for the symbols/classes/dependencies relevant to the task.
• CLI tools: orchestrata-discover (ast_explorer), orchestrata-diagram, ast-query — their output is injected as context.

The model reasons over verified AST facts instead of hallucinating structure. This is deterministic, low-risk, and faithful to the current no-tool-loop design; it can evolve into a DeepSeek function-calling loop later. (Implementation pending — phase 2.)

6. Scenarios

Order: S1 → S2 → S3 (ascending blast radius). S2/S3 require the tool-augmentation (§5) and a worktree-isolated workspace.

7. Success criteria

Per-run (deterministic, no extra LLM calls — mirrors SPEC-102 FR-008 evaluators):

1. Boots — 9 volks registered, providers registered (deepseek with 2 models).
2. Completes — terminal subject reached (implementation_ready / qa_complete) or a clean, explained stop; no unhandled exception.
3. Provenance — conversation.jsonl shows every exchange used deepseek-v4-pro (not claude/fallback); cxp.jsonl + Neo4j show the packet flow; app.jsonl has no ERROR/CRITICAL that aren't explained.
4. Output — S1: Markdown files exist in the scratch workspace; S2: a diff applied in the worktree; S3: a runnable project scaffold.
5. Quality (S2/S3) — syntax_check (ast.parse) + lint_check (ruff) clean on generated Python; optionally an llm_judge (deepseek-v4-pro, temp 0) score.

8. Safety / isolation

• File writes contained to a scratch ORCHESTRATA_WORKSPACE (S1) / git worktree (S2) / /tmp dir (S3) — never the live repo/master.
• DeepSeek key is read from env only, never logged (capture records messages, not keys).
• Command-whitelist + git-push-prevention (SPEC-102 SEC-001/004) apply to S2/S3 driver runs.
• Real shared Neo4j: runs add CXPPacket/audit nodes (no destructive ops).

9. How to run

cd /tmp/oua-master
PYTHONUNBUFFERED=1 ./.venv/bin/python scripts/e2e_run.py \
    --intent "<intent text>" --slug <label> [--model deepseek-v4-pro] [--max-rounds 6]
# watch live:
tail -f logs/e2e-runs/<run>/conversation.jsonl | jq .
tail -f logs/e2e-runs/<run>/app.jsonl | jq 'select(.level=="ERROR" or .level=="WARNING")'

10. Open items / risks

• Tool-augmentation (§5) not yet implemented — S1 (docs) does not strictly need it; S2/S3 do.
• Clarify-phase scripting is generic ("proceed autonomously"); real runs may need scenario-specific answers if the Concierge asks pointed questions.
• spike-profiles.yaml not found on the walk-up path → falls back to defaults (non-fatal; env override still applies). Consider placing it under configs/.
• Gateway capture is phase 2 (requires kernel serve + a client/portal).
• v4-pro latency (~3s+/call) × many volks × clarify rounds → a run can take minutes.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks