Orchestrata Ultimate › Plans › e2e-deepseek-test-results

E2E DeepSeek-v4-pro Dogfood — Test Results

Source: plans/e2e-deepseek-test-results.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-05-27 · 121 lines.

Prose pending.

Plan text

E2E DeepSeek-v4-pro Dogfood — Test Results

Companion to e2e-deepseek-test-plan.md. Records the actual runs of the orchestrata pipeline driven on deepseek-v4-pro via scripts/e2e_run.py, with full real-time capture (volk conversations, CXP flow, structured logs, errors) under logs/e2e-runs/<ts>-<slug>/.

Last updated: 2026-05-28

Outcome summary

The end-to-end pipeline runs to a clean terminal on deepseek-v4-pro and produces real, compiling code — once five defects (none caught by the unit suite) were fixed. Every LLM exchange is deepseek-v4-pro; CXP packets are audited to Neo4j; conversations + logs + errors are captured per run.

Bugs found by the dogfood test (all fixed + committed)

These only surface with a live, loop-bound async provider driven through the synchronous CXP router — invisible to the unit suite.

Commit map: 4138a64c = bugs 1–3 + capture hook + harness + plan; 82afac04 = templates (#4); b82de5b9 = harness path (#5); 9bf9fcfb = retry backoff (#6).

Scenario results

S2 — first attempt (logs/e2e-runs/20260528-003004-newproj-jsonschema/)

Generated 7 real, compiling modules (cli.py, validator.py, models.py, reporter.py, exceptions.py, decorators.py, all deepseek-v4-pro), but U:Cd crashed on a 429 burst (25× 429) before implementation_ready, so the harness maxed its clarification rounds. This exposed bug #6; re-running with the longer backoff.

S1 — logs/e2e-runs/20260528-000325-docs-kernel/

• Status: completed, final_subject: implementation_ready (2 human rounds: clarify → implementation_ready). Duration ~22.6 min.
• Provenance: 29 LLM exchanges, 100% deepseek-v4-pro (19 driver code-gen
  • Cx×2/BA/PM/Ux/Ar×3/Junior×2). Full CXP flow audited to Neo4j.
• Quality gates: 0 CRASH, 0 "Event loop is closed", 0 "No module named scripts". 1 ERROR = a single transient driver API call (All providers failed after 1 attempts); the run retried and recovered.
• Output: a real kernel_developer_docs tool — models.py, inspector.py, flow_describer.py, writer.py, linter.py, cli.py (+ tests/fixtures) and 5 design docs. All generated source compiles (python -m py_compile ✓), with dataclasses, docstrings, and type hints.
• Pipeline path: HUMAN→Cx→BA→PM→Ux→Ar→Junior→Cd→(driver implement/review loop).

Key findings

1. The volk pipeline is a greenfield project builder. Every intent — including "document the kernel" — is interpreted as "scaffold and implement a new project that does X." It does not edit an existing codebase in place. In-place editing of this repo would require a different driver mode (e.g. the aider / claude-code driver pointed at a worktree), not this Cx→…→Cd chain. The plan's "S2: edit existing code" is therefore reframed as a greenfield build.
2. v4-pro latency dominates runtime (~30–90 s per reasoning call; the implement→review loop ran 19 driver iterations). A full run is ~15–25 min.
3. Logging/observability is complete — every provider round-trip is captured to conversation.jsonl (verified: actual completion count == captured count), CXP to Neo4j + cxp.jsonl, structured logs to app.jsonl, errors isolated.
4. The harness reaches a clean terminal once the pipeline can complete — the earlier "maxed-out clarification rounds" was a symptom of the downstream crashes (#4, #5), not a harness bug; with those fixed, S1 ended at implementation_ready in 2 rounds.

CXP-over-Redis verification

Verified: the volks communicate using the CXP protocol, and it is watchable on Redis (db0 @ 10.92.53.10). Evidence:

• Protocol — every packet persisted to Redis is a full CXP v1.0.0 envelope: message_id, from_agent, to_agent, from_instance, to_instance, thread_id, subject, payload, timestamp, priority, delivery_status, work_status, protocol_version, packet_type, ttl_seconds, header.
• Volk-to-volk flow (decoded live from cxp:thread:<id>): HUMAN→Cx(intent) → Cx→BA(analyze_requirements) → BA→PM(scope_review) → PM→Ux(ux_design) → Ux→Ar(implement) → Ar→Junior(design_review) → Junior→Ar(design_feedback) → Ar→Cd(implement).
• Redis layout (postman/sync_router.py, postman/postman.py):
  • cxp:thread:<thread_id> (list) — per-conversation CXP audit trail (RPUSH per packet).
  • postman:q:<volk> (list) — per-volk delivery inbox (LPUSH/RPOP).
  • cxp:dedup:seen (set) — message-id dedup.
• Real-time — a no-LLM probe (Postman.route_sync of one packet) wrote to cxp:thread:* + postman:q:* in 26 ms and was immediately readable.

Watch commands:

redis-cli -h 10.92.53.10 MONITOR | grep -E 'cxp:thread|postman:q|cxp:dedup'  # live writes
redis-cli -h 10.92.53.10 LRANGE cxp:thread:<thread_id> 0 -1                  # a conversation
redis-cli -h 10.92.53.10 --scan --pattern 'postman:q:*'                       # volk inboxes

How to reproduce / inspect

cd /tmp/oua-master
PYTHONUNBUFFERED=1 ./.venv/bin/python scripts/e2e_run.py \
    --intent "<intent>" --slug <label> --max-rounds 10
# artifacts:
ls logs/e2e-runs/<run>/   # conversation.jsonl cxp.jsonl app.jsonl summary.json console.log workspace/
tail -f logs/e2e-runs/<run>/conversation.jsonl | jq .

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks