Orchestrata Ultimate › Plans › e2e-live-trace-plan
E2E live-trace plan — whole-system dogfood with tracing & debugging
Source: plans/e2e-live-trace-plan.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-12 · 178 lines.
Prose pending.
Plan text
E2E live-trace plan — whole-system dogfood with tracing & debugging
User directive 2026-07-12 (session 227): after the in-flight work lands, test the WHOLE system end to end. Mohamed supplies the intents (live, via the UI and/or CLI); Claude traces, logs, debugs and records every issue or error in a findings ledger. Fix-everything rule applies: defects are recorded first, then fixed (blocking defects fixed immediately; the rest in post-session waves with regression tests).
Goal
Prove — or break — the full production path with real human intents:
VizInt UI (browser, ws) kernel CLI / e2e_run.py (in-process)
│ │
▼ ▼
WebSocketGateway (auth → RPC dispatch → handle_intent, pre-subscribe pipeline.<thread>)
│
├── gRPC volk kernel (CXP_GRPC_HOST/PORT, health-probed, preferred)
└── PipelineKernelClient (in-process fallback; ORCHESTRATOR handler only)
▼
IntentBroker → parse / validate / security / governance (Prolog gate, fail-closed)
▼
PipelineRunner → stage volks (U:Cx→U:BA→U:PM→U:Ar→U:Cd→J:Junior→U:Qa→U:Cp per work type)
│ │ LLM round-trips (conversation log)
│ │ tool calls (ToolExecutor + SPEC-062 native tools — NEW this session)
▼ ▼
Postman/CXP routing KG writes (Session, CXPPacket, audit, ToolCall)
▼
broadcast(pipeline.<thread>) → UI gadgets (CXPFeed / Orchestrator / DebugPanel)
Every hop above is a checkpoint with named evidence (§4). An intent that "works" but skips a hop silently (e.g. governance not consulted, KG write missing, UI event dropped) is a finding, not a pass.
1. Roles
| Who | Does |
|---|---|
| Mohamed | Supplies intents interactively (UI first; CLI for the in-process comparison runs). Decides product questions surfaced mid-session. |
| Claude | Runs the rig: starts/verifies the stack, tails all telemetry, traces each intent hop-by-hop, debugs live, records every finding in the ledger, files fixes. Never blocks the session on a question — record and move on. |
2. Pre-flight (gate — do not start the session without all green)
- In-flight work landed: volk tool exposure + orphan-module ratchet reviewed, gated, committed; full suite green as the baseline (last authoritative: 24,744 passed / 1,234 modules / kernel gRPC-verified).
python3 scripts/start_test_services.py— all 4 required services (Redis 6379, Neo4j 7687 @10.92.53.11, Gateway 8765, Kernel gRPC 50053) started + verified. Kernel is REQUIRED — no--skip.python3 scripts/health_check.pyclean (includes the new Rule-8 orphan ratchet once landed).- UI:
cd vizint-ui && npm run dev(http.server :5000), browser toindex.html,localStorage['vizint.gateway.url']→ws://localhost:8765, token inlocalStorage['commandcenter.token'](demo token validates againstdev-secret; note the dev-permissive auth caveat §7). - Run directory created:
logs/e2e-live/<date>-session/— all captures below write under it. - Baseline KG counts snapshotted (Session/CXPPacket/AuditEntry node counts
via
scripts/neo4j_query.py) so per-intent deltas are measurable. - LLM provider: RESOLVED 2026-07-13 — run the session under
SPIKE_PROFILE=kimi(user ruling; set on gateway + kernel processes at startup). Original blocker for the record: BLOCKER found 2026-07-12: the DeepSeek account returns HTTP 402 Insufficient Balance, and thedefaultprofile pins every volk to DeepSeek, so the whole pipeline is LLM-dead on it. Mohamed decides before the session: top up DeepSeek, OR run the session underSPIKE_PROFILE=kimi|qwen|budget|mix|claude(OpenRouter key verified live 200 with credit; profile choice changes per-volk models — seeconfigs/spike-profiles.yaml). The committed default profile is NOT changed unilaterally.
3. The observability rig (started before the first intent)
| Channel | Source | How captured |
|---|---|---|
| Structured app logs (SPEC-100) | all loggers, gateway, CXP route logs | ORCHESTRATA_LOG_JSON → app.jsonl; tail -f | jq filtered per thread_id |
| Volk↔LLM conversations | llm.client._capture_exchange |
ORCHESTRATA_CONVERSATION_LOG=<run>/conversation.jsonl set on gateway+kernel processes |
| CXP packet flow | Postman log | cxp.jsonl filtered by thread_id |
| Gateway ws frames | UI side | browser devtools ws tab + DebugPanel gadget (listens on ws.message bus) |
| KG audit trail | Neo4j @ .11 | scripts/neo4j_query.py / scripts/graph_browser.py — Session, CXPPacket, AuditEntry, ToolCall per thread |
| Redis state | queues, ring buffer, intent lifecycle | redis-cli -h 10.92.53.10 monitor (scoped bursts only) + targeted KEYS/XRANGE per intent id |
| Spans/tracing | core.monitoring.spans, core.tracing |
enabled via config; span dump per intent appended to the trace file |
| Process stderr | gateway + kernel | each service started with stderr teed to <run>/<svc>.log |
Debug tools on standby: scripts/serialize_codebase_ast.py debug (AST-precise
breakpoint insertion), scripts/diagnose_test_failure.py (7-step diagnostic),
scripts/observe_pytest.py (for any regression test written mid-session).
4. Per-intent trace protocol (repeated for EVERY intent Mohamed submits)
Assign a serial INT-NNN. Then walk the checkpoints; each one is
pass / fail / silent-skip (skip = finding):
- Ingress — ws frame received; auth outcome; JSON-RPC method + params logged. Evidence: app.jsonl gateway records, DebugPanel.
- Dispatch —
handle_intentran; caller pre-subscribed topipeline.<thread_id>BEFORE dispatch; which kernel path was taken (gRPC vs in-process fallback — record which; the health probe decides). - Broker — intent parsed → IntentType; validator verdict; security check; governance: the Prolog gate consulted and its allow/deny + rule recorded (fail-closed — an unconsulted gate is a CRITICAL finding).
- Pipeline — work type routed; stage list; per-stage volk, model, round-trips, tool calls (incl. the newly exposed native tools — repl/lsp/ worktree/task/cron/MCP: any volk invoking one is traced end-to-end through ToolExecutor's native bridge), retries/escalations.
- Messaging — CXP packets per hop: ids, subjects, priority, delivery receipts; postman routing decisions.
- Persistence — KG delta vs baseline: Session node, CXPPacket nodes, audit entries, ToolCall events; project-qualified ids correct (multi-project invariant); Redis intent lifecycle records.
- Egress —
broadcast(pipeline.<thread>)events observed in UI; gadgets rendered them; final response surfaced to Mohamed; latency ingress→egress recorded. - Verdict — outcome (result quality is Mohamed's call), plus a trace-completeness score: N/7 checkpoints evidenced.
Record everything per intent in <run>/intents/INT-NNN.md (intent text,
thread id, checkpoint table, timings, evidence file:line pointers).
5. Findings ledger — plans/e2e-live-trace-findings.md
One row per defect, appended live, never edited away:
| ID | Intent | Subsystem | Sev | Symptom | Evidence | Root cause | Fix |
- Sev: CRITICAL (wrong/lost result, governance bypass, crash) / MAJOR (degraded path, silent fallback, missing persistence) / MINOR (log noise, latency, cosmetic UI).
- Evidence = exact file paths + line/offset into the run captures, or KG query.
- Fix column tracks:
open → diagnosed → fixed@<commit> + regression test. - Blocking-the-session defects are fixed immediately, in-session; everything else in post-session fix waves (each fix carries a regression test, gated, committed per the review-before-commit rule).
6. Intent menu (suggestions — Mohamed picks/free-forms; aim to cover every work type + the new tools)
- Analysis: "Analyze the tools package and summarize its architecture" (Cx→BA→Cp path).
- Documentation: "Generate developer docs for the kernel package" (documentor + validate_mermaid + MCP-resource reads).
- Review: "Review the agents/tool_registry.py module for defects"
(QA path — should exercise
repl/lspnative tools). - Code: "Add a --json flag to scripts/flake_report.py" (full Cd→Junior→QA loop; worktree + task tools).
- Refactor: "Refactor X by cohesion" (SPEC-103 REFACTOR intent routing).
- Adversarial: an ambiguous intent, an intent that should be governance- DENIED, an oversized/garbage intent (security path), a duplicate intent (idempotency), and a mid-pipeline client disconnect (recovery).
- Comparison run: one identical intent via
scripts/e2e_run.py(in-process) vs the UI (gateway+gRPC) — diff the traces; divergence is a finding.
7. Known caveats going in (recorded so they aren't "discovered")
- Gateway VizInt auth is dev-permissive (
handle_jsonrpc_authaccepts any token; ~20 tests depend on it). Hardening is a product decision — record, don't flip mid-session. - In-process PipelineKernelClient is wired to the ORCHESTRATOR handler only;
chat (
cxp.send) keeps the canned Concierge reply on that path. gRPC path wires both. PubSubManager.publish()counts,broadcast()delivers — UI evidence must come from broadcast frames.requires_approvalon tools is declarative-only today (no HITL block anywhere in_call_llm_with_tools) — surfaced 2026-07-12; watching whether a live volk invokes an approval-worthy tool ungated is part of this session.AGENT_TOOLS["U:SD"]exposure is inert untilSeniorDevAgententers the tool-calling loop (owed wiring, session backlog).- Session-193 infra-gated UI test gaps: project-explorer seeded metadata, claude-chat thread filtering.
8. Exit criteria
- ≥1 intent per work type traced 7/7 checkpoints, plus the adversarial set.
- Every finding in the ledger has severity + evidence; zero un-triaged.
- Post-session: fix waves complete (or explicitly backlogged with Mohamed's sign-off), each fix regression-tested; full suite re-run authoritative (kernel verified); session CLI log-work/achievement + handoff updated.